Re: delta-syncrepl contextCSN update timing, schema checking

[Howard Chu <hyc@symas.com>]

John Morrissey wrote:
> To make a long story short, it seems that syncrepl doesn't update the
> backend's contextCSN until it's processed its backlog? To check, I stopped
> another consumer and let a backlog build, then started it at debuglevel
> 16384 and watched the backend's contextCSN with ldapsearch(1). contextCSN
> didn't increment until the backlog was completely processed, even though I
> could see the changes it was processing with ldapsearch(1) as soon as they
> were processed.
>
> If a consumer processes replication without updating the backend's
> contextCSN, it will try to re-process the same replication entries when it
> starts up again, which will generally fail. This seems to leave one in a
> bind, either having to manually determine the correct value for contextCSN
> and update it manually, or remove the backend's data files and let syncrepl
> rebuild them from scratch. If this assessment is correct, this behavior
> doesn't seem desirable.

Your description of the behavior is correct. It's required to work that way 
with regular syncrepl; it probably should work differently for delta-sync but 
nobody pointed it out before.

In regular syncrepl the refresh phase does a regular LDAP search, whose 
results are returned in whatever arbitrary order the database normally 
retrieves things. Since that's pretty much guaranteed not to be the same as 
the order in which changes were made, we can't update the contextCSN until all 
of the changes have been received.

In delta-syncrepl the refresh is coming from the log, and unless somebody has 
been explicitly mucking around in the log DB, the entries will always be 
returned in order, so it's possible to update the contextCSN after each entry 
has been received. But it's up to the provider to send the cookie with each 
entry in this case, and the syncprov overlay doesn't really know the 
difference between delta-sync and regular sync, so it doesn't do it.

You could file an ITS for this, but I don't think we'll be changing this in 2.3.
--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/