[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: contextCSN of subordinate syncrepl DBs

Rein Tollevik wrote:

Howard Chu wrote:


It appears that the current code is sending newCookie messages pretty much all
the time. It's definitely too chatty now, and it appears that it's breaking
test050 sometimes, though I still haven't identified exactly why. I thought it
was because the consumer was accepting the new cookie values unconditionally,
but even after filtering out old values test050 still failed. #if'ing out the
relevant code in syncprov.c makes test050 run fine though. (syncprov.c:1675
thru 1723.)


The newCookie messages should only be sent if the local csn set is
updated without being accompanied by a real modification. And in an MMR
setup like test050 that should never happen except maybe during the
initial replication of the databases.

Any occurrences of newCookie message should be a symptom of another bug,
Yes, you're right. I was finally able to trace one of the occurrences and fix it in HEAD. In this particular case, an entry was Added and that event was pushed into the syncprov response queue. Then the entry was Modified and again the response was queued. But when the queue was processed, it was retrieving the entry from the DB at that point in time - so the update was sent out with the Add's CSN in the sync control, but the entry's entryCSN attribute already had the Mod's stamp in it. That's why during the updateCookie step there was a missing CSN...
Anyway, this race condition has been fixed in HEAD by enqueuing the dup'd entry, so that the outbound updates all have consistent state. 


and I do believe one such race condition exist in syncrepl.  One
possible scenario, with 4 or more hosts:

server1 makes two or more changes to the db, with csn n and n+1.
server2 receive both, and starts replicating them to server3.
server3 receives and starts processing the first change from server1. It
updates cs_pvals in the syncrepl structure with the csn n of the first
modification.  Then, the same modification is received from server2, but
is rejected as being too old.  The second modification is received from
server2, this time being accepted.  This second modification is tagged
with csn n+1, which gets stored in the db by syncrepl_updateCookie and
picked up by syncprov. syncprov on server3 replicates the second change
with csn n+1 to server4.
server4 accepts the second modification from server3, without having
received the first change.  And when that arrives from server1 or 2 it
will be rejected as being too old.
Cannot happen. Every server sends its changes out in order; server 4 cannot receive csn n+1 from server 3 unless it has already received csn n from server 3. 


If the second modify operation is received and processed by server3
after it have added csn n to the csn queue, but before it is committed,
the second modification will be tagged with csn n.  The csn being
written to the db is still csn n+1 though, which will be picked up by
syncprov and trigger a newCookie message.  Even without this, the csns
stored in the db on server3 is invalid and will result in an incomplete
db should it fail before the first modification completes.
Sholdn't happen now; the cs_pmutex will prevent a new sync op from starting. Likewise syncprov_op_response will prevent a new mod from completing. 


The csns for any given sid are sent by the originating server in order,
I think the fix should be to always process them in the same order in
syncrepl.  For each sid in the csn set there should be one mutex, and
modifications with any given sid should only take place in the thread
holding the mutex.  To avoid stalling too long it must be possible for
the other syncrepl stanzas to note that a csn is too old without waiting
on the mutex for the csn sid.


That sounds ok.


I don't think it is correct for syncrepl to fetch csn values from
syncprov either. The only csn syncprov can update is the one with the
local sid, and syncrepl should simply ignore modifications tagged with
csn values with its own sid.  Provided syncrepl starts the replication
phase with a csn value with its own sid that is. The latter is to cover
the case where a server is being reinitialized from one of its peers, it
should then accept any changes that originated on the local server
before it was reinitialized.  Upon completing the initial replication
phase it will receive a csn set that may include its own sid, and it
should start ignoring modification with that sid.


Makes sense.


Neither of these changes should interfere with ordinary multi-master
configurations where syncrepl and syncprov are both use on the same
(glue) database.


Having spent the last 12 hours prodding at test050 I find that whenever I have
it working well, test058 "breaks" with contextCSN mismatches. At this point I
really have to question the rationale behind test058. First and foremost,
syncprov should not be sending gratuitous New Cookie messages to consumers
whose search terms are outside the scope of the update. I.e., if the actual
data update didn't go to the consumer, then the following cookie update should
not either. In such an asymmetric configuration, it should be expected that
the contextCSNs will not match across all the servers, and forcing them all to
match is beginning to look like an error, to me.


Whenever the provider makes a local change that should not be replicated
to the consumer the consumers database state continutes to be in sync.
Yet, its csn set indicates that it isn't and it will always start out
replicating all changes made after the oldest csn it holds.
Maybe. I think much of it will be a no-op because the intervening changes will prove to be irrelevant for that consumer. 


 Which can
be quite a lot.  The only way to fix this is to send the newCookie messages.


--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/