Re: syncrepl consumer is slow

[Emmanuel Lécharny <elecharny@gmail.com>]

Le 03/02/15 05:11, Howard Chu a écrit :
> Hallvard Breien Furuseth wrote:
>> On 29. jan. 2015 04:12, Howard Chu wrote:
>>> I'm considering adding an option to the consumer to write its entries
>>> with
>>> dbnosync during the refresh phase. The rationale being, there's
>>> nothing to
>>> lose anyway if the refresh is interrupted. I.e., the consumer can't
>>> update
>>> its contextCSN until the very end of the refresh, so any partial
>>> refresh that
>>> gets interrupted is wasted effort - the consumer will always have to
>>> start
>>> over from the beginning on its next refresh attempt.
>>
>> dbnosync loses consistency after a system crash, and it loses the
>> knowledge
>> that the DB may be inconsistent.  At least with back-mdb.   The safe
>> thing
>> to do after such a crash is to throw away the DB and fetch the entire
>> thing
>> from the provider.  Which I gather would need to happen automatically
>> with such an option.
>>
> Another option here is simply to perform batching. Now that we have
> the TXN api exposed in the backend interface, we could just batch up
> e.g. 500 entries per txn. much like slapadd -q already does.
> Ultimately we ought to be able to get syncrepl refresh to occur at
> nearly the same speed as slapadd -q.

Batching is ok, except that you never know how many entries you'll going
to have, thus you will have to actually write the data after a period of
time, even if you don't have the 500 entries.

This is where it would be cool to extend the cookie to receive the
expected number of updates you are going to receive (which will be
obviously be 1 in a normal running R&P replication, but > 1 most of the
time when reconnecting). In this case, youc an anticipate the batching
operation without having to tke care of the time issue.

My 2 cts.