[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: syncrepl consumer is slow



Hallvard Breien Furuseth wrote:
On 29. jan. 2015 04:12, Howard Chu wrote:
I'm considering adding an option to the consumer to write its entries
with
dbnosync during the refresh phase. The rationale being, there's
nothing to
lose anyway if the refresh is interrupted. I.e., the consumer can't
update
its contextCSN until the very end of the refresh, so any partial
refresh that
gets interrupted is wasted effort - the consumer will always have to
start
over from the beginning on its next refresh attempt.

dbnosync loses consistency after a system crash, and it loses the knowledge
that the DB may be inconsistent.  At least with back-mdb.   The safe thing
to do after such a crash is to throw away the DB and fetch the entire thing
from the provider.  Which I gather would need to happen automatically
with such an option.

Another option here is simply to perform batching. Now that we have the TXN api exposed in the backend interface, we could just batch up e.g. 500 entries per txn. much like slapadd -q already does. Ultimately we ought to be able to get syncrepl refresh to occur at nearly the same speed as slapadd -q.

--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/