Re: syncrepl: large datasets and expediting consumer's initialization

[Dieter Kluenter <dieter@dkluenter.de>]

Am Mon, 5 Apr 2010 14:28:49 -0230
schrieb Paul Fardy <paul.fardy@utoronto.ca>:

> I'm having trouble getting the consumer synced in reasonable time.
> My tests were with fewer than 20 entries in the datastore and I saw
> no problems.
> 
> But we have 260,000 inetOrgPersons (with only a few attributes for  
> each user: uid cn sn givenName mail userPassword).
> 
> I've set up syncrepl:
> 
> PROVIDER
> > # Indices to maintain for this database
> > index objectclass,entryCSN,entryUUID eq
> > index ou,cn,mail,surname,givenname      eq,sub
> > index uidNumber,gidNumber,loginShell    eq
> > index uid,memberUid                     eq,sub
> > index nisMapName,nisMapEntry            eq,sub
> >
> > overlay syncprov
> > syncprov-checkpoint 100 1
> > syncprov-sessionlog 100
> >
> > limits dn.children="ou=replicators,dc=service,dc=utoronto,dc=ca"  
> > size=unlimited time=unlimited
> 
> (I index attributes I'm not currently using. I presume that's not
> the problem.)
> 
> CONSUMER
> > syncrepl rid=123
> >     provider=ldap://PROVIDER:389
> >     type=refreshAndPersist
> >     interval=00:00:10:00
> >     retry="60 10 300 +"
> >     searchbase="dc=service,dc=utoronto,dc=ca"
> >     filter="(objectClass=*)"
> >     scope=sub
> >     schemachecking=off
> >     starttls=critical
> >     bindmethod=simple
> >      
> > binddn="uid=replicator,ou=replicators,dc=service,dc=utoronto,dc=ca"
> 
> I've tried with and without slapcat/slapadd to initialize the  
> consumer. On our slower system, slapadd took 98 minutes to rebuild
> the database; the faster was 35 minutes (and I have only one
> consumer right now).
> 
> A full transfer via syncrepl is slow: 10 entries per second:
[...]

What filesystem are you running?

-Dieter

-- 
Dieter Klünter | Systemberatung
sip: +49.40.20932173
http://www.dpunkt.de/buecher/2104.html
GPG Key ID:8EF7B6C6