Re: why is adding records slow?

[Ricky Charlet <rcharlet@redcreek.com>]

Howdy,

	Thanks for your post. It's very encouraging news. But I'm not sure what
accounts for the discrpenancy between our results. See note inline
below...

Mark Whitehouse wrote:
> 
> I have analyzed add performance for a while and have been getting much
> better performance than these results.
> 
> At present I can load about 10,000 records, each in about 60 secs.  This
> result is obtained using:
> 
>   Linux RH 6.2
>   Open LDAP 2.0.6
>   Berkeley DB 3.2.9 (make sure you are using BDB for the best results)
> 
> Also, I am using the C API to add the entries.  I open a single connection
> and then repeatedly add the entries.  Are you calling ldapadd repeatedly
> (i.e. 10k times) to add your entries?  If you are, that might explain your
> results as ldapadd opens a new connection each time.
> 

I use ldapadd <filename.ldif where my .ldif file has 10K records in it.
I do not know if this results in a network connection establishment per
record or not. I would hope not, but then again, my performance results
would make one suspect that it may be making a seperate connection per
record.



> The hardware I use is a PIII 750 server, 500MB, w. SCSI disks.  However, I
> get similar results (maybe 10k in 100 secs) with an off the shelf PIII 500
> client, 128MB, EIDE drive.
> 

The hardware I am running my test runs on is a 350 Mhz PII with 256Mb
RAM and a not so hot IDE disk.

> A couple of things which give huge write performance boosts (which your
> results confirm):
>   - turning off db syncing
>   - turning off substring indexes (equality indexes don't make much
> difference)
> 
> Mark
> 
> -----Original Message-----
> From: owner-openldap-software@OpenLDAP.org
> [mailto:owner-openldap-software@OpenLDAP.org]On Behalf Of Ricky Charlet
> Sent: Tuesday, June 12, 2001 12:11 PM
> To: openldap-software@OpenLDAP.org
> Subject: Re: why is adding records slow?
> 
> Howdy,
> 
>         I made a cut and past error in the names of my tests. The sequece of
> tests should be named First Second Third and Fourth, instead of First
> Second Second and Second.
> 
> >
> > Test = add 10K records to an empty ldbm database through ldapadd.
> > Measure time to completion.
> >
> > slapd.conf for First run:
> > -----------------------
> > schemacheck on
> > cachesize 100000
> > dbcachesize 15000000
> > dbnosync
> > #dbcachenosync
> > pidfile /var/run/slapd.pid
> > argsfile        /var/run/slapd.args
> > database        ldbm
> > suffix          "dc=ricky, dc=com"
> > rootdn          "cn=Manager, dc=ricky, dc=com"
> > rootpw          "******"
> > directory       /var/lib/ldap
> > # Indices to maintain
> > index   objectClass,uid,uidNumber,gidNumber     eq
> > index   cn,mail,surname,givenname               eq,subinitial
> >
> > time to add 10K records = 7692 seconds
> >
> > ======================================================
> >
> > slapd.conf for Second run: (turn off indexing)
> > -----------------------
> > schemacheck on
> > cachesize 100000
> > dbcachesize 15000000
> > dbnosync
> > #dbcachenosync
> > pidfile /var/run/slapd.pid
> > argsfile        /var/run/slapd.args
> > database        ldbm
> > suffix          "dc=ricky, dc=com"
> > rootdn          "cn=Manager, dc=ricky, dc=com"
> > rootpw          "******"
> > directory       /var/lib/ldap
> > # Indices to maintain                           eq
> > #index  objectClass,uid,uidNumber,gidNumber     eq
> > #index  cn,mail,surname,givenname               eq,subinitial
> >
> > time to add 10K records = 2516 seconds
> >
> > ======================================================
> >
> > slapd.conf for Second run: (turn on dbcachenosync)
> > -----------------------
> > schemacheck on
> > cachesize 100000
> > dbcachesize 15000000
> > dbnosync
> > dbcachenosync
> > pidfile /var/run/slapd.pid
> > argsfile        /var/run/slapd.args
> > database        ldbm
> > suffix          "dc=ricky, dc=com"
> > rootdn          "cn=Manager, dc=ricky, dc=com"
> > rootpw          "******"
> > directory       /var/lib/ldap
> > # Indices to maintain                           eq
> > #index  objectClass,uid,uidNumber,gidNumber     eq
> > #index  cn,mail,surname,givenname               eq,subinitial
> >
> > time to add 10K records = 1900 seconds
> >
> > ======================================================
> >
> > slapd.conf for Second run: (turn schemacheck to off)
> > -----------------------
> > schemacheck off
> > cachesize 100000
> > dbcachesize 15000000
> > dbnosync
> > dbcachenosync
> > pidfile /var/run/slapd.pid
> > argsfile        /var/run/slapd.args
> > database        ldbm
> > suffix          "dc=ricky, dc=com"
> > rootdn          "cn=Manager, dc=ricky, dc=com"
> > rootpw          "******"
> > directory       /var/lib/ldap
> > # Indices to maintain                           eq
> > #index  objectClass,uid,uidNumber,gidNumber     eq
> > #index  cn,mail,surname,givenname               eq,subinitial
> >
> > time to add 10K records = 1900 seconds (no change from last test)
> >
> > ======================================================
> >
> > --
> >   Ricky Charlet   : Redcreek Communications   : usa (510) 795-6903
> 
> --
>   Ricky Charlet   : Redcreek Communications   : usa (510) 795-6903

-- 
  Ricky Charlet   : Redcreek Communications   : usa (510) 795-6903