Re: Slow to add 1 million items

[Brent Bice <bbice@sgi.com>]

On 02/04/2014 02:58 AM, Andrew Eross wrote:
> Hello all,
>
> I've been Google'ing around and searching the archives, but I haven't
> quite been able to find an answer, so I wanted to ask the list.
>
> I've been experimenting with OpenLDAP adds to see how quickly we can get
> data inserted into the DB.

   I've got a few OpenLDAP instances that I use for writing log data 
to, so write performance is critical, but since I'm building it from log 
data, absitively, posolutely, guaranteed perfect DB consistency isn't. 
I can always replay log data to rebuild the DB if, say, I had a power 
outage, the UPS failed, the RAID write-cache failed, the planets 
aligned, and I lost data. :-)

   First off, disable logging to syslog.  The first bottleneck I've run 
into (reads or writes) was not OpenLDAP but syslogd trying to write 
everything OpenLDAP told it.  Set "loglevel 0" in your slapd.conf. 
'Might also consider setting "threads" in slapd.conf if you've got 
multiple threads doing writes.  (see the man pages for this)

   Then, to optimize write performance, I added this to slapd.conf (in 
the database section):
checkpoint      1024000 60
dbnosync
envflags writemap

   And read the man pages so you understand these options and what 
you're getting into.  Some would consider this risky but it's perfect 
for my particular situation, where the data could always be rebuilt if 
needed (though I've never needed to).

   With this setup, on a single-CPU-core VM, I got around 5400 writes 
per second with several threads.  If memory serves, I got over 4000 
writes per second with one thread.  This was using the new MDB database 
back-end which I highly recommend. VERY fast. :-)

Brent

> I'm using Ubuntu 10.04, and I've tried both the packaged OpenLDAP 2.4.21
> using hdb, and just recently the latest OpenLDAP 2.4.39 using lmdb, both
> with relatively similar results.
>
> The short version is: to insert 1 million records, it's taking about 8
> hours on a machine with 2GB RAM / 3Ghz / SSD, which seems like a long
> time to me.
>
> The insert method is to use a single big ldiff file like this:
>
> ### snip ###
> dn: cn=newtest0,ou=customer,dc=test,dc=com
> objectclass: inetOrgPerson
> userPassword: 90sac90sudasdjcao0sdjtest
> telephoneNumber: 61400000000
> sn: none
>
> dn: cn=newtest1,ou=customer,dc=test,dc=com
> objectclass: inetOrgPerson
> userPassword: 90sac90sudasdjcao0sdjtest
> telephoneNumber: 61400000000
> sn: none
> ### snip ###
>
> Then add it with ldapadd:
> ldapadd -x -D cn=admin,dc=test,dc=com -w test -f ./test
>
> My configuration (for mdb in this instance) used with 2.4.39 is this:
> ### config ###
> include         /usr/local/etc/openldap/schema/core.schema
> include         /usr/local/etc/openldap/schema/cosine.schema
> include         /usr/local/etc/openldap/schema/inetorgperson.schema
>
> database  mdb
> directory /usr/local/var/openldap-data
> suffix    "dc=test,dc=com"
> rootdn    "cn=admin,dc=test,dc=com"
> rootpw    test
> maxsize   1980732000
>
> #Indexes
> index cn,telephoneNumber eq
> index   objectClass     eq
> ### config ###
>
> Any ideas or things I could try?
>
> Thank you!
> Andrew