[Date Prev][Date Next] [Chronological] [Thread] [Top]

Speed of slapadd for BDB on a Linux machine



Hello,

I'm trying to migrate about 19 million entries from OpenLDAP 2.0 to the new 2.3. Conversion and such things are done, but inserting the data takes days.

I hope someone can point me to some helpful direction, because 22 Entries/sec is not too good, especially not for real life use...

I'm using the following to insert the data
slapadd -q -v -c

My DB_CONFIG looks like this:

set_cachesize 1 524288000 1
set_lg_regionmax 262144
set_lg_bsize 2097152


Our OpenLDAP version is 2.3.27.

The speed is like following (took that from the output of my insertion script):

***********************************************************
Started run at: 1161817043

Partly run took 642 seconds for 577552 entries
Avg.: 899,614 entries/second
Partly run took 715 seconds for 577519 entries
Avg.: 807,719 entries/second
Partly run took 1001 seconds for 607053 entries
Avg.: 606,447 entries/second
Partly run took 1639 seconds for 610732 entries
Avg.: 372,625 entries/second
Partly run took 3311 seconds for 610547 entries
Avg.: 184,4 entries/second
Partly run took 7078 seconds for 610305 entries
Avg.: 86,2256 entries/second
Partly run took 13104 seconds for 610531 entries
Avg.: 46,5912 entries/second
Partly run took 19093 seconds for 610394 entries
Avg.: 31,9695 entries/second
Partly run took 22353 seconds for 610609 entries
Avg.: 27,3166 entries/second
Partly run took 23831 seconds for 610425 entries
Avg.: 25,6147 entries/second
Partly run took 24903 seconds for 610223 entries
Avg.: 24,504 entries/second
Partly run took 25121 seconds for 610223 entries
Avg.: 24,2913 entries/second
Partly run took 25382 seconds for 610177 entries
Avg.: 24,0398 entries/second
Partly run took 25013 seconds for 610042 entries
Avg.: 24,389 entries/second
Partly run took 25048 seconds for 610250 entries
Avg.: 24,3632 entries/second
Partly run took 24881 seconds for 610460 entries
Avg.: 24,5352 entries/second
Partly run took 24587 seconds for 610152 entries
Avg.: 24,816 entries/second
Partly run took 24907 seconds for 610252 entries
Avg.: 24,5012 entries/second
Partly run took 24841 seconds for 610605 entries
Avg.: 24,5805 entries/second
Partly run took 24627 seconds for 610432 entries
Avg.: 24,7871 entries/second
Partly run took 24344 seconds for 610229 entries
Avg.: 25,0669 entries/second
Partly run took 23958 seconds for 610343 entries
Avg.: 25,4755 entries/second
Partly run took 24186 seconds for 610629 entries
Avg.: 25,2472 entries/second
Partly run took 24349 seconds for 610377 entries
Avg.: 25,0678 entries/second
Partly run took 24634 seconds for 610679 entries
Avg.: 24,7901 entries/second
Partly run took 24897 seconds for 610230 entries
Avg.: 24,5102 entries/second
Partly run took 24902 seconds for 610258 entries
Avg.: 24,5064 entries/second
Partly run took 25800 seconds for 610327 entries
Avg.: 23,6561 entries/second
Partly run took 26605 seconds for 610478 entries
Avg.: 22,946 entries/second
Partly run took 27022 seconds for 610301 entries
Avg.: 22,5853 entries/second
Partly run took 27535 seconds for 610207 entries
Avg.: 22,1611 entries/second

Finished run at: 1162437948
Run took 620309 seconds for 18914119 entries
Avg.: 30,4914 entries/second
***********************************************************


We are using the following machine:

Linux ldaprep4 2.6.15.3 #1 SMP Mon Feb 13 09:18:43 CET 2006 i686 GNU/Linux
MemTotal:      5975412 kB
SwapTotal:     2150152 kB
2 * Intel(R) Pentium(R) III CPU family      1133MHz


The slapd.conf is as follows (Don't mind the /tmp as path, I changed that ;-)):


--------------------------------------------------------------------------------------------------
include         /tmp/etc/openldap/schema/core.schema
include         /tmp/etc/openldap/schema/freenet.schema
pidfile         /tmp/var/ldap/run/slapd.pid
argsfile        /tmp/var/ldap/run/slapd.args
modulepath      /tmp/lib
moduleload      back_bdb.la
access          to * by * write
loglevel        0
sizelimit       10000
timelimit       3600
cachesize       1000000
backend         bdb

#######################################################################
# BDB database definitions
#######################################################################

# first database definition & config directives
database        bdb

directory       /var/lib/ldap/
replogfile      /tmp/log/replica.log

rootdn          "cn=root,o=....."
rootpw          .....

suffix          "o=....."

#replica uri=ldap://ldaprep1:389 binddn="cn=root,o=..." bindmethod=simple credentials=...
#replica uri=ldap://ldaprep2:389 binddn="cn=root,o=..." bindmethod=simple credentials=...
#replica uri=ldap://ldaprep3:389 binddn="cn=root,o=..." bindmethod=simple credentials=...


#attribute       homeDirectory   ces
#attribute       folderName      ces
#attribute       locked          ces
#
index           cid          pres,eq
index           cn           pres,eq,sub
index           objectClass  pres,eq
index           folderName   pres,eq
index           locked       pres,eq
--------------------------------------------------------------------------------------------------


Thans in advance, Ralf