Re: slapindex, 1 million entries, some numbers

[Andreas <andreas@conectiva.com.br>]

On Sun, Jun 01, 2003 at 09:52:40AM -0500, Jon Roberts wrote:
> it may also be enlightening to try loading/indexing this data with 
> logging turned off for comparison. Sorry if you already did this and I 

Logging is turned off, yes. Anyway, I don't think the loglevel
directive in slapd.conf has any meaning when using slapadd to load the data.

> In my own little experience, if there is a noticable gap in performance 
> between OpenLDAP and iPlanet, it's here. I loaded lots of LDIF data into 

I can't comment on that, I never used any other directory server. But I
do accept tips to improve performance.
Slapadd (with indexes) finished:

# time slapadd -l saida.ldif

real    1937m6.185s
user    162m18.640s
sys     84m53.190s

Transaction log files were up to 157 (at 100Mb each in my case). People should
really make sure they have some procedure for dealing with these log files if
they expect to have lots of writes to the database.

Just for the record (so it's easier for people searching the mailing list),
these were the settings for this run (some info extracted from the boot
messages):

- machine: quad PIII-700MHz
- two scsi disks: 160.000MB/s transfers (80.000MHz DT, offset 63, 16bit)
- scsi interface: Adaptec aic7899 Ultra160 SCSI adapter
- 8Gb of RAM, but, silly me, I had the wrong kernel booted up so it only
  used 900Mb.
Excerpt from slapd.conf:
schemacheck off
loglevel 0
database        bdb
directory       /var/lib/openldap-data
dbnosync
checkpoint 100000 10
cachesize 100000
index   objectClass     eq
index   cn,sn,mail      eq,sub
index   uid,givenName   eq
index   uidNumber,gidNumber     eq

DB_CONFIG:
set_flags       DB_TXN_NOSYNC
set_lg_dir      /storage/ldap
set_lg_max      104857600
set_lg_bsize    26214400
set_cachesize   0 1048576 0

(/storage/ldap is on sdb1, /var/lib/openldap-data is on sda1)

Search speed is blazingly fast on indexed attributes as expected. I have yet to come
up with some search benchmarks, I only used single ldapsearch commands for now and
I know they are not suited for this, being a shell command, binding every time, etc.
Typical searches on indexed attributes take milliseconds.