strange performance problem

[Cuong bui <xoror@infuse.org>]

Hello all,

I'm doing some research on openldap to see if it's suitable for a certain kind 
of application. This application requires more write operations from the ldap 
server then an 'average' usage of ldap.

For this purpose i've written an multithreaded application to simulate 
simultaneous access to the ldap server. This application creates an variable 
number of threads that simulate an client. There's a central queue of task
that have to be performed. The queue has 3 kind of task (for now).

1. filter task. Simple search on a certain key
2. modify task. modification of a 'record'.
3. authentication task. (simple bind and unbind, simple authentication for 
now)

The task queue is built in such a way that the type of tasks are mixed with 
eachother.

i've generated a test dataset of 50.000 records and the .bdb files have been 
copied to an other location (ofcourse  when the server is offline) in order 
to restore data (instead of regenerating the test dataset, it can now be 
copied back).

The testruns were very promising. for instance with 1000 tasks (300 filter, 
600 modification, 100 authentication) and 32 threads only took 9 seconds to 
complete. However performing this same test a few times gives me a rather 
surprise effect. the first 4 runs all took about 9-13 seconds to complete. 
The 5th run and up takes up to 5 mins (!) or more.  Simple searches with 
ldapsearch takes up a few mins (after 4 testruns). (before that they only 
took a fraction of a second).

during the first 4 testruns the openldapserver used 98% cpu processing time. 
(seen in top) After the initial 4 runs, slapd only consumes 40%-75% cpu 
processing time. somethimes it's even dropping to a few %.

The only way to get slapd working 'normal' again, is to shutdown the server 
and restore the dataset. I have defined an index on  the field that i'm 
filtering on. (without it searches will take a very long time to complete).

The test server consist of a pentium 4 2,53 ghz running gentoo linux with 
kernel 2.4.20 and ultra dma (enabled) hdd   with 512 MB ram. Openldap 2.1.16 
was running on this machine. (but also tested with 2.1.12 and 2.1.15 they all 
gave this result) BDB was chosen as backend for this test.)

The testclient consist of an pentium 4 1,8ghz with gentoo linux running kernel 
2.4.19. The 2 systems were connected on a 100 mbit (switched) network. All 
the software were compiled with g++ 3.2.1

Has anyone an explanation for this behaviour ? Is this a known problem ? All 
hints are welcome. 


Thank in advance (and for your time)
Cuong

btw: are there any documentation on implementations details of openldap ? if 
yes, where can i find them :) Other documentation that might explain what i'm 
seeing is also (very) welcome.