RE: Openldap scalability issues

["Armbrust, Daniel C." <Armbrust.Daniel@mayo.edu>]

I'm curious as to why openLdap requires so much cache for doing inserts.  Does it have something to do with the indexing process?  I just can't picture why it needs to be caching much of anything to do inserts.  I would expect caches to have more of an affect on reads and searches....

>From what I am seeing, if you want to programmatically build a ldap database - we are limited to a database of about 3 times the cache size.  And its not that it just becomes a lot slower at that point... It literally dies.  I'm going to try another load tonight, with the max cache I can allocate, and see what happens... 

I would appreciate it if anyone who knows the internals could provide some insight - is this an issue of something that would just require more work in openldap - so that the runtime process can handle loading large databases (by my definition) like the bulk loader can, a bug in openldap (not releasing things from memory), a bug (or a lack of capability) in BerkeleyDB (not releasing memory) or just a flat out design limitation?

I guess I can also play with turning off schema check (which I don't expect to have any affect) or turning off our indexes... And trying to reindex later - but since the reindex command doesn't seem to work, I would have to dump out the ldif, then turn on the indexes, and then re-add it...  Certainly not something I want to do on a regular basis.

Thanks,

Dan