Migrating a "big" custom 1.2.X installation to 2.X

["Martin Becht" <webmaster@ausfallprofis.de>]

Hi,

I am currently using OpenLDAP 1.2.13 in an installation with the
following data:

1xDedicated LDAP-"Root" Server
-> has the ldap-root o=foo,c=de
-> runs slapd and slurpd
-> gets only used for writing data to ldap-dir
-> has about 40 backends with root dn's: serverid=1,o=foo,c=de,
serverid=2,o=foo,c=de etc. one backend for each server whoes
configuration is contained in the tree
-> replicates each of these backends to precisely one machine (via
slurpd)
-> Data contained in LDAP is also present in a SQL-DB, LDAP-DB ist
rebuilt from the DB every night
-> Rebuild is done on a ramdisk, db-files are then copied to all slaves
over the network before slapd is restartet
-> RedHat 7.2 installation (kernel 2.4.17 - server crashes on
ldap-rebuild on 2.4.9)
-> ldif is about 100MB, db-size is about 700MB


40xSlave
-> runs slapd
-> stores posixaccounts, groups and other custom stuff in ldap
-> is used for reads only
-> redhat 6.2 installation
-> is running under heavy load a lot


I need to be able to scale the size of the directory and number of
slaves by factor 10 therefore my this should still work with 400 slaves
and a 1gig ldif.

Unfortunately I ran into some problems with my current setup:

1. Too many slurpds 
-> Root-Server is currently running about 2000 processes since slurpd
forks itself a lot for every backend replicated -> this will not scale
very well

2. slapd is unstable
-> slapd regularily crashes on the slave machines - about once per week
one of the slapds on the 40 slaves crashes
-> copying the backend-dir from the Root-Server to the slave "fixes" the
problem

3. Rebuilding makes LDAP unavaible for writing
-> While rebuilding the ldap-db from a ldif I have to prevent writes on
the running instance of slapd since they will not be incorporated into
the new ldap-db otherwise. This makes the directory unavaible to
write-accesses about 90min/day


I'd love to solve these problems. Do you think that upgrading to
OpenLDAP 2.X will solve some of the problems mentioned above given the
architecture I described ? Do you recommend changing the architecture in
any way ?


- How does OpenLDAP 2.X compare to 1.X in raw query performance ? (same
db, same indexes) Did anybody benchmark this / is there data available
on the web ?

- Has OpenLDAP 2.X proven to be stable under very high load ?

- What DB-Backend do you recommend for OpenLDAP 2.X to achive maximum
performance + reliability ?

- Do you see a way to solve problem no 3 with openldap ? I have found a
tool called "ldapdiff" that seems to do just what I want but I wonder if
this is fast enough / stable. Any experience with ldapdiff and "big"
(100MB-1Gig ldif) directories ?

- Is there a way to make openldap 2.X clients communicate with a local
server over a socket instead of the loopback-device ? (does the server
support this ?)


I'd really appreciate any input on this, OpenLDAP already did a lot of
good stuff for me and I wonder if my 2 years old setup described above
can be improved.


best regards,
Martin