Advice needed on a process to sync LDAP database w. Oracle

["Karl O. Pinc" <kpinc@artic.edu>]

Hi,

Thanks in advance for any help.  I don't think I'm doing anything wierd,
but I've got problems and would really like to know ahead of time if this
is going to fail.

I'm looking for comments on a procedure (see below) I cooked up which
updates our openldap database.  In particular I'd like to know:

- Is there something wrong with my procedure that makes the whole thing
unfeasible?

- Is release 2.0.7 is stable enough to use given my application?  I
developed the sync procedure on 1.2.5, am having problems, and it seems
sensible to upgrade while I'm still developing -- and to know that it's
going to work with the new version.

- If there's anything I'm doing wrong that would cause my cpu utilization
to go through the roof.  (Maybe this is a linux threads issue?)

- Am I doing something wrong by loading the database with ldif2ldbm?  I've
come across comments in the mailing list archives that say this is a bad
idea, that the database should be loaded with ldapadd.  However, I can't
seem to find the mail in the archives that's supposed to tell me how to
make ldapadd work fast enough to be feasible, or what's wrong with
ldif2ldbm.  Is there something I should know about schema violations?

- Why I can't use my ldap database after upgrading (on the slave) from
1.2.5 to 1.2.11?


The Application:

LDAP is used for an authentication database (via pam) for pop, radius, and
ftp, as a uid/gid database for ftp users, and as a mail routing database
(replacement for /etc/aliases).  There are about 6,000 entries.  There are
both a master and a slave to ensure availability.  An oracle database used
to run the institution serves as the source of information as to who has
accounts, what their email addresses are, etc.  The ldap database contains
some information, such as authentication tokens, unix uid/gid,  and
creation timestamps, not in the oracle database.  The procedure, below, is
used to sync the LDAP database with the oracle database. (Nightly, or
perhaps otherwise on demand.)  The only information that changes
independently of the oracle database is the authentication token, and there
are few of these changes.  The LDAP database must always be available (the
mail must go through!), but changes to the authentication token can be
denied for a short period while the  LDAP database is brought into sync
with the oracle database.


The Procedure that syncs LDAP with oracle:

Put the master in read-only mode.  (Stop slapd, alter slapd.conf, start slapd.)

Create an ldif file which is a merge of the LDAP-specific information
(authentication token) with a copy of the current oracle database.  (Dump
LDAP specific entries, such as the basename entry, in ldif format.  Append
to that a merge (mergesort on Unix uid style) of ldif of people and groups
from the LDAP database with the current oracle database.  The result is an
ldif of the new database we desire.  I've left out some details, like
deleting, etc. but you get the idea.)

Use ldif2ldbm to create a new master LDAP database.  (Use a modified copy
of slapd.conf with an alternate "directory" directive so the current master
is not yet touched.)

Shut down master LDAP server.

Copy new database directory in place.  (Archive the old directory just in
case.)

Restart master LDAP server.

Shut down slave LDAP server.

Copy the master LDAP database directory to the slave server.

Restart the slave LDAP server.

Take the master out of read-only mode.  (Stop slapd, alter slapd.conf,
start slapd.)



Other info:

Decided to create the database offline as it took _way_ to long to do it
on-line.  There are other services running on both the master and the slave
and it's not accecptable to hose the servers for very long.

The complete replacement, in effect, of the LDAP database with a fresh copy
of the oracle database ensures that the two don't get out of sync as they
might if the changes were incremental.

Running RedHat Linux 6.0 with 2.2.17 kernel (master) and RedHat Linux 6.1
with 2.2.12 kernel (slave.)

My program works fine on a small test sample, but trial runs using a full
set of real data lock up both the primary and secondary ldap servers.  (For
some reason the slave server seems to be the one used, it's cpu goes sky
high.  But both the master and the slave are affected, after a while X
crashes on both and you have to wait 20 minutes for a shell to login.)  So,
I tried upgrading from 1.2.5 to 1.2.12, discovered the aforementioned
problems, and that ldif2ldbm goes away in 2.0.7.  Frightened of a major
upgrade, I decided to write this note.  (I keep getting dragged off this
project for months at a time -- time to make something work.)

Thanks again, and thanks to the developer for openldap.

(FYI, we are currently using LDAP for authentication, as above.  It works,
with some group related problems that I expect to go away when we upgrade
from 1.2.5.  The mail routing will be new, and we can then eliminate the
old transfer-from-oracle program, ditch /etc/passwd, and update LDAP
directly.)

Karl

May the Legos (TM) always be swept from your path in the night.