Re: OpenLDAP in Production; what is "big" ? also, core-dumps.

[asr@ufl.edu]

=> On Wed, 26 Feb 2003 19:03:46 +0000, pod <pod@herald.ox.ac.uk> said:

> In the Computing Services at the University of Oxford we are currently
> running OpenLDAP to support a user database of approx 30000 users for a
> mailstore cluster called Herald.  This is a critical core service and might
> be considered the equivalent of a 'business application'.

What does the OpenLDAP community consider to be "big"?

I'm running into some discouraging instabilities as I try to support our LDAP
installation at UF.  We've got ~1.3M people, about 1.5M objects so far,
probably ending up closer to 2-3M once we're really going.  I've been running
on 2.1.12, but am considering reverting to the 2.0 line.

I cannot 'ldapadd' my group of 'people who might be able to authenticate' for
love nor money.  It's about a 7MB group, ~130,000 members. It's not even
indexed yet.

I ran the ldapadd against a '-d -1' server instance, and it completes reading
the add request.  It then sets about ber_scanf'ing the buffer.

ber_scanf fmt ({m) ber:
ber_dump: buf=0x21611ed8 ptr=0x21611edb end=0x21c35ece len=6438899
ber_scanf fmt ({m{W}}) ber:
ber_dump: buf=0x21611ed8 ptr=0x21611f11 end=0x21c35ece len=6438845
ber_scanf fmt ({m{W}}) ber:
ber_dump: buf=0x21611ed8 ptr=0x21611f2a end=0x21c35ece len=6438820
ber_scanf fmt ({m{W}}) ber:
ber_dump: buf=0x21611ed8 ptr=0x21611f3a end=0x21c35ece len=6438804
ber_scanf fmt ({m{W}}) ber:
ber_dump: buf=0x21611ed8 ptr=0x21611f5e end=0x21c35ece len=6438768
ber_scanf fmt ({m{W}}) ber:
ber_dump: buf=0x21611ed8 ptr=0x21611f94 end=0x21c35ece len=6438714
ber_scanf fmt ({m{W}}) ber:
ber_dump: buf=0x21611ed8 ptr=0x21611fc0 end=0x21c35ece len=6438670

>From some examination of the ber_dumps, it appears that it's chopping one
attribute at a time off the front of the load, as is also suggested by the
advancing values of ptr. The coredump comes immediately (as measured by log
motion) after the call to ber_scanf that _ought_ to start reading members, the
last one in the list above.

When I shorten the group to ~100 members, and look at that debug trace, it
appears that all the members are slurped up in one pass.  Note that my
core-dump on the big add doesn't appear even to _get_ to the point where I'd
have the O(N^2) add operation to contend with.

... So am I just fooling myself that I ought to expect OpenLDAP to do this?

As I said earlier, I'm thinking about reverting to the 2.0 line; We didn't
have nearly this many objects before our changeover, but we didn't have
multiple core-dumps a day either.


- Allen S. Rout