RE: slapadding a large bdb database

["Howard Chu" <hyc@symas.com>]

> -----Original Message-----
> From: owner-openldap-software@OpenLDAP.org
> [mailto:owner-openldap-software@OpenLDAP.org]On Behalf Of John Morrissey
> Sent: Friday, August 09, 2002 1:41 PM

Just to finalize on the list...

> On Mon, Aug 05, 2002 at 11:13:22AM -0700, Howard Chu wrote:
> % > Of the 458,904 entries in the ldif, I can only "find" 31,729 of them.
> %
> % Not sure what to make of that. If you slapcat the database does
> the output
> % match the LDIF files that you fed into slapadd?
>
> Yup, if I slapcat the database, the number of entries matches exactly.
>
> For laughs, I threw a larger spindle into this machine and added the entire
> ldif in one go. It only returns 263,818 entries in this case (out
> of ~450k).
> Removing all the bdb files other than id2entry and reindexing yields the
> same results.

There was a bug in back-bdb's search routine that hits when the LDIF fed into
slapadd  is not well-ordered. I.e., back-bdb expected you to create all
parent entries before creating any child entries. This was a bad assumption,
since slapadd is intended to accept input in any order (and does, correctly).
The particular bug has been fixed in CVS and the fix will be in release
2.1.5.

However, back-bdb's search performance can suffer if your slapadd input is
not well-ordered. So, even though the randomly ordered LDIF files are valid
and usable, it would be best to order them properly before using them. We may
provide a mechanism for sorting the input, haven't worked out what the best
approach is yet.
>
> What's also interesting is that I get lots of (~45,000):
>
> Aug  8 16:59:47 oh.roc.frontiernet.net slapadd: bdb(o=frontier): Duplicate
> data items are not supported with sorted data
>
> when *adding* with slapadd. (I started with a pristine environment - no
> files whatsoever.)

The message is only a warning, there is no problem with the resulting index.
The code has been fixed to prevent the warning from being triggered.

  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support