RE: slapadding a large bdb database

["Howard Chu" <hyc@highlandsun.com>]

> -----Original Message-----
> From: owner-openldap-software@OpenLDAP.org
> [mailto:owner-openldap-software@OpenLDAP.org]On Behalf Of John Morrissey

> I'm beginning to move from OpenLDAP 2.0 to 2.1 (2.1.2). I slapcatted an
> existing ldbm database (~450k entries) and I'm trying to slapadd it into a
> bdb database.
>
> After about eight hours, I'm getting:
>
> slapadd: could not add entry dn="uid=foo,dc=example,dc=com" (line=1736045):
> txn_aborted! Cannot allocate memory (12)

What OS are you on? There seem to be OS-dependent memory leaks here but
we haven't got enough info from anyone to confirm it. Doesn't appear
to occur on Linux/glibc.
>
> The only appreciable rlimits in my shell are -s, stack size (8MB).
> Interestingly, this is the 65545th entry in this ldif, close
> enough to 2**16
> that it makes me wonder.

The Berkeley DB library itself has memory cache size limits that must be
explicitly configured. Read the BDB reference manual.
>
> STFWing yielded nothing. If anyone has any ideas, I'd appreciate them!
>
> Also, slapadd is extremely disk-bound during the add; is there any way to
> speed this up (other than getting faster disk)? dbnosync maybe?

dbnosync should help a little. Again, read the Berkeley DB reference guide,
you need to tune the log blocksize and the cachesize. For best performance
you need to put the transaction log on a separate physical disk from the rest
of
the database files.

  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support