RE: slapadding a large bdb database

["Howard Chu" <hyc@symas.com>]

> -----Original Message-----
> From: owner-openldap-software@OpenLDAP.org
> [mailto:owner-openldap-software@OpenLDAP.org]On Behalf Of John Morrissey

> Lastly, I noticed that reindexing ~450k entries (330MB of data in ldif
> format with about two dozen indices) generates about 6-8GB of BDB
> transaction logs. I haven't checked the source, do slap{add,index} perform
> all their operations in a single transaction? If so, is there any way to
> break these up, perhaps by having slapadd/slapindex use multiple
> transactions (i.e., every x entries, recover the database to free up some
> transaction logs and start a new transaction)? If not, could I modify the
> source to do something like this? I'd like to cut down on disk usage by
> purging some transaction logs while slap{add,index} is running.

You just need to use the "checkpoint" back-bdb directive in slapd.conf.
  checkpoint <kbytes> <minutes>

(see slapd-bdb(5))

After every transaction, the DB_ENV->txn_checkpoint() function is called
with these two parameters. If either <kbytes> data has been logged or
<minutes> time has past since the last checkpoint, a new checkpoint will
be performed. All log files up to the most recent checkpoint can be removed,
assuming no catastrophic loss occurs that requires restoring from an old
backup.
The BDB db_archive command tells you exactly which log files are no longer
needed. Beyond that, it's really up to you to choose and implement a policy
about how often to remove log files.

I suppose it's a bit of an inconvenience that even non-programmers must
thoroughly read the Berkeley API docs just to be able to use BDB effectively.
But the information is there, and I personally have better things to do
than to re-digest the knowledge and spit it out again in an OpenLDAP
document. Yes, it's complicated, there's a ton of knobs and dials to twist
and tune. One size does not fit all, we can't preconfigure and simplify
this for you.

  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support