Re: slapcat and BDB transaction-logs

[Buchan Milne <bgmilne@staff.telkomsa.net>]

On Friday 15 August 2008 13:51:37 tamarin p wrote:
> Hello,
>
> I'm trying to migrate a directory from OpenLDAP 2.3.x to an OpenLDAP 2.4.11
> test installation. My LDAP-experience is less than extensive, but I've run
> slapcat on the old base, and am now trying to import it back into 2.4.11
> with slapadd. The base is large enough that the import will take several
> days at the current rate (3gb LDIF/approx 4 million entries), and so I
> can't monitor the process at all times.

It shouldn't take much more than 4 hours on any hardware you should be running 
in production ...

> My problem is with the backend transaction logs, though. These are chucked
> out at a rate of approximately one per minute until the disk runs out of
> space. I did a quick google on the issue and found that you should be able
> to remove logs not currently involved in any transactions by running
> db_archive -d (
> http://www.oracle.com/technology/documentation/berkeley-db/db/ref/transapp/
>logfile.html). I've also tried adding a set_flags DB_LOG_AUTOREMOVE in my
> DB_CONFIG with no luck. Then I figured I could set up a cronjob to run
> db_archive hourly, but when I run db_archive -d (or more to the point,
> slapd_db_archive2.4 -d, as I've installed using Buchan Milne's RPMs), not a
> single log-file is removed.

What is your full command line ? If the working directory is not the database 
directory, you need to specify the path to the database directory after the -h 
flag.

However, for bulk imports you should use the -q flag to slapadd, which will 
disable transactions for the duration of the import (and it has such a big 
impact on import speed that one typically doesn't worry that interrupting the 
import will result in a corrupt database).

> Moreover, none of the *.bdb files are
> increasing in size while slapadd is running, only additional log-files and
> __db.001 etc.Does slapadd basically do all adds in one huge transaction
> before committing at the end, or am I doing something wrong? Can I simply
> erase log-files with rm on the fly instead? The checkpointing in slapd.conf
> is set to 256k/5min as per default. Anything else I'm missing?

-q ...

Regards,
Buchan