Re: Problem loading 100K entries into OpenLDAP2.2.24

[Howard Chu <hyc@symas.com>]

I'm seeing something similar, although DB_LOG_AUTOREMOVE is working for 
me. The 2.2.24 slapd starts at ~300MB and keeps growing. The 2.3/HEAD 
slapd doesn't do this. Probably should file an ITS with the particulars.

Kurt Zeilenga wrote:

> Hello,
>
>
>
> I found a strange problem when populating my LDAP database with 100.000
> entries (30 attrs. each).
>
> I'm loading all 100K entries from one LDIF file using ldapmodify (maybe
> I should use slapadd?).
>
>
>
> At the begining total consumpton of RAM is about 200 MB (of 1GB), when I
> start loading the memory
>
> consumption of slapd process is about 3MB, but it keeps increasing for
> about 20 min during the load and reaches 
>
> 290 MB!!!. The total consumption of RAM is 1GB of 1GB.  The slapd
> process is killed by kernel after about 30K entries
>
> (of 100K) inserted. 
>
> syslog says:
>
>            ...
>
>        kernel: Out of Memory: Killed process 15748 (slapd)
>
>            ...
>
> See configuration for BDB and slapd bellow.
>
>
>
> The second strange problem is increasing number of transaction log files
> in a DB_HOME directory which
>
> aren't removed automaticaly after checkpoints (see configuration
> bellow). I believe that the first problem is related to this one. 
>
> Even if I do a manual checkpoint (db_checkpoint) and check for log files
> that can be removed, db_archive
>
> says that all log files are in use. I can only remove log files
> (db_archive -d) after stopping an restarting slapd server. 
>
> Can't log files be removed after checkpoints during loading? I think
> that's why I'm getting memory problems.
>
>
>
> I checked also db_stat -m during the loading, but I didn't find anything
> strange. I'll db files are on 96%-100% cache.
>
>
>
> What I'm doing wrong? How should I configure my slapd, DB_CONFIG?
>
> Thanks for tips,
>
> BR,
>
> Domen
>
>
>
>
>
>
>
>
>
> I'm working with:
>
> OpenLDAP 2.2.24
>
> BerkeleyDB 4.2.52 + 4 patches (BDB)
>
>
>
> Server info:
>
> 4 x 1,6 Ghz CPU
>
> 1 GB RAM
>
> OS: Monta Vista Linux CGE 3.1
>
>
>
>
>
>
>
> slapd configuration:
>
>
>
> cachesize 100000
>
> sizelimit 150000
>
> checkpoint      1000    3
>
>
>
> index cn eq,pres
>
> index itRelease eq,pres
>
> index itNeNode eq,sub,pres
>
> index itProductName eq,sub,pres
>
> index itProductCategory eq,sub,pres
>
> index itNeNodeId eq,sub,pres
>
> index itSerialNumber eq,sub,pres
>
> index itParentContainerId eq,sub,pres
>
>
>
>
>
>
>
> BDB configuration
>
>
>
> set_lk_detect DB_LOCK_DEFAULT
>
> set_lg_max 10485760
>
> set_cachesize   0       20000000        0
>
> set_lg_regionmax 262144
>
> set_lg_bsize 2097152
>
>
>
> # Automatically remove log files that are no longer needed.
>
> set_flags DB_LOG_AUTOREMOVE
>
> # Just use these settings when doing slapadd...
>
> #set_flags DB_TXN_NOSYNC
>
>
>
>
>  


--
 -- Howard Chu
 Chief Architect, Symas Corp.       Director, Highland Sun
 http://www.symas.com               http://highlandsun.com/hyc
 Symas: Premier OpenSource Development and Support