Re: Problem loading 100K entries into OpenLDAP2.2.24

["Kurt D. Zeilenga" <Kurt@OpenLDAP.org>]

For the record, I note that this post is actually from
<d.mavric@iskratel.si>.  Seems the submission software
replaced the From header for some reason when I
manually approved this post.

Kurt

>Hello,
>
> 
>
>I found a strange problem when populating my LDAP database with 100.000
>entries (30 attrs. each).
>
>I'm loading all 100K entries from one LDIF file using ldapmodify (maybe
>I should use slapadd?).
>
> 
>
>At the begining total consumpton of RAM is about 200 MB (of 1GB), when I
>start loading the memory
>
>consumption of slapd process is about 3MB, but it keeps increasing for
>about 20 min during the load and reaches 
>
>290 MB!!!. The total consumption of RAM is 1GB of 1GB.  The slapd
>process is killed by kernel after about 30K entries
>
>(of 100K) inserted. 
>
>syslog says:
>
>            ...
>
>        kernel: Out of Memory: Killed process 15748 (slapd)
>
>            ...
>
>See configuration for BDB and slapd bellow.
>
> 
>
>The second strange problem is increasing number of transaction log files
>in a DB_HOME directory which
>
>aren't removed automaticaly after checkpoints (see configuration
>bellow). I believe that the first problem is related to this one. 
>
>Even if I do a manual checkpoint (db_checkpoint) and check for log files
>that can be removed, db_archive
>
>says that all log files are in use. I can only remove log files
>(db_archive -d) after stopping an restarting slapd server. 
>
>Can't log files be removed after checkpoints during loading? I think
>that's why I'm getting memory problems.
>
> 
>
>I checked also db_stat -m during the loading, but I didn't find anything
>strange. I'll db files are on 96%-100% cache.
>
> 
>
>What I'm doing wrong? How should I configure my slapd, DB_CONFIG?
>
>Thanks for tips,
>
>BR,
>
>Domen
>
> 
>
> 
>
> 
>
> 
>
>I'm working with:
>
>OpenLDAP 2.2.24
>
>BerkeleyDB 4.2.52 + 4 patches (BDB)
>
> 
>
>Server info:
>
>4 x 1,6 Ghz CPU
>
>1 GB RAM
>
>OS: Monta Vista Linux CGE 3.1
>
> 
>
> 
>
> 
>
>slapd configuration:
>
> 
>
>cachesize 100000
>
>sizelimit 150000
>
>checkpoint      1000    3
>
> 
>
>index cn eq,pres
>
>index itRelease eq,pres
>
>index itNeNode eq,sub,pres
>
>index itProductName eq,sub,pres
>
>index itProductCategory eq,sub,pres
>
>index itNeNodeId eq,sub,pres
>
>index itSerialNumber eq,sub,pres
>
>index itParentContainerId eq,sub,pres
>
> 
>
> 
>
> 
>
>BDB configuration
>
> 
>
>set_lk_detect DB_LOCK_DEFAULT
>
>set_lg_max 10485760
>
>set_cachesize   0       20000000        0
>
>set_lg_regionmax 262144
>
>set_lg_bsize 2097152
>
> 
>
># Automatically remove log files that are no longer needed.
>
>set_flags DB_LOG_AUTOREMOVE
>
># Just use these settings when doing slapadd...
>
>#set_flags DB_TXN_NOSYNC