Re: Problem loading 100K entries into OpenLDAP2.2.24

[Howard Chu <hyc@symas.com>]

This is now ITS#3666 and fixed in CVS OPENLDAP_REL_ENG_2_2 if you wish 
to test it. As noted before, the bug is not present in the 2.3 release.

Howard Chu wrote:

> I'm seeing something similar, although DB_LOG_AUTOREMOVE is working 
> for me. The 2.2.24 slapd starts at ~300MB and keeps growing. The 
> 2.3/HEAD slapd doesn't do this. Probably should file an ITS with the 
> particulars.
>
> Kurt Zeilenga wrote:
>
> > Hello,
> >
> >
> >
> > I found a strange problem when populating my LDAP database with 100.000
> > entries (30 attrs. each).
> >
> > I'm loading all 100K entries from one LDIF file using ldapmodify (maybe
> > I should use slapadd?).
> >
> >
> >
> > At the begining total consumpton of RAM is about 200 MB (of 1GB), when I
> > start loading the memory
> >
> > consumption of slapd process is about 3MB, but it keeps increasing for
> > about 20 min during the load and reaches
> > 290 MB!!!. The total consumption of RAM is 1GB of 1GB.  The slapd
> > process is killed by kernel after about 30K entries
> >
> > (of 100K) inserted.
> > syslog says:
> >
> >            ...
> >
> >        kernel: Out of Memory: Killed process 15748 (slapd)
> >
> >            ...
> >
> > See configuration for BDB and slapd bellow.
> >
> >
> >
> > The second strange problem is increasing number of transaction log files
> > in a DB_HOME directory which
> >
> > aren't removed automaticaly after checkpoints (see configuration
> > bellow). I believe that the first problem is related to this one.
> > Even if I do a manual checkpoint (db_checkpoint) and check for log files
> > that can be removed, db_archive
> >
> > says that all log files are in use. I can only remove log files
> > (db_archive -d) after stopping an restarting slapd server.
> > Can't log files be removed after checkpoints during loading? I think
> > that's why I'm getting memory problems.
> >
> >
> >
> > I checked also db_stat -m during the loading, but I didn't find anything
> > strange. I'll db files are on 96%-100% cache.
> >
> >
> >
> > What I'm doing wrong? How should I configure my slapd, DB_CONFIG?
> >
> > Thanks for tips,
> >
> > BR,
> >
> > Domen
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > I'm working with:
> >
> > OpenLDAP 2.2.24
> >
> > BerkeleyDB 4.2.52 + 4 patches (BDB)
> >
> >
> >
> > Server info:
> >
> > 4 x 1,6 Ghz CPU
> >
> > 1 GB RAM
> >
> > OS: Monta Vista Linux CGE 3.1
> >
> >
> >
> >
> >
> >
> >
> > slapd configuration:
> >
> >
> >
> > cachesize 100000
> >
> > sizelimit 150000
> >
> > checkpoint      1000    3
> >
> >
> >
> > index cn eq,pres
> >
> > index itRelease eq,pres
> >
> > index itNeNode eq,sub,pres
> >
> > index itProductName eq,sub,pres
> >
> > index itProductCategory eq,sub,pres
> >
> > index itNeNodeId eq,sub,pres
> >
> > index itSerialNumber eq,sub,pres
> >
> > index itParentContainerId eq,sub,pres
> >
> >
> >
> >
> >
> >
> >
> > BDB configuration
> >
> >
> >
> > set_lk_detect DB_LOCK_DEFAULT
> >
> > set_lg_max 10485760
> >
> > set_cachesize   0       20000000        0
> >
> > set_lg_regionmax 262144
> >
> > set_lg_bsize 2097152
> >
> >
> >
> > # Automatically remove log files that are no longer needed.
> >
> > set_flags DB_LOG_AUTOREMOVE
> >
> > # Just use these settings when doing slapadd...
> >
> > #set_flags DB_TXN_NOSYNC
> >
> >
> >
> >
> >  


--
 -- Howard Chu
 Chief Architect, Symas Corp.       Director, Highland Sun
 http://www.symas.com               http://highlandsun.com/hyc
 Symas: Premier OpenSource Development and Support