[Date Prev][Date Next] [Chronological] [Thread] [Top]

Problem loading 100K entries into OpenLDAP2.2.24

 

Hello,

 

I found a strange problem when populating my LDAP database with 100.000
entries (30 attrs. each).

I'm loading all 100K entries from one LDIF file using ldapmodify (maybe
I should use slapadd?).

 

At the begining total consumpton of RAM is about 200 MB (of 1GB), when I
start loading the memory

consumption of slapd process is about 3MB, but it keeps increasing for
about 20 min during the load and reaches 

290 MB!!!. The total consumption of RAM is 1GB of 1GB.  The slapd
process is killed by kernel after about 30K entries

(of 100K) inserted. 

syslog says:

            ...

        kernel: Out of Memory: Killed process 15748 (slapd)

            ...

See configuration for BDB and slapd bellow.

 

The second strange problem is increasing number of transaction log files
in a DB_HOME directory which

aren't removed automaticaly after checkpoints (see configuration
bellow). I believe that the first problem is related to this one. 

Even if I do a manual checkpoint (db_checkpoint) and check for log files
that can be removed, db_archive

says that all log files are in use. I can only remove log files
(db_archive -d) after stopping an restarting slapd server. 

Can't log files be removed after checkpoints during loading? I think
that's why I'm getting memory problems.

 

I checked also db_stat -m during the loading, but I didn't find anything
strange. I'll db files are on 96%-100% cache.

 

What I'm doing wrong? How should I configure my slapd, DB_CONFIG?

Thanks for tips,

BR,

Domen

 

 

 

 

I'm working with:

OpenLDAP 2.2.24

BerkeleyDB 4.2.52 + 4 patches (BDB)

 

Server info:

4 x 1,6 Ghz CPU

1 GB RAM

OS: Monta Vista Linux CGE 3.1

 

 

 

slapd configuration:

 

cachesize 100000

sizelimit 150000

checkpoint      1000    3

 

index cn eq,pres

index itRelease eq,pres

index itNeNode eq,sub,pres

index itProductName eq,sub,pres

index itProductCategory eq,sub,pres

index itNeNodeId eq,sub,pres

index itSerialNumber eq,sub,pres

index itParentContainerId eq,sub,pres

 

 

 

BDB configuration

 

set_lk_detect DB_LOCK_DEFAULT

set_lg_max 10485760

set_cachesize   0       20000000        0

set_lg_regionmax 262144

set_lg_bsize 2097152

 

# Automatically remove log files that are no longer needed.

set_flags DB_LOG_AUTOREMOVE

# Just use these settings when doing slapadd...

#set_flags DB_TXN_NOSYNC