[Date Prev][Date Next] [Chronological] [Thread] [Top]

RE: Problem loading 100K entries into OpenLDAP2.2.24

Hello,

I've performed some tests with 2.3.2 beta release and in 
general it works better, slapd stays alive, but it still takes about 
200 MB of memory and it does NOT free it after the loading.

Also the log files are still not removed after the checkpoints, this
is also the case when doing manual checkpoint and "db_archive -d".
There are 104 log files in DB_HOME directory, each of 10MB.
The only way to clear the logs is to shutdown the server.

What else I should check? I've run out of ideas to solve this problem.

Thanks,
BR,
Domen


-----Original Message-----
From: Howard Chu [mailto:hyc@symas.com] 
Sent: Saturday, April 16, 2005 9:05 PM
To: Mavric Domen ITWEN5
Cc: openldap-software@OpenLDAP.org
Subject: Re: Problem loading 100K entries into OpenLDAP2.2.24

This is now ITS#3666 and fixed in CVS OPENLDAP_REL_ENG_2_2 if you wish 
to test it. As noted before, the bug is not present in the 2.3 release.

Howard Chu wrote:

> I'm seeing something similar, although DB_LOG_AUTOREMOVE is working 
> for me. The 2.2.24 slapd starts at ~300MB and keeps growing. The 
> 2.3/HEAD slapd doesn't do this. Probably should file an ITS with the 
> particulars.
>
> Kurt Zeilenga wrote:
>
>>
>>
>> Hello,
>>
>>
>>
>> I found a strange problem when populating my LDAP database with
100.000
>> entries (30 attrs. each).
>>
>> I'm loading all 100K entries from one LDIF file using ldapmodify
(maybe
>> I should use slapadd?).
>>
>>
>>
>> At the begining total consumpton of RAM is about 200 MB (of 1GB),
when I
>> start loading the memory
>>
>> consumption of slapd process is about 3MB, but it keeps increasing
for
>> about 20 min during the load and reaches
>> 290 MB!!!. The total consumption of RAM is 1GB of 1GB.  The slapd
>> process is killed by kernel after about 30K entries
>>
>> (of 100K) inserted.
>> syslog says:
>>
>>            ...
>>
>>        kernel: Out of Memory: Killed process 15748 (slapd)
>>
>>            ...
>>
>> See configuration for BDB and slapd bellow.
>>
>>
>>
>> The second strange problem is increasing number of transaction log
files
>> in a DB_HOME directory which
>>
>> aren't removed automaticaly after checkpoints (see configuration
>> bellow). I believe that the first problem is related to this one.
>> Even if I do a manual checkpoint (db_checkpoint) and check for log
files
>> that can be removed, db_archive
>>
>> says that all log files are in use. I can only remove log files
>> (db_archive -d) after stopping an restarting slapd server.
>> Can't log files be removed after checkpoints during loading? I think
>> that's why I'm getting memory problems.
>>
>>
>>
>> I checked also db_stat -m during the loading, but I didn't find
anything
>> strange. I'll db files are on 96%-100% cache.
>>
>>
>>
>> What I'm doing wrong? How should I configure my slapd, DB_CONFIG?
>>
>> Thanks for tips,
>>
>> BR,
>>
>> Domen
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> I'm working with:
>>
>> OpenLDAP 2.2.24
>>
>> BerkeleyDB 4.2.52 + 4 patches (BDB)
>>
>>
>>
>> Server info:
>>
>> 4 x 1,6 Ghz CPU
>>
>> 1 GB RAM
>>
>> OS: Monta Vista Linux CGE 3.1
>>
>>
>>
>>
>>
>>
>>
>> slapd configuration:
>>
>>
>>
>> cachesize 100000
>>
>> sizelimit 150000
>>
>> checkpoint      1000    3
>>
>>
>>
>> index cn eq,pres
>>
>> index itRelease eq,pres
>>
>> index itNeNode eq,sub,pres
>>
>> index itProductName eq,sub,pres
>>
>> index itProductCategory eq,sub,pres
>>
>> index itNeNodeId eq,sub,pres
>>
>> index itSerialNumber eq,sub,pres
>>
>> index itParentContainerId eq,sub,pres
>>
>>
>>
>>
>>
>>
>>
>> BDB configuration
>>
>>
>>
>> set_lk_detect DB_LOCK_DEFAULT
>>
>> set_lg_max 10485760
>>
>> set_cachesize   0       20000000        0
>>
>> set_lg_regionmax 262144
>>
>> set_lg_bsize 2097152
>>
>>
>>
>> # Automatically remove log files that are no longer needed.
>>
>> set_flags DB_LOG_AUTOREMOVE
>>
>> # Just use these settings when doing slapadd...
>>
>> #set_flags DB_TXN_NOSYNC
>>
>>
>>
>>
>>  
>>
>
>


-- 
  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support