[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Re: Problem loading 100K entries into OpenLDAP2.2.24
I think it is stange that memory is not released after loading,even after restart slapd!!
but in fact,The memory is not freed util you delete data file!!
my test:
DB_CONFIG:
set_cachesize 1 0 0
# For Loading indexing only
set_flags DB_TXN_NOSYNC
slapd.conf
cachesize 1
before loading:3549M memory free
CPU states: 90.2% idle, 0.0% user, 0.7% kernel, 9.1% iowait, 0.0% swap
load averages: 0.00, 0.00, 0.01 15:09:51
64 processes: 63 sleeping, 1 on cpu
CPU states: 94.6% idle, 0.0% user, 0.6% kernel, 4.8% iowait, 0.0% swap
Memory: 4096M real, 3549M free, 16G swap free
PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
16720 pldap 1 58 0 2496K 1624K sleep 0:00 0.02% csh
16728 pldap 1 59 0 1448K 1304K cpu0 0:00 0.00% top
16690 pldap 1 58 0 2496K 1544K sleep 0:00 0.00% csh
loading 1k data:
<121 [pldap] :/export/home/pldap>slapadd -f $PLDAP_HOME/config/slapd.conf -l 1k_testdata.ldif
after loading:3543M memory free,reduce as large as size of openldap-data,i also try to load 1000k data,after load,No free memory(total 6G)
leave.after delete the datafile,free memory is as large as that before load.
[pldap] :/export/home/pldap>var
<123 [pldap] :/export/home/pldap/pldap_run/var>du -sk openldap-data
5747 openldap-data
CPU states: 90.2% idle, 0.0% user, 0.7% kernel, 9.1% iowait, 0.0% swap
load averages: 0.00, 0.00, 0.01 15:12:11
64 processes: 63 sleeping, 1 on cpu
CPU states: 99.6% idle, 0.0% user, 0.4% kernel, 0.0% iowait, 0.0% swap
Memory: 4096M real, 3543M free, 16G swap free
PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
16728 pldap 1 59 0 1448K 1304K cpu2 0:00 0.00% top
16720 pldap 1 58 0 2496K 1624K sleep 0:00 0.00% csh
16690 pldap 1 58 0 2496K 1544K sleep 0:00 0.00% csh
----- Original Message -----
From: "Mavric Domen ITWEN5" <d.mavric@iskratel.si>
To: "罗志坚" <luozhijian@huawei.com>; <openldap-software@OpenLDAP.org>
Sent: Thursday, April 21, 2005 2:27 PM
Subject: RE: Problem loading 100K entries into OpenLDAP2.2.24
> Hi,
>
> I've tested also 2.2.24 release with patches suggested by
> Mr. Chu. Now, automatic removal of log files works as expected and
> consumption of memory is not increasing so quickly. I configured now slapd.conf to take about 120MB of RAM. The memory is not released after loading and the total consumption of memory (all processes)increases from 200MB to 600MB (of 1GB). When loading the next portion of 100K entries, total memory grows to 900MB (of 1GB), slapd takes 130MB.
> Is it normal that memory is not released after loading, even after
> performing checkpoints, and cleaning log files?
>
> Thanks, BR,
> Domen
>
>
> -----Original Message-----
> From: 罗志坚 [mailto:luozhijian@huawei.com]
> Sent: Thursday, April 21, 2005 4:22 AM
> To: openldap-software@OpenLDAP.org
> Subject: Fw: Problem loading 100K entries into OpenLDAP2.2.24
>
>
> > > Mavric Domen ITWEN5 wrote:
> > >
> > > > Hello,
> > > >
> > > > I've performed some tests with 2.3.2 beta release and in general it
> > > > works better, slapd stays alive, but it still takes about 200 MB of
> > > > memory and it does NOT free it after the loading.
> > >
> > > That is normal; you can reduce the entry cachesize in slapd.conf if you
> > > want it to keep less memory.
> > I think it is strange!!
> > After loading,
> > free memory reduce size = data file size
> > And the size have relation with slapd.conf,only affected by data file size.
> > Even restart slapd,the free memory size didn't increase.
>
> sorry,It's "the size have no relation with slapd.conf,only affected by data file size",not "the size have relation with slapd.conf,only affected by data file size"
>
> >
> > >
> > > > Also the log files are still not removed after the checkpoints, this
> > > > is also the case when doing manual checkpoint and "db_archive -d".
> > > > There are 104 log files in DB_HOME directory, each of 10MB. The only
> > > > way to clear the logs is to shutdown the server.
> > > >
> > > > What else I should check? I've run out of ideas to solve this
> > > > problem.
> > >
> > > You need an additional patch to BerkeleyDB and to back-bdb. The
> > > BerkeleyDB patch is here
> > >
> > > http://www.openldap.org/devel/cvsweb.cgi/build/BerkeleyDB42.patch
> > >
> > > and the back-bdb patch is in CVS HEAD, back-bdb/cache.c.
> > >
> > > The BerkeleyDB patch file will be distributed in the next 2.3 beta
> > > release, and the back-bdb patch will of course be present as well.
> > >
> > > --
> > > -- Howard Chu
> > > Chief Architect, Symas Corp. Director, Highland Sun
> > > http://www.symas.com http://highlandsun.com/hyc
> > > Symas: Premier OpenSource Development and Support
> > >