[Date Prev][Date Next] [Chronological] [Thread] [Top]
Re: (ITS#6660) paged result searches fail to deallocate memory until slapd shutdown

To: openldap-its@OpenLDAP.org
Subject: Re: (ITS#6660) paged result searches fail to deallocate memory until slapd shutdown
From: masarati@aero.polimi.it
Date: Thu, 30 Sep 2010 11:14:48 GMT
Auto-submitted: auto-generated (OpenLDAP-ITS)
> --On Wednesday, September 29, 2010 8:34 PM +0200 masarati@aero.polimi.it
> wrote:
>
>>> --On Wednesday, September 29, 2010 12:38 AM +0000 quanah@zimbra.com
>>> wrote:
>>>
>>>> It appears to be a problem with the entry cache, which is set to
>>>> 25,000:
>>>
>>> Running with a fix from Howard, the entry cache behaves correctly.
>>> However, slapd still grows at the same rate.
>>>
>>> If I limit to only 10 paged results searches, slapd grows at a rate of
>>> 300MB Virtual and 300MB Resident for every set of 10 paged results
>>> searches
>>> I do concurrently, up until I run slapd out of memory.  There's
>>> something
>>> very wrong with paged results searches.
>>
>> Could it be configuration-specific?  I tested with a plain configuration
>> resulting from test003; maybe some player in the middle, say, is causing
>> entries to be duplicated and leaked, or read-locks on originals are not
>> released correctly?  Can you post a configuration that shows the issue?
>
> Hi Pierangelo,
>
> My testing shows the issue is only really visible with large databases
> that
> return giant result sets.  I don't expect you to see it with a small
> database and test003, because the amount of "lost" memory will be a few
> bytes at best.

If it's something related to bdb's cache size I agree; if it's related to
paged results I'd expect to notice it anyway using valgrind or so.

> My configuration itself is very minimal, but it is in cn=config format. ;)
>
> dn: olcDatabase={2}hdb
> objectClass: olcDatabaseConfig
> objectClass: olcHdbConfig
> olcDatabase: {2}hdb
> olcSuffix:
> olcAccess: {0}to attrs=userPassword  by anonymous auth  by
> dn.children="cn=adm
>  ins,cn=zimbra" write
> olcAccess: {1}to dn.subtree="cn=zimbra"  by
> dn.children="cn=admins,cn=zimbra"
>  write
> olcAccess: {2}to
> attrs=zimbraZimletUserProperties,zimbraGalLdapBindPassword,zi
>  mbraGalLdapBindDn,zimbraAuthTokenKey,zimbraPreAuthKey,zimbraPasswordHistory,
> z
>  imbraIsAdminAccount,zimbraAuthLdapSearchBindPassword  by
> dn.children="cn=admi
>  ns,cn=zimbra" write  by * none
> olcAccess: {3}to attrs=objectclass  by dn.children="cn=admins,cn=zimbra"
> write
>    by dn.base="uid=zmpostfix,cn=appaccts,cn=zimbra" read  by
> dn.base="uid=zmam
>  avis,cn=appaccts,cn=zimbra" read  by users read  by * none
> olcAccess: {4}to attrs=@amavisAccount  by
> dn.children="cn=admins,cn=zimbra"
> wr
>  ite  by dn.base="uid=zmamavis,cn=appaccts,cn=zimbra" read  by * +0 break
> olcAccess: {5}to attrs=mail  by dn.children="cn=admins,cn=zimbra" write
> by
> dn
>  .base="uid=zmamavis,cn=appaccts,cn=zimbra" read  by * +0 break
> olcAccess: {6}to attrs=zimbraAllowFromAddress  by
> dn.children="cn=admins,cn=zi
>  mbra" write  by dn.base="uid=zmpostfix,cn=appaccts,cn=zimbra" read  by *
> none
> olcAccess: {7}to filter="(!(zimbraHideInGal=TRUE))"
> attrs=cn,co,company,dc,di
>  splayName,givenName,gn,initials,l,mail,o,ou,physicalDeliveryOfficeName,posta
> l
>  Code,sn,st,street,streetAddress,telephoneNumber,title,uid,homePhone,mobile,p
> a
>  ger  by dn.children="cn=admins,cn=zimbra" write  by
> dn.base="uid=zmpostfix,cn
>  =appaccts,cn=zimbra" read  by users read  by * none
> olcAccess: {8}to
> attrs=zimbraId,zimbraMailAddress,zimbraMailAlias,zimbraMailCa
>  nonicalAddress,zimbraMailCatchAllAddress,zimbraMailCatchAllCanonicalAddress,
> z
>  imbraMailCatchAllForwardingAddress,zimbraMailDeliveryAddress,zimbraMailForwa
> r
>  dingAddress,zimbraPrefMailForwardingAddress,zimbraMailHost,zimbraMailStatus,
> z
>  imbraMailTransport,zimbraDomainName,zimbraDomainType,zimbraPrefMailLocalDeli
> v
>  eryDisabled  by dn.children="cn=admins,cn=zimbra" write  by
> dn.base="uid=zmpo
>  stfix,cn=appaccts,cn=zimbra" read  by * none
> olcAccess: {9}to attrs=entry  by dn.children="cn=admins,cn=zimbra" write
> by *
>   read
> olcLastMod: TRUE
> olcMaxDerefDepth: 15
> olcReadOnly: FALSE
> olcRootDN: cn=config
> olcSizeLimit: unlimited
> olcTimeLimit: unlimited
> olcMonitoring: TRUE
> olcDbDirectory: /opt/zimbra/data/ldap/hdb/db
> olcDbCacheSize: 25000
> olcDbCheckpoint: 64 5
> olcDbConfig: {0}#
> olcDbConfig: {1}# Set the database in memory cache size.
> olcDbConfig: {2}#
> olcDbConfig: {3}set_cachesize   0       52428800        0
> olcDbConfig: {4}
> olcDbConfig: {5}#
> olcDbConfig: {6}# Set database flags.
> olcDbConfig: {7}# Automatically remove log files that are no longer
> needed.
> olcDbConfig: {8}set_log_config DB_LOG_AUTO_REMOVE
> olcDbConfig: {9}
> olcDbConfig: {10}#
> olcDbConfig: {11}# Set log values.
> olcDbConfig: {12}#
> olcDbConfig: {13}set_lg_regionmax        262144
> olcDbConfig: {14}set_lg_max              10485760
> olcDbConfig: {15}set_lg_bsize            2097152
> olcDbConfig: {16}set_lg_dir              /opt/zimbra/data/ldap/hdb/logs
> olcDbConfig: {17}# Increase locks
> olcDbConfig:: ezE4fXNldF9sa19tYXhfbG9ja3MJMzAwMA==
> olcDbConfig:: ezE5fXNldF9sa19tYXhfb2JqZWN0cwkxNTAw
> olcDbConfig:: ezIwfXNldF9sa19tYXhfbG9ja2VycwkxNTAw
> olcDbNoSync: FALSE
> olcDbDirtyRead: FALSE
> olcDbIDLcacheSize: 25000
> olcDbIndex: objectClass eq
> olcDbIndex: entryUUID eq
> olcDbIndex: entryCSN eq
> olcDbIndex: cn pres,eq,sub
> olcDbIndex: uid pres,eq
> olcDbIndex: zimbraForeignPrincipal eq
> olcDbIndex: zimbraYahooId eq
> olcDbIndex: zimbraId eq
> olcDbIndex: zimbraVirtualHostname eq
> olcDbIndex: zimbraVirtualIPAddress eq
> olcDbIndex: zimbraMailDeliveryAddress eq,sub
> olcDbIndex: zimbraAuthKerberos5Realm eq
> olcDbIndex: zimbraMailForwardingAddress eq
> olcDbIndex: zimbraMailCatchAllAddress eq,sub
> olcDbIndex: zimbraShareInfo sub
> olcDbIndex: zimbraMailTransport eq
> olcDbIndex: zimbraMailAlias eq,sub
> olcDbIndex: zimbraACE sub
> olcDbIndex: zimbraDomainName eq,sub
> olcDbIndex: mail pres,eq,sub
> olcDbIndex: zimbraCalResSite eq,sub
> olcDbIndex: givenName pres,eq,sub
> olcDbIndex: displayName pres,eq,sub
> olcDbIndex: sn pres,eq,sub
> olcDbIndex: zimbraCalResRoom eq,sub
> olcDbIndex: zimbraCalResCapacity eq
> olcDbIndex: zimbraCalResBuilding eq,sub
> olcDbIndex: zimbraCalResFloor eq,sub
> olcDbLinearIndex: FALSE
> olcDbMode: 0600
> olcDbSearchStack: 16
> olcDbShmKey: 0
> olcDbCacheFree: 1000
> olcDbDNcacheSize: 0
> structuralObjectClass: olcHdbConfig
> entryUUID: 152ab0a8-333e-102d-8700-d562901af228
> creatorsName: cn=config
> createTimestamp: 20081020215916Z
> entryCSN: 20081020215916.275992Z#000000#000#000000
> modifiersName: cn=config
> modifyTimestamp: 20081020215916Z

There's no mention of overlays; I was wondering whether or not it could be
related to some adverse interaction, e.g. with sssvlv, which has some
paged-results specific code.  Can you also post on overlays either global
or specific to "olcDatabase={2}hdb"?

p.
Prev by Date: ITS#6661 (Was: FW: (6661))
Next by Date: Re: (ITS#6660) paged result searches fail to deallocate memory until slapd shutdown
Index(es):
- Chronological
- Thread