Re: bdb/hdb cachesize calculation

[Jonathan Clarke <jonathan@phillipoux.net>]

On Tue, 13 Jul 2010 11:19:06 +0200, openldap-ml@stresst.net wrote:
> On 07/12/2010 07:40 PM, Quanah Gibson-Mount wrote:
>> --On Monday, July 12, 2010 5:01 PM +0200 openldap-ml@stresst.net wrote:
>>
>>> Attached to this message you'll find a quick and dirty bash script that
>>> should determine the cachesize for an in-production bdb/hdb backend by
>>> taking dn2id and id2entry as well as all existing indexes into account.
>>>
>>> Internally the script calls the db_stat tool in combination with the
>>> formulas found in the 2.4 admin guide under:
>>> http://www.openldap.org/doc/admin24/tuning.html#Caching
>>
>> Are you trying to determine the size of the cache in use, or what the
>> size of the cache should be?  If it is the latter, you just use the du
>> -c -h output of *.bdb.  If it is the former, it sounds like your  script
>> isn't calculating the right data.
> 
> Ahh I see - thanks! It seems that we do not need to calculate the 
> cachesize (see below). Nevertheless, as I'm using cn=config I'm a little 
> bit confused regarding the available different cache settings: The 
> related cn=config configuration directives are all prefixed by 
> "olcDb..." where olcDbCachesize for example seems to store the related 
> DB_CONFIG setting but olcDbDnCachesize seems to be focused on slapd 
> internal use.

Ah, these are two different settings:
- olcDbCachesize is the size (in number of entries) of OpenLDAP's internal
cache. This is not related to BerkeleyDB. It is recommended to set this to
a value bigger than the total number of entries, if possible.
- set_cachesize from DB_CONFIG is the size (in bytes) of BerkeleyDB's
cache. It is separate form the former, and does not store entries, but
indexes (basically). This can be set for convenience via the attribute
olcDbConfig in cn=config.

> BTW: slapd.conf's "dbconfig" directive ("convenience settings" mentioned 
> in man slapd-bdb.5) are not transformed into cn=config format when using 
> slaptest -f -F (I'm using slapd 2.4.21). slapd.conf's dbconfig directive 
> seems to be ignored during slaptest -f -F.

I've not tested this, but the "dbconfig" should be transformed into
"olcDbConfig" attribute. If this is not the case in the latest release,
please file a bug in the ITS.

> Last but not least please let me summarize my current understanding 
> after reading AdminGuide2.4 once again:
> 
> Section 21.4.1. ==> Berkeley DB Cache
> 
> 	21.4.1.(a) ==> slapadd import cachesize
> 	Cachesize is the overall filesize of *.bdb
> 
> 	21.4.1.(b) ==> operation cachesize
> 	Cachesize is the filesize of id2entry.bdb + 10%
> 
> 	BDB Cachesize is configureable various ways:
> 	- slapd.conf: (dbconfig ...)
> 	- cn=config: (olcDbCacheSize: x)

No: as explained above, this is a different setting: olcDbConfig is what
you're looking for.

> 	- DB_CONFIG: (set_cachesize 0 x 1)
> 	DB_CONFIG (if exists) takes precedence

Yes.

> Section 21.4.1.1. ==> Calculating Cachesize
> 
> 	only needed in case the above file sizes do not completely fit
> 	into available free memory. As we've currently enough free RAM
> 	we can skip these calculations.
> 
> 
> Section 21.4.2. ==> slapd(8) Entry Cache (cachesize)
> 
> 	cn=config (olcDbDNcacheSize: y)

Entry cache is olcDbCachesize. olcDbDnCacheSize is the DN cache setting.
See man slapd-bdb(5) for details.

Jonathan