RE: 2.1.17 performance issues

["Howard Chu" <hyc@highlandsun.com>]

> -----Original Message-----
> From: owner-openldap-software@OpenLDAP.org
> [mailto:owner-openldap-software@OpenLDAP.org]On Behalf Of Andreas

> On Wed, Apr 23, 2003 at 02:20:38PM -0700, Howard Chu wrote:
> > Andreas is correct. back-bdb maintains its own entry cache separately
from
> > whatever caching the BDB library itself uses. In very early OpenLDAP 2.1
code
> > there was no entry cache for back-bdb because we believed that the BDB
> > library's caching would be "good enough" but subsequent benchmarking
showed
> > that an entry cache was needed.
>
> So, configuring this parameter in slapd.conf *and* a cache in
> DB_CONFIG will cause
> two caches to be used, one in openldap and the other in the
> bdb library?

Right. The BDB library cache needs to be large enough to contain the active
set of bookkeeping info that BDB maintains about its on-disk data structures,
as noted in the SleepyCat documentation. If the cache isn't large enough,
then database lookups will cause lots of data to be paged in and out of the
cache while traversing the database, and performance will be terrible.

I frequently see questions about why all of this essential configuration and
documentation isn't integrated into OpenLDAP. I have answered this before,
but will repeat myself this once:

The SleepyCat documentation is very well written. Any further effort on our
part to document BDB's features would be redundant effort, and would require
continuous redundant effort to maintain as BDB moves forward. We don't have
time to spend doing this, but you're always welcome to submit new
documentation for inclusion into the distribution.

The SleepyCat configuration mechanism is also very well developed. But not
only is it redundant for us to integrate it into slapd, it is potentially
dangerous, and my position has been to put as little BDB-specific
configuration into slapd as possible. The problem is this - the BDB library
always reads the DB_CONFIG file after all other configuration is done, and
any settings in the DB_CONFIG file override anything that was set before. So,
if we were to provide slapd.conf keywords for all of the necessary BDB
tunables, you may run into situations where your carefully chosen settings
are completely ignored by the BDB library.

The BDB settings are persistent; when the BDB environment is created the
settings remain in the environment forever, unless db_recover is run to
destroy/recreate it. Here's where it gets dangerous - the BDB environment is
created by the first BDB application to touch a database. If the first
application is not from OpenLDAP (slapd/slapadd/etc.), (e.g., if you used
db_load or some other BDB utility) then the environment and databases will be
created with default settings that are probably inappropriate for slapd's
use. The only way to guarantee that the default settings that slapd requires
are loaded correctly, is to put them in the DB_CONFIG file where *every*
application will be forced to use them. The alternative is to tell you never
to use SleepyCat's BDB utilities with a slapd database, which would be
completely impractical - you *need* db_stat to tell you how the BDB
environment is performing, you *need* db_archive to tell you which log files
can be removed, etc. etc...

The bottom line - because the back-bdb databases must be managed by both the
OpenLDAP tools and the SleepyCat tools, we must use the SleepyCat DB_CONFIG
file in addition to slapd.conf.

  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support