RE: slapadding a large bdb database

["Howard Chu" <hyc@symas.com>]

Sorry to make you go on a wild goose chase. This is probably also due to a
bug (ITS#1939) in 2.1.3 back-bdb/idl.c. This has been patched in the CVS
HEAD.

The symptoms of this bug are that adds fail somewhere around 65535 entries,
just as you've encountered. This is an indication that an index slot in an
index database (objectclass, most likely) has just been maxed out. There is
code to convert this index slot from a fixed-size list into a range
specifier, and that code is broken in 2.1.3. If you can, try getting the
current idl.c (rev 1.51 as of today) from CVS and rebuild your slapd, that
should take care of this problem.

And again, for tuning purposes, you'll get better performance if you can put
the transaction logs on a separate physical disk from the actual databases.
Bumping up the log buffer size and setting NOSYNC also helps, of course.

  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support

> -----Original Message-----
> From: owner-openldap-software@OpenLDAP.org
> [mailto:owner-openldap-software@OpenLDAP.org]On Behalf Of John Morrissey

> First off, thanks for all your help, Howard. I really appreciate it.
>
> On Wed, Jul 10, 2002 at 01:02:24PM -0700, Howard Chu wrote:
> % Look again. http://www.sleepycat.com/docs/index.html Under "Additional
> % Documentation"
> % is a link to "Programmer's Tutorial and Reference Guide" (which is
> % http://www.sleepycat.com/docs/reftoc.html) In that document, start with
> % section 3 "Access Method Configuration", specifically 3.4.2 "Selecting a
> % cache size".
> %
> % Also read section 5.9 "Access Method Tuning".
> %
> % Then read section 9 "Berkeley DB Transactional Data Store
> Applications" with
> % special attention to section 9.21 "Transaction Tuning."
> %
> % To actually set most of these options, read section 7 "The Berkeley DB
> % Environment". While some of the BDB library options can be set
> in slapd.conf,
> % all of them can be set thru the BDB environment.
>
> I've given all this a thorough read. Here's what I've got in my DB_CONFIG
> now:
>
> set_cachesize 0 500000000 2
> set_flags DB_TXN_NOSYNC
> set_lg_regionmax 500000
> set_tx_max 500000
>
> slapadd speed has improved considerably with DB_TXN_NOSYNC, which
> takes care
> of one problem.
>
> However, the other remains. To recap, I've slapcatted an existing ldbm
> database (~450k entries) and I'm slapadding it as a bdb database. slapadd
> fails at entry 65545 with:
>
> slapadd: could not add entry dn="uid=foo,dc=example,dc=com" (line=29):
> txn_aborted! Cannot allocate memory (12)
>
> At this point, /var/lib/ldap/ is about 950MB, about 770MB of which is BDB
> transaction logs (log.*).
>
> If it helps, here's the cache-summary output of db_stat -m:
>
> 596MB 47KB 576B Total cache size (625000000 bytes).
> 2       Number of caches.
> 312M    Pool individual cache size.
> 116     Requested pages found in the cache (60%).
> 0       Requested pages mapped into the process' address space.
> 76      Requested pages not found in the cache.
> 0       Pages created in the cache.
> 76      Pages read into the cache.
> 66      Pages written from the cache to the backing file.
> 0       Clean pages forced from the cache.
> 0       Dirty pages forced from the cache.
> 0       Dirty buffers written by trickle-sync thread.
> 76      Current clean buffer count.
> 0       Current dirty buffer count.
> 65542   Number of hash buckets used for page location.
> 116     Total number of times hash chains searched for a page.
> 1       The longest hash chain searched for a page.
> 116     Total number of hash buckets examined for page location.
> 1470    The number of region locks granted without waiting.
> 0       The number of region locks granted after waiting.
>
> I'm at a loss as to what else to tune; this machine's got 768MB of RAM, of
> which it's using all but ~100MB for the BDB cache. The log region
> is several
> multiples of its default size (which, according to the docs, I shouldn't
> have to increase in my situation, anyway). The docs don't say what the
> default for tx_max is, but I doubt I'm hitting on a 500k simultaneous
> transaction limit.
>
> The rest of the BDB docs seem to apply more to performance tuning
> (BDB block
> size, lookup method [hash vs. btree, etc.]) or seem related to initial BDB
> creation, which I really have no control over.
>
> I've searched everyplace I can think of and spent a lot of time reading the
> BDB reference manual, but I'm still stuck. Is there something really simple
> I'm missing here? BTW, this is with OpenLDAP 2.1.3.
>
> thanks,
> john
> --
> John Morrissey          _o            /\         ----  __o
> jwm@horde.net        _-< \_          /  \       ----  <  \,
> www.horde.net/    __(_)/_(_)________/    \_______(_) /_(_)__
>