back-bdb IDL cache (was re: openldap versions and silent exit)

[Howard Chu <hyc@symas.com>]

Quanah Gibson-Mount wrote:
> --On Wednesday, August 11, 2004 8:29 AM -0400 Rich Graves 
> <rcgraves@brandeis.edu> wrote:
>
> > On Tue, 10 Aug 2004, Quanah Gibson-Mount wrote:
> >
> > > > idlcachesize?  I don't have this anywhere
> > > >
> > > > Backend is BDB 4.2.52.

Please note, "BDB 4.2.52" is the Sleepycat database library. It is not a 
"backend." The Sleepycat database library can be used by any of the 
OpenLDAP back-bdb, back-hdb, and back-ldbm backends. Don't confuse the 
backend with the database library, only the actual backend is OpenLDAP 
software. Just saying "my backend is BDB 4.2.52" (a) is wrong and (b) 
doesn't actually tell anyone exactly what piece of OpenLDAP you're using.

> > [...]
> >
> > > idlcachesize might be a 2.2 only option, I'm not sure, I abandoned 2.1 a
> > > while ago.

The IDL cache was first released in 2.1.10; it's been available for 
quite a long time.

> > idlcachesize is specific to bdb (or hdb) backend. It is missing from both
> > slapd.conf.5 and your example at
> > http://www.stanford.edu/services/directory/openldap/configuration/slapd-c
> > onf-replica.html though it does appear in the backend-specific
> > slapd-bdb.5 man page.

> Yeah, backend specific settings go in the backend specific man pages.

> I just added that option to Stanford's OpenLDAP servers last week, so I 
> haven't gotten around to updating the web page yet. ;)

> > Setting it does help!!! I'd been satisfied before, but now it's 
> > positively
> > zippy.

> > Any reason not to set idlcachesize to store all indices in RAM?

> As long as you have enough RAM, I don't think so.
>
> Howard gave me a general rule of thumb:
>
> For BDB, idlcachesize should be equal to cachesize in slapd.conf
>
> For HDB, idlcachesize should be 3 times cachesize in slapd.conf.
>
> I'll put that in my page too. ;)

Note that these are just wild guesses. I haven't done any testing to see 
what an optimal size might be. The number of IDLs in use depends on the 
nature of your search traffic. In back-bdb one IDL is used per search 
location - i.e.,
   If all of your searches start at the same base, using the same 
scope, then that's a single IDL.
   If you use different scopes with a given base, then one IDL per scope.
   If you use different bases, then one IDL per base (per scope).
Additionally, at least one IDL per indexed filter clause. Presence and 
equality clauses use one IDL each, substring clauses use some number 
proportional to the length of the substring filter.

back-hdb uses the same indexing code as back-bdb so its IDL usage is the 
same there. However, it uses a different approach for indexing search 
locations, and needs more IDL slots there.
--
  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support