Re: scope not ok errors on very large databases (ITS#3343)

[hyc@symas.com]

daniel.armbrust@mayo.edu wrote:

>Full_Name: Daniel Armbrust
>Version: 2.2.17
>OS: Fedora Core 2
>URL: ftp://ftp.openldap.org/incoming/
>Submission from: (NULL) (129.176.151.126)
>
>
>I have hit a problem where Openldap seems to stop using its indexes (or just
>does something wrong with them) and instead of quickly do a one level search, it
>ends up scanning the entire database, returning tons of "scope not ok"
>messages.
>
>The interesting part is that this only happens on 1 particular node in my
>database.  Renaming the node to something else has no affect.  It also seems to
>be size related - as when we split up the ldif that we load the database from,
>such that we load this node in pieces, we can load some parts of this node, but
>not all.  
>  
>
Exactly how large is this "node" ? You say it fails when you do a 
one-level search under it - how many immediate children does it have? 
There is a known limitation in back-bdb's index design; when any index 
slot hits 65536 entries it gets converted from an explicit list of 
entries into a "range". If the entries in this slot were not added in 
sorted order, then the range may span a large portion of the database.

For example, assuming the slot size was 4, and you had an index slot 
with entry IDs
     2,6,25,57
if you added a new entry under this slot, entry ID 99, this index slot 
would be converted into a range
     2-99
which would include quite a large number of entries that really have 
nothing to do with that slot.

You can tweak the slot sizes in back-bdb/idl.h BDB_IDL_DB_SIZE and 
BDB_IDL_UM_SIZE and recompile. I believe UM_SIZE must always be at least 
twice the DB_SIZE. You will also need to dump the database to LDIF 
before making this change, and reload from scratch afterward.

Also, loading your database in sorted order will help minimize the 
impact of this problem. I.e., make sure that all of the children of a 
particular node are loaded contiguously, without other intervening 
entries. This only helps when the DIT is relatively flat.

Originally back-hdb did not have this problem, although it does now 
because it shares the same search/indexing mechanism.

-- 
  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support