(ITS#6646) back-hdb dncache disregarded in some searches

[ondrej.kuznik@acision.com]

Full_Name: Ondrej Kuznik
Version: 2.4.23
OS: 
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (62.168.56.1)


Example:
database hdb
suffix o=test
dncache 1000

is a database that has most of its 1M entries as uid=%d,o=one,o=test.

perform an unindexed search:
ldapsearch -b o=test uid=1

the entry is first in the db, so is spit out immediately. Watching dn
cn=Database 2,cn=Databases,cn=Monitor shows:
olmBDBDNCache: 1002

This is OK, but:

perform an unindexed search with base o=one,o=test:
ldapsearch -b o=one,o=test uid=1

watching dn cn=Database 2,cn=Databases,cn=Monitor shows that all 1M dns are
loaded into the cache and only after that the entry uid=1,o=one,o=test is spit
out. Then the dncache starts decreasing as the database filters out the
remaining entries:
olmBDBDNCache: 1000002

Another twist:
perform an unindexed search with base o=one,o=test but either kill ldapsearch
while hdb is still crunching or add a sizelimit greater than the number of
entries that should be returned:

ldapsearch -z2 -b o=one,o=test "uid=1*"

olmBDBDNCache contains most of the database and never decreases.
olmBDBDNCache: 1000002

A dn is later freed from the cache only in case it is referenced, should there
be another similar arc e.g. o=another,o=test, its dns are added inside the cache
too if we use the same technique.

This makes hdb unusable for a large database (whose set of dns alone can eat the
entire memory if loaded into dncache) unless one can guarantee that every search
is either indexed or with search base == suffix of the db.

In the example scenario, the value of cachefree affects almost nothing.