Re: openldap 2.4.16 hangs with dncachesize smaller than max number of records

[Buchan Milne <bgmilne@staff.telkomsa.net>]

On Monday 15 June 2009 04:44:21 Rodrigo Costa wrote:
> OpenLdap group,
>
> I'm having a possible issue that could be a problem. I have a DB with
> around 4 million entrances. In my slapd.conf I use the following cache
> constraints :
>
> #Cache values
> cachesize       10000
> dncachesize     3000000
> idlcachesize    10000
> cachefree       10

Your idlcachesize might be too small for your cachesize (but, you don't say 
which backend you are using). You haven't listed any Berkeley DB tuning.

>
> I'm also running the system in 2 machines in MirrorMode without problems
> about this configuration.
>
> My DB has exactly 3882992 entrances. In this way the dncachesize is
> smaller than the number of records. After I move the dncache constraint
> to a size smaller than the number of records(memory concern) I start to
> have some issues related with ldapsearch, for example.
>
> After the number of entrances in cache match the constraints(always pass
> a little) the system hangs for new searches. It appears to be for
> records "not yet" cached. If a record not cached is searched the
> ldapsearch bind but hangs during search. One example can be seen below :
>
> [root@brtldp11 ~]# time ldapsearch -LLL -x -D
> "cn=admin,ou=CONTENT,o=domain,c=fr" -w secret  -b
> "ou=CONTENT,o=domain,c=fr" -H ldap://10.142.15.170:389
> 'pnnumber=+554184011071'
> real    0m40.140s
> user    0m0.003s
> sys     0m0.001s
>
> Just after I press CTRL-C the command stopped. It would stay forever in
> this state. This happens after I ldapsearch the full DB and cache is
> filled.

But, not every entry in the DB is cached. slapd needs to access the remaining 
entries by some means to determine if they match the filter or not.

> If I then search this same record in the mirror I have the return very
> fast. See example :
>
> [root@brtldp11 ~]# time ldapsearch -LLL -x -D
> "cn=admin,ou=CONTENT,o=domain,c=fr" -w secret  -b
> "ou=CONTENT,o=domain,c=fr" -H ldap://10.142.15.172:389
> 'pnnumber=+554184011071'
> dn:
> pnnumber=\2B554184011071,uid=1219843774965\2B554184011071,ou=REPOSITORY,ou
>  =CONTENT,o=domain,c=fr
> subpnid: 0
> pntype: 2
> pncaps: 7
> objectClass: phoneinfo
> pnnumber: +554184011071
>
> real    0m0.257s
> user    0m0.002s
> sys     0m0.003s

Most likely because the OS filesystem cache has the index cached in memory. You 
may want to consider filling the OS filesystem cache (e.g. by copying files 
larger than the total system memory) to check ...

> So this is doesn't appear to be a record problem since also both systems
> are in mirror mode. Also the return is in a reasonable time, in around
> 257 miliseconds. After cached it can even be faster.

What is cached? Cached in which cache? Are you sure? How did you check?

> See the number of entrances I could search before system hangs :
>
> [root@brtldp12 ~]# wc -l /backup/temp2.txt
> 3078804 /backup/temp2.txt

How did you populate this file? With what search (specifically the filter).

> The number was 3,078,804 records since even dncache boundary is
> 3,000,000 it always pass a little.
>
> Appears that if a cache boundary is smaller than number of DB records
> slapd can hang when searching for new records not yet cached. Since I
> have multiple DBs with this order of records I needed to impose this
> boundary smaller than the number of records. My expectation was some
> performance degradation after cache is filled but not that system could
> hang(at least for non cached records).

You may be thrashing the Berkeley DB cache, if your indexes are too big for 
the cachesize you have set in DB_CONFIG (if you haven't then the default of 
256k is definitely way too much for 3 million records).

Also, the fact that you seem to only be searching on the naming attribute may 
be giving you a false idea of the real performance of your directory within 
the dncachesize limits (I am not sure if slapd can answer searches on the 
naming attribute faster from the dn cache than searchs on a non-naming 
attribute ...).

> After this situation happens the slapd process does not ends correctly
> only after kill -9 is given in the process.
>
> I would like to know if someone already passed by this situation and I
> believe it can be easily reproduced with cache configuration smaller
> than the number of records. I believe any DB in this situation could
> reproduce the same behavior.

Anyone running an OpenLDAP directory with the bdb or hdb backend with more 
than 10 000 entries who doesn't know what their Berkeley DB configuration is 
like in terms of DB cache is looking for problems.


Regards,
Buchan