RE: ldap query performance issue

[Chris Card <ctcard@hotmail.com>]

----------------------------------------
> From: ctcard@hotmail.com
> To: quanah@zimbra.com
> Subject: RE: ldap query performance issue
> Date: Thu, 23 May 2013 17:37:18 +0000
>
> ----------------------------------------
>> Date: Thu, 23 May 2013 10:06:51 -0700
>> From: quanah@zimbra.com
>> To: ctcard@hotmail.com; openldap-technical@openldap.org
>> Subject: Re: ldap query performance issue
>>
>> --On Thursday, May 23, 2013 4:40 PM +0000 Chris Card <ctcard@hotmail.com>
>> wrote:
>>
>>> Hi all,
>>>
>>> I have an openldap directory with about 7 million DNs, running openldap
>>> 2.4.31 with a BDB backend (4.6.21), running on CentOS 6.3.
>>>
>>> The structure of the directory is like this, with suffix dc=x,dc=y
>>>
>>> dc=x,dc=y
>>> account=a,dc=x,dc=y
>>> mail=m,account=a,dc=x,dc=y // Users
>>> ....
>>> licenceId=l,account=a,dc=x,dc=y // Licences,
>>> objectclass=licence ....
>>> group=g,account=a,dc=x,dc=y // Groups
>>> ....
>>> // etc.
>>>
>>> account=b,dc=x,dc=y
>>> ....
>>>
>>> Most of the DNs in the directory are users or groups, and the number of
>>> licences is small (<10) for each account.
>>>
>>> If I do a query with basedn account=a,dc=x,dc=y and filter
>>> (objectclass=licence) I see wildly different performance, depending on
>>> how many users are under account a. For an account with ~30000 users the
>>> query takes 2 seconds at most, but for an account with ~60000 users the
>>> query takes 1 minute.
>>>
>>> It only appears to be when I filter on objectclass=licence that I see
>>> that behaviour. If I filter on a different objectclass which matches a
>>> similar number of objects to the objectclass=licence filter, the
>>> performance doesn't seem to depend on the number of users.
>>>
>>> There is an index on objectclass (of course), but the behaviour I'm
>>> seeing seems to indicate that for this query, at some point slapd stops
>>> using the index and just scans all the objects under the account.
>>>
>>> Any ideas?
>>
>> Increase the IDL range. This is how I do it:
>>
>> --- openldap-2.4.35/servers/slapd/back-bdb/idl.h.orig 2011-02-17
>> 16:32:02.598593211 -0800
>> +++ openldap-2.4.35/servers/slapd/back-bdb/idl.h 2011-02-17
>> 16:32:08.937757993 -0800
>> @@ -20,7 +20,7 @@
>> /* IDL sizes - likely should be even bigger
>> * limiting factors: sizeof(ID), thread stack size
>> */
>> -#define BDB_IDL_LOGN 16 /* DB_SIZE is 2^16, UM_SIZE is 2^17
>> */
>> +#define BDB_IDL_LOGN 17 /* DB_SIZE is 2^16, UM_SIZE is 2^17
>> */
>> #define BDB_IDL_DB_SIZE (1<<BDB_IDL_LOGN)
>> #define BDB_IDL_UM_SIZE (1<<(BDB_IDL_LOGN+1))
>> #define BDB_IDL_UM_SIZEOF (BDB_IDL_UM_SIZE * sizeof(ID))
> Thanks, that looks like it might be the issue. Unfortunately I only see the issue in production, so patching it might be a pain.
I've tried this change, but it made no difference to the performance of the query.

Chris