Re: multiple attribute search with single return slowness

[Buchan Milne <bgmilne@obsidian.co.za>]

On Mon, 26 Apr 2004, Blomberg David wrote:

> I am having a slowness response problem using Postfix on OpenLDAP (I
> have narrowed it down to LDAP response time being my largest bottleneck
> when querying multiple attributes.
> (This system supports over 4000 user accounts so before telling me how
> fast it is for you I noticed a big speed didference between 2000 users
> which was quick and when I went up to 4000---there might be something I
> misssed or a change I need to make for the larger user base if anyone
> knows of this please let me know)
> 
> for instance:
> 
> (&(|(mail=%s)(mailalternateaddress=%s))(accountstatus=active))
> which I ask to return "mailmessagestore"
> 
> and 
> 
> (|(mail=%s)(mailalternateaddress=%s))
> which I have return "mailforwardingaddress"
> 
> System:
> SMP Xeon based system with 4 GB RAM running Red Hat Enterprise Server
> 3.0 using Postfix (fixed version that works against LDAP) and OpenLDAP
> as provided by Red Hat. (I am coming here as Red Hat support is all but
> non-existant and I am tring a custom compiled from Open LDAP version in
> parallel to the full Red Hat system) 

I am working on replacing a system runing  OpenLDAP-2.0.27/ldbm/gdbm on 
RH7.3 running on Dell 1550s (ie probably <1GHz Xeons - I haven't seen the 
production boxen) with 2GB ram, with over 150000 qmail-style entries, 
serving about 10000 queries/minute under peal loaks.

Queries are of the form 
"(&(objectClass=qmailUser)(|(mail=$i)(mailAlternateAddress=$i)))"

FYI, on the prototype boxen (Dell 1650's with 1 or two 1333 Xeon's and 
1GB+ ram), OpenLDAP-2.1.2{5,9} (bdb) with db-4.2.52.{1,2} and 
OpenLDAP-2.0.27 with either gdbm or berekely are between 50% and 10* 
slower when running RHEL3 than RHEl2 (and the system wwe are replacing), 
and RH has not been able to give me any further insight on this, so we 
aren't recommending RHEL3.

The prototype system runs RPMS of 2.1.29/db4-2.52.2 based on the RHEL2.1 
SRPMS with some fixes from the RHEL3 RPMS forward-ported, and many 
customisations to actually make the packages work reliably with Red Hat's 
Cluster Manager (there is no way *any* RH-provided RPMS will work with 
RHCM if they don't at least fix the basic problems their packages have ... 
most of which aren't present on other distros ...).

> 
> What I have tried up until now:
> 1-enabling Postfix side LDAP caching 
> 	ldapXXXX_cache = yes
> 	ldapXXXX_cache_expiry = 60000
> 	ldapXXXX_cache_size = 32768
> 
> 	This did not work at all it only resulted in a number of errors to the
> log and a web search that blamed it not working of LDAP..... (Actually I
> believe this would fall to Postfix, but regardless it is broken so I am
> unable to cache the results
> 
> 2-I started the search lower on the tree before I searched in
> "o=domain.com" now I search in "ou=local_people,o=domain.com" and
> "ou=remote_people,o=domain.com"  
> 
> 	This didnt show any speed ups.
> 
> 3-I tried downloading and compiling OpenLDAP from scratch.
> 	After fully populating the database I find it a little quicker and on
> various test I come up as twice as fast for single attribute look ups,
> but only marginally faster for multiple attribute lookups.
> 	(As an aside I found that Red Hat was allowing some bad data to be
> input that OpenLDAP strict checking does not allow but Red Hat LDIF
> import never complaigned about so I fixed the entries-if nothing else at
> least I got some mess cleaned out of the system)

No, it was openldap-2.0.x that was doing it, regardless of what you were 
running it on.


> 
> 4-I tried adjusting my indexes in slapd.conf:
> 	index
> mail,mailalternateaddress,accountstatus,mailmessagestore,mailforwardingaddress eq

Your search filter contains the attribute accoutstatus, you probably want 
to index that too (anything in your common search filters should be 
indexed at minimum).

> 	Unfortunately after doing this and rebuilding the indexes I started
> loosing the ability to log in and other odd errros developed until I
> undid this.
> **actaully I am hoping someone can tell what I messed up on this ont as
> proper indexes are my best hope.**
> 

I assume your slapd was not running while you were running slapindex? If 
it was, that would definitely explain this.

> 5-Disabled the binding and all look up are now annonymous
> 	slight performance increase.


You shuld be able to get much better performance with minimal effort ...

Regards,
Buchan