Re: question about limits

["Pierangelo Masarati" <ando@sys-net.it>]

> Pierangelo Masarati wrote:
>
>>>Hi,
>>>
>>>I have a question about limits:
>>>
>>>when I do a query like :
>>>
>>>ldapsearch -x -b"o=mail,dc=example,dc=com" -w password
>>>-D"userlogin=user,o=admins,dc=example,dc=com" -LLL
>>>"(&(mail=*@example.com)(objectClass=mailUser))"
>>>
>>>where o=mail,dc=example,dc=com (objectClass=mailUser) contains about
>>>50000 entries (indexed)
>>>
>>>
>>
>>What indices?
>>
>>
>>
> # Indexing options
> index           objectClass,customerRoot        eq,pres
> index           mailDomain,mail,userlogin       eq,pres,sub
> index           o,cn,sn                         eq,pres,sub
> index           maildir,primarySmtp             eq,pres
> index           mailmessagestore                eq,pres
> index           homeDirectory,maildrop          eq,pres
> index           FTPHomedirectory                eq,pres
> index           dc,associatedDomain             eq,pres
>
>>>the query takes about 35 seconds to complete.
>>>
>>>but when I do it like this:
>>>
>>>ldapsearch -x -b"o=mail,dc=example,dc=com" -w password
>>>-D"cn=admin,dc=example,dc=com" -LLL
>>>"(&(mail=*@example.com)(objectClass=mailUser))"
>>>
>>>where : cn=admin,dc=example,dc=com is the root database admin the query
>>>takes about 0.4 seconds
>>>
>>>
>>
>>ACL problem?  How many (regex) ACLs are you using?
>>
>>
>>
> access to dn=".*,o=admins,dc=example,dc=com"
>         attr=userPassword
>         by anonymous auth
>         by self write
>         by * none
>
> access to dn="o=admins,dc=example,dc=com"
>         by dn="userlogin=[^,]+,o=admins,dc=example,dc=com" read
>         by * none
>
> access to dn=".*,customerRoot=([^,]+),o=customers,dc=example,dc=com"
>         attr=userPassword
>         by dn="userlogin=[^,]+,o=admins,dc=example,dc=com" write
>         by
> dn="userlogin=[^,]+,customerRoot=$1,o=customers,dc=example,dc=com" write
>         by anonymous auth
>         by self write
>         by * none
>
> access to dn=".*,customerRoot=([^,]+),o=customers,dc=example,dc=com"
>         by dn="userlogin=[^,]+,o=admins,dc=example,dc=com" write
>         by
> dn="userlogin=[^,]+,customerRoot=$1,o=customers,dc=example,dc=com" write
>         by * none
>
> access to dn="o=customers,dc=example,dc=com"
>         by dn="userlogin=[^,]+,o=admins,dc=example,dc=com" write
>         by * none
>
> access to dn=".*,customerRoot=([^,]+),o=[^,]+,dc=example,dc=com"
>         attr=userPassword
>         by dn="userlogin=[^,]+,o=admins,dc=example,dc=com" write
>         by
> dn="userlogin=[^,]+,customerRoot=$1,o=customers,dc=example,dc=com" write
>         by anonymous auth
>         by self write
>         by * none
>
> access to dn=".*,customerRoot=([^,]+),o=[^,]+,dc=example,dc=com"
>         by dn="userlogin=[^,]+,o=admins,dc=example,dc=com" write
>         by
> dn="userlogin=[^,]+,customerRoot=$1,o=customers,dc=example,dc=com" write
>         by * read
>
> access to *
>         by dn="userlogin=[^,]+,o=admins,dc=example,dc=com" write
>         by * read

I note that most of those ACLs are invalid for 2.2.X, while you claim you
tested both 2.1.X and 2.2.X with comparable results.  Valid corresponding
ACLs for 2.2 would be

access to dn.children="o=admins,dc=example,dc=com"
        attr=userPassword
        by anonymous auth
        by self write

access to dn="o=admins,dc=example,dc=com"
        by dn.regex="^userlogin=[^,]+,o=admins,dc=example,dc=com$$" read

access to dn.regex=".+,customerRoot=([^,]+),o=customers,dc=example,dc=com$"
        attr=userPassword
        by dn.regex="^userlogin=[^,]+,o=admins,dc=example,dc=com$$" write
        by
dn.regex="^userlogin=[^,]+,customerRoot=$1,o=customers,dc=example,dc=com$$"
write
        by anonymous auth
        by self write

access to dn.regex="^.+,customerRoot=([^,]+),o=customers,dc=example,dc=com$"
        by dn.regex="^userlogin=[^,]+,o=admins,dc=example,dc=com$$" write
        by
dn.regex="^userlogin=[^,]+,customerRoot=$1,o=customers,dc=example,dc=com$$"
write

access to dn.exact="o=customers,dc=example,dc=com"
        by dn.regex="^userlogin=[^,]+,o=admins,dc=example,dc=com$$" write

access to dn.regex="^.+,customerRoot=([^,]+),o=[^,]+,dc=example,dc=com$"
        attr=userPassword
        by dn.regex="^userlogin=[^,]+,o=admins,dc=example,dc=com$$" write
        by
dn.regex="^userlogin=[^,]+,customerRoot=$1,o=customers,dc=example,dc=com$$"
write
        by anonymous auth
        by self write

access to dn.regex="^.+,customerRoot=([^,]+),o=[^,]+,dc=example,dc=com$"
        by dn.regex="^userlogin=[^,]+,o=admins,dc=example,dc=com$$" write
        by
dn.regex="^userlogin=[^,]+,customerRoot=$1,o=customers,dc=example,dc=com$$"
write
        by * read

access to *
        by dn.regex="^userlogin=[^,]+,o=admins,dc=example,dc=com$$" write
        by * read

As I suspected you're overly using regex matching where not structly
required.  For instance, in half of your "by" clauses, all you really need
to do is replace the "customerRoot" distinguished value and ensure that
the identity's naming uses "userlogin" as naming attribute.  If the latter
can be relaxed (if you do not expect other identities to show up rooted
exactly at "customerRoot=$1,o=customers,dc=example,dc=com") you can rather
use

access to ...
        by
dn.onelevel,expand="customerRoot=$1,o=customers,dc=example,dc=com"
<level>

In the other half, you don't even need the expansion, so:

access to ...
        by dn.onelevel="o=admins,dc=example,dc=com" <level>

This would really save __A_LOT__ when regular users or even admins are
accessing the DSA.

>
>
>>>and  when I do this:
>>>
>>>ldapsearch -x -b"o=mail,dc=example,dc=com" -w password
>>>-D"userlogin=user,o=admins,dc=example,dc=com" -LLL
>>>"(&(mail=*@example.com*)(objectClass=mailUser))"
>>>
>>>the query takes about 0.05 seconds.
>>>
>>>
>>
>>Again, what indices did you use?
>>
>>
>
>>>I have tried:
>>>- sizelimt size.unckecked=-1
>>>- sizelimit size.hard=-1
>>>
>>>
>>
>>It is not clear what limits have to do with your problem.  If limits are
>>hit, either the search doesn't occur or only a fraction of the expected
>>results is returned.  I don't think this will solve your problem; only,
>>they should help in alleviating your server's load for those undesired
>>operations.  I note that sizelimit.unchecked=-1 is the default, so
>> setting
>>it is useless unless previouly overriden by any other directive.  Also,
>>size.hard=-1 is only useful if you specify a sizelimit request with your
>>search (e.g. you use the -z flag with ldapsearch).
>>
>>
>>
>>>but query 1 still takes about 30 seconds (with a cpu load of 99%)
>>>
>>>and query 3 is totally strange....
>>>
>>>
>>
>>what do you mean with "strange"?  Is it just faster than search 1?
>>
>>
> why should a search with more wildcards be quicker...

well, "*xyz" is different from "*xyz*" because the former activates
"final" subindexing, while the latter activates "any" substrings indexing.
 I'm unable to infer, at the moment, why one should be faster than the
other by simply speculating on the numbers you report, but at least I'm
not totally surprised of seeing a difference.

>
> and even more strange the 'slowness' is not on every domain searched,
> there are domains where the results are returned quickly with query 1.
>
> I have tried this with 1 ACL and the problem is still there....

If by "1 ACL" you mean the only one that is activated by your search, then
we're back to the ACL problem.  But in your case I rather suspect some
indexing problem, in the sense that for one search (the "final" one)
you're not hitting the ALLID limit, so indices are used, while in the
other (the "any" case) you're, so indices are not used.  I suggest you
carefully recheck how things behave in 2.2.X and, if the problem persists,
try an alternative way of filtering data, e.g. adding a "mail domain"
attribute for exact searches, or anything like that.

p.

-- 
Pierangelo Masarati
mailto:pierangelo.masarati@sys-net.it


    SysNet - via Dossi,8 27100 Pavia Tel: +390382573859 Fax: +390382476497