Re: LDAp performance limites to 50 searchs /s?

[Buchan Milne <bgmilne@obsidian.co.za>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Leigh Porter wrote:
| Hiya Folks,
|
| OK, I have this LDAP tree running on 2.2.20 on Linux 2.6 on a AMD64
| 3200+ with about 2Gb of
| memory. There are around 3500 quite small entries, I am running  a
| simple search on multiple machines
| with the following:
|
| for i in `cat email`
| do
| ldapsearch  -LLL -h db1 -x  -b "dc=testdomain,dc=co,dc=uk"
| "(mail=$i@testdomain.co.uk)" mailMessageStore > /dev/null
| done
|
| I run say 5 of these on about 3 boxes just to give the LDAP box some
| traffic.

I did a very similar test (I guess you're also benchmarking for
qmail-ldap) a few months ago, with OpenLDAP-2.1.25->2.1.30 on bdb
backend with db-4.2.52.2, on Dell PowerEdge 1750's (IIRC 2.8GHz Xeon's)
with 2GB ram, with 250 000 entries in the directory. I ran 4 instances
of my test script on each of 3 machines, and consistently achieved 200
queries/sec. OpenLDAP-2.2.x should do quite a bit better ...

|
| The problem is that no matter how many instances of this test script I
| run, I only ever get about 50 searches
| per second form the LDAP server, I would have expected more..
|
| Thanks to the helpful comments from this list I have created a DB_CONFIG
| with the following:
|
| #
| # Set the database in memory cache size - get the whole thing in memory.
| #
| set_cachesize   0       102400000        0
| # Set database flags.
| set_flags       DB_TXN_NOSYNC

^
You should not do this except for mass loading via slapadd.

| # Set log values.
| set_lg_regionmax        1048576
| set_lg_max              10485760
| set_lg_bsize            2097152
| set_lg_dir              /usr/local/var/openldap-data/logs
| # Set temporary file creation directory.
| set_tmp_dir             /tmp
|
| The large cache is to get the whole database into  memory.
|
| Here is slapd.conf (relavent bits only)
|
| threads 100

You shouldn't need that many. You might be over-tuning here (+- 8MB
memory required per thread IIRC). IIRC I was running with 36.

| loglevel 0
|
| database        bdb
| index   objectClass     eq
| index mailMessageStore sub,eq,pres

^ Do you really have clients searching on the mailMessageStore that often?

| index mailServices sub,eq,pres

Your searches use a searchfilter of "mail=<string>", yet you aren't
indexing mail. You probably need something like:
index           mail                                eq,pres

| sizelimit unlimited

^
Bad idea ...

| cachesize   10000000

Caching 10 000 000 LDAP entries when you only have 3500 seems a bit
weird ...

| dbcachesize 10000000
^
This one is deprecated by the BDB configuration in DB_CONFIG

|
| This is what I get from checking the Search Operations Initiated on the
| monitor db:
| (never more than 50 searches/s if that)
|
| monitorOpInitiated: 4000
| monitorOpInitiated: 4044
| monitorOpInitiated: 4088
| monitorOpInitiated: 4132
| monitorOpInitiated: 4176
| monitorOpInitiated: 4220
| monitorOpInitiated: 4264
| monitorOpInitiated: 4309
| monitorOpInitiated: 4354
| monitorOpInitiated: 4396
| monitorOpInitiated: 4439
| monitorOpInitiated: 4484
|
| Assuming my attempt at a DB_CONFIG and the slapd config are OK, I am
| thinking that this may be a limitation
| somewhere els eon the box.. Can anybody help out with this please?

Index the attributes you use in your search filters ...

Regards,
Buchan

- --
Buchan Milne                      Senior Support Technician
Obsidian Systems                  http://www.obsidian.co.za
B.Eng                                RHCE (803004789010797)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFB9OEBrJK6UGDSBKcRAq62AJ0eFZrmo8FScfYK2qryqM+KMupopwCfZYdI
qt1Ij4ZpkRIhadiVEG0arJ8=
=mcER
-----END PGP SIGNATURE-----