search / add operations suddenly became hopelessly slow

["Alexei Monastyrnyi" <alexeim@orcsoftware.com>]

Hi List!
I've got a problem with search / add operations on OpenLDAP server (using
version 2.1.25 + BDB 4.1.25 on Solaris 9).

The following operation with result in 20 lines takes 15 sec... which took
ten times less before.

/usr/local/bin/ldapsearch -LLL -s one -b "ou=Group,dc=orcsoftware,dc=com"
"(cn=*)" dn | awk '{ print $2 }' > group.1~

LDAP DB is not that big, maximum 500 items in ou=People, all the rest is
much smaller...

I tried to reindex DB, still waiting for results... slapindex is being
running for 4 hours on such a small DB. truss shows that slapindex is doing
something with index files in openldap-data dir.

I have the same OpenLDAP configuration on different server with the same DB
and it works like a dream.

It worked perfectly before but one day it suddenly became slow. Nothing
critical was done no the system at that particular moment (recommended
patch_cluster was installed several days earlier but logs showed that was OK
with search/add operations timing several days after that).

Turning on loglevel to -1 I can see big delay (7 sec) during the search
operation above.

...
Mar 24 15:40:32 ra slapd[23392]: [ID 934353 local4.debug] =>
bdb_filter_candidat
es
Mar 24 15:40:32 ra slapd[23392]: [ID 873669 local4.debug]       PRESENT
Mar 24 15:40:32 ra slapd[23392]: [ID 578218 local4.debug] =>
bdb_presence_candid
ates (cn)
Mar 24 15:40:33 ra slapd[23392]: [ID 336836 local4.debug] => key_read
Mar 24 15:40:33 ra slapd[23392]: [ID 187165 local4.debug] bdb_idl_fetch_key:
[00
000001]
!!!!!!!!!!!!!!!! delay !!!!!!!!!!!!!!!!
Mar 24 15:40:40 ra slapd[23392]: [ID 766657 local4.debug] <= bdb_index_read
1590
 candidates
Mar 24 15:40:40 ra slapd[23392]: [ID 366904 local4.debug] <=
bdb_presence_candid
ates: id=1590 first=9 last=23897
Mar 24 15:40:40 ra slapd[23392]: [ID 730587 local4.debug] <=
bdb_filter_candidat
es: id=1590 first=9 last=23897
...

I think there is some issue with database but cannot understand what is
wrong...

Any hint would be highly appreciated.

Sincerely,
Alexei Monastyrnyi,
System Engineer,
Orc Software AB.
Birger Jarlsgatan 32A Box 7742   Reception   +46 8 407 38 00
SE-103 95 Stockholm Sweden      Fax              +46 8 407 38 01
www.orcsoftware.com                 Direct          +46 8 407 38 02
alexeim@orcsoftware.com           Mobile        +46 70 377 5777