Re: ldapsearch hangs

[Quanah Gibson-Mount <quanah@zimbra.com>]

--On Thursday, September 10, 2009 8:56 PM +0200 Tihomir Culjaga 
<tculjaga@gmail.com> wrote:

> So, the situation is that i have 2 ldif files i'm recreating the database
> from.
>
> /usr/local/libexec/slapadd -l /home/tculjaga/file2.ldif -f
> /usr/local/etc/openldap/slapd.conf
> /usr/local/libexec/slapadd -l /home/tculjaga/file2.ldif -f
> /usr/local/etc/openldap/slapd.conf

I would suggest you just make these a single file, so all the work can be 
done at one time.

> I tried to re-index with /usr/local/libexec/slapindex -f
> /usr/local/etc/openldap/slapd.conf -v
> restart slapd process, restart the machine ... it is always the same
> issue.

Nothing here indicates a problem with your indices.  Running slapindex 
repeatedly is a waste of your time.

> [root@l01lnp2 traces]# /usr/local/libexec/slapd -V
> @(#) $OpenLDAP: slapd 2.4.16 (Sep  9 2009 14:39:44) $
>     root@l01lnp2:/home/tculjaga/openldap-2.4.16/servers/slapd

I would strongly urge you to upgrade to 2.4.18 (for reasons I will note 
further down)


> [root@l01lnp2 traces]# /usr/local/BerkeleyDB.4.7/bin/db_stat -V
> Berkeley DB 4.7.25: (May 15, 2008) - unpached!

You need to rebuild BDB 4.7.25 with the 4 patches from Oracle.  There are 
known issues when running BDB 4.7 without them.

> [root@l01lnp2 traces]# du -c -h /var/lib/ldap/*.bdb
> 200K    /var/lib/ldap/bestMatchPrefix.bdb
> 3.8G    /var/lib/ldap/dn2id.bdb
> 6.2G    /var/lib/ldap/id2entry.bdb
> 1.8M    /var/lib/ldap/objectClass.bdb
> 1.2M    /var/lib/ldap/originatorPrefixID.bdb
> 48M    /var/lib/ldap/uniqueID.bdb
> 10G    total

Since your database is a total of 10 GB in size, for slapadd to work at 
optimum efficiency, you need at least 10GB of cache for your DB_CONFIG 
file.  Unfortunately, you only have 10GB of RAM.  Essentially, your system 
is under powered for your database size.



> [tculjaga@l01lnp2 ~]$ cat ot.ldif | grep -c "dn: "
> 101588
> [tculjaga@l01lnp2 ~]$ cat l01sipdir1.ldif | grep -c "dn: "
> 9994864
> [tculjaga@l01lnp2 ~]$

So you have 10,096,452 entries total.

> [root@l01lnp2 traces]# cat /var/lib/ldap/DB_CONFIG | grep -v "#"
>
> set_cachesize 0 3221225472 1
> set_lg_regionmax 262144
> set_lg_bsize 2097152

You only have a 3GB DB cachesize configured here.  Expect things to perform 
sub optimally.  It would have been easier to set this by going

set_cachesize 3 0 1

Which would have the same effect, since the first number is the number of 
gigabytes to allocate.

> Please find attached slapd.conf

Ok, so the relevant bits from here are:

cachesize 2500000
idlcachesize 7500000
cachefree 1000

Which means you have a cachesize of 2.5 million, an idlcachesize of 7.5 
million, and (with OL 2.4.16) a dncachesize of 5 million.

I would highly advise you upgrade to OpenLDAP 2.4.18, and change the 
slapd.conf settings to:

dncachesize 0 (which means unlimited).

And setting no cache or idlcachesize, and fixing your DB_CONFIG.  But you 
also need to buy a substantial amount of RAM for a DB of this size. :P  I 
would advise you upgrade to at least 32GB total.  Then you can more 
optimally tune the system.

--Quanah

--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra ::  the leader in open source messaging and collaboration