Re: slapadd *very* slow: tuning advice?

[Quanah Gibson-Mount <quanah@zimbra.com>]

--On Wednesday, August 22, 2012 7:15 AM +1000 Nick Urbanik 
<nicku@nicku.org> wrote:

> Dear Quanah,
>
> Thank you for your generous help.
>
> In the biggest LDAP directory on the (now finally converted) machine:
> 6.1G

So for optimal *slapadd*, this should be your DB_CONFIG setting.  You can 
reduce it down to just the size of id2entry.bdb + dn2id.bdb + some room for 
growth (10-20% of that dn2id.bdb +id2entry.bdb)  after the slapadd 
completes.

So if id2entry.bdb is 1 GB and dn2id is 1 GB, I would probably use a 2.5 GB 
cachesize after slapadd completes.

> Over all eleven LDAP directories on that same machine:
> 13G

For running slapd, the DB_CONFIG sizes should match the above formula.

> > Right now, you only have a 272MB BDB cache set... which is definitely
> > going to be too small. I would change DB_CONFIG For that database to
> > be something like:
> >
> > set_cachesize 6 0 0
>
> Okay; with 12G total RAM, and 6.9G of other directories, am I likely
> to overcommit memory?

Not for the slapadd.  See above.

> > I would strongly advise examining using a shared memory key for that
> > DB as well.
>
> Okay: I could create that with a command like
> ipcmk -M 16777216

*No*.  Please see the shm_key parameter in the manual pages for slapd-bdb.

> > Finally, look at using MDB, which has zero tuning bits required, and
> > only requires a tool-threads of 2.
>
> So with eight cores (and 16 hyperthreaded siblings) I am better off
> specifying tool-threads 2?

No.  tool-threads is only used by slapadd.  Since no other databases are 
being accessed while you are performing the load, you would want all CPUs 
dedicated to this slapadd process.

> Okay, my next one I'll convert to mdb and see how that behaves.
>
> slapd-mdb(5) seems to indicate that mdb needs no shared memory
> configuration.   Is that correct?

Correct.  There is no tuning for MDB at all, other than:

a) Ensuring either noatime or relatime is set on the filesystem hosting the 
DB

b) Ensuring you have a maxsize value that is large enough.  This is for the 
maximum size you ever want the DB to grow, not a performance tuning value. 
It needs to be large enough to contain your DB and have room for growth. 
It cannot be larger than the size of the disk it is located on.

c) toolthreads for slapadd with mdb should be set to 2.  No more, no less.

--Quanah

--

Quanah Gibson-Mount
Sr. Member of Technical Staff
Zimbra, Inc
A Division of VMware, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration