Re: slapadd/bdb tuning newbie

[Howard Chu <hyc@symas.com>]

Gary LaVoy wrote:

> Ok, I admit it, when I go through the BDB docs, they are pretty daunting 
> to me regarding how to effectively "tune" openldap for various things.
>
> I imported about 750,000 records in to the db using slapadd. It took 
> over *52 hours* - so I am not tuning something right, but I am really 
> not sure what I set wrong or maybe didn't include.

I'm surprised you waited so long to let it finish. I suppose it's a good 
excuse to get other work done though...

> I am running openldap 2.1.22 and BDB 4.1.(latest) as this is what ships 
> currently on OS X 10.3 stock. Xserve 1.0 Gh Dual CPU with 2 Gb RAM.

That's too bad. There's easily 80 or so bug fixes between 2.1.22 and the 
current (2.1.30) release, countless fixes between 2.1.X and 2.2. Perhaps 
your vendor needs help staying up to date. (Oh, wait, you *are* the 
vendor. Hm...)

> Here is the slapd.conf entry for that DB that I set up (I have 3 
> separate DB in the file) (yes I need all those indexes)
>
> database bdb
> suffix "o=Apple"
> rootdn "cn=Manager,o=apple"
> directory /ngs/app/openldap2/apple_openldap/var/openldap-data/db/apple

> and here is the DB_CONFIG I used:
>
> (I picked these based on a first attempt to get it to be faster than 
> with no config since everything I have ever read says you better have one.)
>
> #
> # Set the database in memory cache size.
> #
>
> set_cachesize 0 52428800 0

Now that you've invested those 52 hours in loading this thing, did you 
look at db_stat to see how well this cache performed? Have you looked at 
the stats on the individual database files to see how many pages they 
each consume? It seems, given your 2GB RAM available, that the 52MB 
you've configured here is rather miserly. But without knowing the 
overall data volume you're dealing with, it's impossible to draw any 
conclusions.

> #
> # Set log values.
> #
> set_lg_regionmax 1048576
> set_lg_max 10485760
> set_lg_bsize 2097152
> set_lg_dir 
> /ngs/app/openldap2/apple_openldap/var/openldap-data/db/txlogs/apple

Have you actually mounted a separate disk under this directory path? As 
the docs state in several places, you need to put the logs on a 
physically separate spindle from the database files to get the best 
performance. Otherwise, just realize that however large your input data 
is, your single disk needs to handle double-to-triple that data volume 
because all of the data and all of the logs are going on the same disk, 
and there's a ton of seek overhead along with it all. Not even RAIDs 
perform well in the face of heavy random access patterns...

> set_tmp_dir /tmp

The BDB library doesn't use temporary files in back-bdb, so this setting 
is pointless.

--
  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support