Re: slapadd performance degradation from 2.3.43 to 2.4.12

[Ralf Haferkamp <rhafer@suse.de>]

Am Freitag 21 November 2008 14:21:06 schrieb Ralf Haferkamp:
> Hi,
>
> While doing a few slapadd testruns comparing the RE23 with the RE24 version
> I ran into a strange issue. I ran test with different LDIFs (100k, 500k and
> 1000k Entries) and especially with the 500k and 1000k LDIF, slapadd from
> 2.3.43 was significantly faster than the 2.4.12 version.
>
> 2.3.43 loaded the 500k database in 13m54s while it took 33m49s with 2.4.12.
> The 1000k testcase to 29m41s on 2.3 (still faster than the 500k on 2.4). I
> didn't finish the 2.4 run with the 1000k database I stopped it after about
> an hour.
>
> I used exactly the same configuration on exactly the same hardware/os for
> the tests (a HP Proliant DL 580 G3 with 4 3.33GHz Xeons, 8GB of RAM,
> SLES10-SP2, ext3 filesystem). BerkelyDB Version was 4.5.20 with the
> following DB_CONFIG:
>
> set_cachesize 0 4294967295 1
> set_lg_regionmax 262144
> set_lg_bsize 2097152
>
> cachesize in slapd.conf was large enough to hold the entire database and
> tool- threads was set to 4.
>
> I did some profiling (with valgrinds callgrind tool) to find out where all
> the time is spend and it revealed that 2.4 spend a significantly larger
> amount of systime in the pwrite() function than 2.3. Most of that seemed to
> come from the bdb_tool_trickle_task() that calls libdb's memp_trickle()
> function. Just to verify this I ran a testbuild with the trickle_task
> disabled(). And slapadd's performance was back to a normal level,
> comparable to the 2.3.43 release.
>
> AFAIK the trickle_task() was introduced into 2.4 to increase slapadd
> throughput but has exactly the opposite effect on my test system. Did
> anybody else make similar experiences? Or do you see anything thats
> obviously wrong with my testcases?

Btw, I just re-ran the test with an unmodified 2.4.12 compiled against 
db-4.7.25. With that combination slapadd is still a bit slower than the 
OpenLDAP 2.3/db-4.5.20 combination, but the difference not quite as huge: 
16m8s compared to 13m54s for the 500k Entries. 

-- 
regards,
	Ralf