[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: slapadd performance degradation from 2.3.43 to 2.4.12

Ralf Haferkamp wrote:
Am Freitag 21 November 2008 14:21:06 schrieb Ralf Haferkamp: 
Hi,

While doing a few slapadd testruns comparing the RE23 with the RE24 version
I ran into a strange issue. I ran test with different LDIFs (100k, 500k and
1000k Entries) and especially with the 500k and 1000k LDIF, slapadd from
2.3.43 was significantly faster than the 2.4.12 version.

2.3.43 loaded the 500k database in 13m54s while it took 33m49s with 2.4.12.
The 1000k testcase to 29m41s on 2.3 (still faster than the 500k on 2.4). I
didn't finish the 2.4 run with the 1000k database I stopped it after about
an hour.

I used exactly the same configuration on exactly the same hardware/os for
the tests (a HP Proliant DL 580 G3 with 4 3.33GHz Xeons, 8GB of RAM,
SLES10-SP2, ext3 filesystem). BerkelyDB Version was 4.5.20 with the
following DB_CONFIG:

set_cachesize 0 4294967295 1
set_lg_regionmax 262144
set_lg_bsize 2097152

cachesize in slapd.conf was large enough to hold the entire database and
tool- threads was set to 4.

I did some profiling (with valgrinds callgrind tool) to find out where all
the time is spend and it revealed that 2.4 spend a significantly larger
amount of systime in the pwrite() function than 2.3. Most of that seemed to
come from the bdb_tool_trickle_task() that calls libdb's memp_trickle()
function. Just to verify this I ran a testbuild with the trickle_task
disabled(). And slapadd's performance was back to a normal level,
comparable to the 2.3.43 release.

AFAIK the trickle_task() was introduced into 2.4 to increase slapadd
throughput but has exactly the opposite effect on my test system. Did
anybody else make similar experiences? Or do you see anything thats
obviously wrong with my testcases?


Btw, I just re-ran the test with an unmodified 2.4.12 compiled against
db-4.7.25. With that combination slapadd is still a bit slower than the
OpenLDAP 2.3/db-4.5.20 combination, but the difference not quite as huge:
16m8s compared to 13m54s for the 500k Entries.

Hmm. As I recall, the trickle task improved performance when the DB was much larger than the cachesize. The problem being, when the cache gets filled, without the trickle task your adds get stuck waiting for cache pagein/out to occur (synchronously), while with the trickle task the add threads can run unimpeded. Perhaps we need a config option to control this.

--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/