[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: back-mdb status



Howard Chu wrote:
A couple new results for back-mdb as of today.

			first		second		slapd size
back-hdb, 10K cache	3m6.906s	1m39.835s	7.3GB
back-hdb, 5M cache	3m12.596s	0m10.984s	46.8GB
back-mdb		0m19.420s	0m16.625s	7.0GB
back-mdb		0m15.041s	0m12.356s	7.8GB

Next, the time to execute multiple instances of this search was measured,
using 2, 4, 8, and 16 ldapsearch instances running concurrently.
				average result time
		2		4		8		16
back-hdb, 5M	0m14.147s	0m17.384s	0m45.665s	17m15.114s
>> back-mdb	0m16.701s	0m16.688s	0m16.621	0m16.955s
> back-mdb	0m12.009s	0m11.978s	0m12.048s	0m12.506s

This result for back-hdb just didn't make any sense. Going back, I discovered that I'd made a newbie mistake - my slapd was using the libdb-4.7.so that Debian bundled, instead of the one I had built in /usr/local/lib. Apparently my LD_LIBRARY_PATH setting that I usually have in my .profile was commented out when I was working on some other stuff.

While loading a 5 million entry DB for SLAMD testing, I went back and rechecked these results and got much more reasonable numbers for hdb. Most likely the main difference is that Debian builds BDB with its default configuration for mutexes, which is a hybrid that begins with a spinlock and eventually falls back to a pthread mutex. Spinlocks are nice and fast, but only for a small number of processors. Since they use atomic instructions that are meant to lock the memory bus, the coherency traffic they generate is quite heavy, and it increases geometrically with the number of processors involved.

I always build BDB with an explicit --with-mutex=POSIX/pthreads to avoid the spinlock code. Linux futexes are decently fast, and scale much better as the number of processors goes up.

With slapd linked against my build of BDB 4.7, and using the 5 million entry database instead of the 3.2M entry database I used before, the numbers make much more sense.

slapadd -q times
back-hdb	real 66m09.831s  user 115m52.374s  sys 5m15.860s
back-mdb	real 29m33.212s  user  22m21.264s  sys 7m11.851s

ldapsearch scanning the entire DB
		first		second		slapd size	DB size
back-hdb, 5M	4m15.395s	0m16.204s	26GB		15.6GB
back-mdb	0m14.725s	0m10.807s	10GB		12.8GB

multiple concurrent scans
 				average result time
 		2		4		8		16
back-hdb, 5M	0m24.617s	0m32.171s	1m04.817s	3m04.464s
back-mdb	0m10.789s	0m10.842s	0m10.931s	0m12.023s

You can see that up to 4 searches, probably the BDB spinlock would have been faster. Above that, you need to get rid of the spinlocks. If I had realized I was using the default BDB build I could of course have configured the BDB environment with set_tas_spins in the DB_CONFIG file. We used to always set this to 1, overriding the BDB default of (50 * number of CPUs), before we just decided to omit them entirely at configure time.

But I think this also illustrates another facet of MDB - reducing config complexity, so there's a much smaller range of mistakes that can be made.

re: the slapd process size - in my original test I configured a 32GB BDB environment cache. This particular LDIF only needed an 8GB cache, so that's the size I used this time around. The 47GB size reported was the Virtual size of the process, not the Resident size. That was also a mistake, all of the other numbers are Resident size.

When contemplating the design of MDB I had originally estimated that we could save somewhere around 2-3x the RAM compared to BDB. With slapd running 2.7x larger with BDB than MDB on this test DB, that estimate has been proved to be correct.

The MVCC approach has also proved its value, with no bottlenecks for readers and response time essentially flat as number of CPUs scales up.

There are still problem areas that need work, but it looks like we're on the right track, and what started out as possibly-a-good-idea is delivering on the promise.

--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/