[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Re: back-mdb status
Howard Chu wrote:
A bit of a summary of how the backend is shaping up. I've been testing with a
variety of synthetic LDIFs as well as an actual application database (Zimbra
accounts).
I noted before that back-mdb's write speeds on disk are quite slow. This is
because a lot of its writes will be to random disk pages, and also the data
writes in a transaction commit are followed by a meta page write, which always
involves a seek to page 0 or page 1 of the DB file. For slapadd -q this effect
can be somewhat hidden because the writes are done with MDB_NOSYNC specified,
so no explicit flushes are performed. In my current tests with synchronous
writes, back-mdb is one half the speed of back-bdb/hdb.
(Even in fully synchronous mode, BDB only writes its transaction logs
synchronously, and those are always sequential writes so there's no seek
overhead to deal with.)
With that said, slapadd -q for a 3.2M entry database on a tmpfs:
back-hdb: real 75m32.678s user 84m31.733s sys 1m0.316s
back-mdb: real 63m51.048s user 50m23.125s sys 13m27.958s
On an XFS partition, the same job took
back-hdb real 80m34.403s user 86m2.439s sys 1m39.662s
back-mdb real 85m48.598s user 49m40.606s sys 14m48.668s
Note that back-hdb runs a trickle-sync thread to flush dirty DB pages in the
background, which is why its user time is greater than the real time.
back-mdb actually completed the load in 64m16.19s according to slapadd's
progress meter. But back-mdb performs an mdb_env_sync() on close, and that
sync took the remaining 21 minutes. (back-hdb pretty much does the same, it
does a checkpoint on close, but it skips it in Quick mode. So to be
apples-to-apples, back-mdb's final sync should have been omitted as well.)
For back-hdb, BDB was configured with a 32GB environment cache. The resulting
DB directory consumed 14951004KB including data files and environment files.
For back-mdb, MDB was configured with a 32GB mapsize. The resulting DB
directory consumed 18299832KB. The input LDIF was 2.7GB, and there were 29
attributes indexed. Currently MDB is somewhat wasteful with space when dealing
with the sorted-duplicate databases that are used for indexing, there's
definitely room for improvement here.
Also this slapadd was done with tool-threads set to 1, because back-mdb only
allows one writer at a time anyway. There is also obviously room for
improvement here, in terms of a bulk-loading API for the MDB library.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/