[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Re: shared memory vs memory mapped files
--On Wednesday, February 18, 2009 12:26 AM +1000 "Brett @Google"
<brett.maxfield@gmail.com> wrote:
Hello,
Does anyone have any opinions on the merits of bdb/hdb backends, in
conjunction with using shared memory (adding shm_key <n> to a hdb or bdb
backend) vs., in comparison with plain-old memory mapped files ?
See below.
Shared memory does not seem to write any transactional bdb logs to the
disk, so i was wondering how robust it is with respect to recovery in the
face if improper or unexpected server shutdowns, when compared to memory
mapped files, whih can be recovered with db_recover to get back to a
consistant state ?
I assume shared memory either does not have bdb transaction logs, or they
are stored in memory so would be lost in the event of a server crash.
No. When you use slapadd -q, it turns off the transaction logs past the
first log file. SHM still writes the log files out to disk once the server
is up and running.
I note the benchmark files from Symas use memory mapped files, so i'm
assuming there is a clear performance benefit, is this true ?
Yes.
So if there is a clear query performance or other benefit, is there any
additional risk of corruption over memory mapped files (vs. the 25% load
speed improvement) ?
I recently did a test where I loaded 10 million entries (of zimbra users)
into a system with only 16GB of memory, giving BDB 4.7.25+patches only 12GB
of memory to use. The resulting DB was 22GB in size, and it only took 6
hours 15 minutes to load the entire thing. This was using shm. I haven't
yet had a chance to go back and use it without an shm key, but I'm pretty
certain it would take substantially longer.
When I did those tests for Symas, shm keys at the time made a substantial
difference on Solaris only. That was with BDB 4.2 & OpenLDAP 2.3 however.
With BDB 4.7.25 & OpenLDAP 2.4, they seem to make an impressive difference
on Linux as well.
As a side note, a 3 million user database on OpenLDAP 2.3 with BDB 4.2.52
on a 16GB server where the DB fits entirely into the BDB cache took 15
hours without SHM.
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration