Brian G. Merrell wrote:http://symas.com/mdb/doc/
Hi all,
First, I'm having trouble finding resources to answer a question like this
myself, so please forgive me if I've missed something.
Your reader process should be using read transactions.Â
I'm considering using LMDB (versus LevelDB) for a project I'm working on where
I'll be receiving a high volume (hundreds per second) of high priority
requests (over HTTP) and issuing multiple (<10) database queries per request.
I'll also have a separate process receiving updates for the data and writing
to the database. ÂThis will happen often (several times a minute, perhaps),
but the priority is much lower than the read requests.
LMDB appealed to me because of the read performance and that I could have one
processing reading data from LMDB and another process writing data updates to
LMDB.
For proof of concept, I hacked up the following (I'll use pseudocode since I
used the Go bindings for my actual programs, and hopefully my question is
sufficiently abstract not to matter):
Process 1, the writer, simply writes a random integer (from 0 to 1000) to a
defined set of keys:
env = NewEnv()
env.Open("/tmp/foo", 0, 0664)
txn = env.BeginTxn(nil, 0)
dbi = txn.DBIOpen(nil, 0)
txn.Commit()
txn = env.BeginTxn(nil, 0)
n_entries = 5
for i = 0; i < n_entries; i++ {
  Âkey = sprintf("Key-%d", i)
  Âval = sprintf("Val-%d", rand.Int(1000))
  Âtxn.Put(dbi, key, val, 0)
}
txn.Commit()
env.DBIClose(dbi)
env.Close()
Process 2, the reader, simply loops forever and does random access reads on
the data from process 1 (I won't benefit from a cursor for my actual problem),
and prints out that data occasionally:
env = NewEnv()
env.Open("/tmp/foo", 0, 0664)
while {
  Âtxn = BeginTxn(nil, 0)
  Âdbi = txn.DBIOpen(nil, 0)
  Âtxn.Commit()
  Âfor i = 0; i < n_entries; i++ {
    Âkey = sprintf("Key-%d", i)
    Âval = txn.Get(dbi, key)
    Âprint("%s: %s", key, value)
  Â}
  Âenv.DBIClose(dbi)
  Âsleep(5)
}
So my high level question is: What am I doing wrong? ÂThis seems to work OK,
but a lot of it was guesswork, so I'm sure I'm doing some silly things.
In the actual LMDB API read transactions can be reused by their creating thread, so they are zero-cost after the first time. I don't know if any of the other language wrappers leverage this fact.
For example, first I put the BeginTxn() and DBIOpen() calls in process 2
outside of the while loop, but when I did that, I never saw the updates values
upon running process 1 simultaneously. ÂIn my real-world application, it seems
like adding these calls to every request (to be sure the data being read is
up-to-date) could be an unnecessary performance penalty.
Opening a DBI only needs to be done once per process. Opening per transaction would be stupid, like reopening a file handle on every request.
--
I was suspect there are flags that I can/should be using, but I'm not sure.
Thanks for any input.
    ÂBrian
 -- Howard Chu
 CTO, Symas Corp.      http://www.symas.com
 Director, Highland Sun   http://highlandsun.com/hyc/
 Chief Architect, OpenLDAP Âhttp://www.openldap.org/project/