[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Re: LMDB and multiple processes
Brian G. Merrell wrote:
On Wed, Jun 4, 2014 at 10:22 AM, Howard Chu <hyc@symas.com
<mailto:hyc@symas.com>> wrote:
Brian G. Merrell wrote:
Hi all,
First, I'm having trouble finding resources to answer a question like this
myself, so please forgive me if I've missed something.
http://symas.com/mdb/doc/
Thanks. I did see and skim the API portion of the docs before asking, but I
was just having trouble knowing how the pieces fit together to solve a problem.
Skimming isn't going to cut it.
Your reader process should be using read transactions.
OK, I interpret this as meaning that I need to pass the MDB_RDONLY flag to
mdb_txn_begin. Is that correct?
Yes.
In the actual LMDB API read transactions can be reused by their creating
thread, so they are zero-cost after the first time. I don't know if any of
the other language wrappers leverage this fact.
This helps a lot. I will investigate what the case is with gomdb.
Opening a DBI only needs to be done once per process. Opening per
transaction would be stupid, like reopening a file handle on every request.
I suspected so. The fact that mdb_dbi_open takes a transaction had me
confused a bit, because I thought I would need to pass in the new transaction
every time I got a transaction from mdb_txn_begin.
mdb_dbi_open takes a txn because it needs one if you're creating a DB for the
first time. I.e., it must write metadata for the DB into the environment, and
all writes to MDB must be inside a txn. But once that txn is committed, the
DBI itself lives on until mdb_dbi_close. This is all already explained in the
doc for mdb_dbi_open; if you hadn't skimmed you would have seen it already.
Most of this is only a concern when you're using named subDBs. The default
unnamed DB always exists, so its DBI is always valid anyway.
I've refactored the reader to look like this:
env = NewEnv()
env.Open("/tmp/foo", 0, 0664)
txn = BeginTxn(nil, mdb.RDONLY) // parent txn is the nil arg
dbi = txn.DBIOpen(nil, 0)
txn.Abort()
You want mdb_txn_reset() here, not abort. Abort frees/destroys the txn handle
so it cannot be reused.
while {
txn = BeginTxn(nil, mdb.RDONLY) // parent txn is the nil arg
and here you want mdb_txn_renew(), to reuse the txn handle instead of creating
a new one.
for i = 0; i < n_entries; i++ {
key = sprintf("Key-%d", i)
val = txn.Get(dbi, key)
print("%s: %s", key, value)
}
txn.Commit()
and you want mdb_txn_reset() here too, not commit. Commit also frees/destroys
the txn handle.
sleep(5)
}
You can abort or commit the txn during your process teardown phase to dispose
of it.
env.DBIClose(dbi)
Now, I guess the big question that BeginTxn inside the loop is zero-cost.
Thanks for the tips so far Howard; it has been very helpful.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/