Mark Zealey wrote:
I'm not doing *any* commits just one big txn for all the data...
The below C works fine up until i=4m (ie 500mb of residential memory
shown in top), then has massive slowdown, shared memory (again, as seen
in top) increases, waits about 20-30 seconds and then disks get hammered
writing 10mb/sec (200txns) when they are capable of 100-200mb/sec
streaming writes... Does it do the same for you?
...
Kyoto writes async by default. You should do the same here, use
MDB_NOSYNC on the env_open.
MDB_NOSYNC makes no difference in my test case above - seeing exactly
the same memory, speed and disk patterns. Are you able to reproduce it?
Yes, I see it here, and I see the problem. LMDB was not originally designed to
handle transactions of unlimited size. It originally had a txn sizelimit of
about 512MB. In 0.9.7 we added some code to raise this limit, and it's
performing quite poorly here. I've tweaked my copy of the code to alleviate
that problem but your test program still fails here because the volume of data
being written also exceeds the map size. You were able to run this to completion?