Re: (ITS#7841) high disk utilization

[leo@yuriev.ru]

The attached patch file is derived from OpenLDAP Software. All of the
modifications to OpenLDAP Software represented in the following
patch(es) were developed by Leonid Yuriev <leo@yuriev.ru>. I have not
assigned rights and/or interest in this work to any party.

The attached modifications to OpenLDAP Software are subject to the
following notice:

Copyright 2014 Leonid Yuriev.
Copyright 2014 Peter-Service LLC, Moscow, Russia.
Redistribution and use in source and binary forms, with or without
modification, are permitted only as authorized by the OpenLDAP Public
License.

https://github.com/leo-yuriev/openldap-lmdb-challenge/pull/1
or
https://github.com/leo-yuriev/openldap-lmdb-challenge/ branch master-devel

commit 841059330fd44769e93eb4b937c3ce42654fad6f
Author: Leo Yuriev <leo@yuriev.ru>
Date:   2014-09-20 07:16:15 +0400

     BUGFIX - lmdb: lock meta-pages in writemap-mode to avoid unexpected write,
               before the data pages would be synchronized.

     Without locking the meta-pages may be writen by OS before other data,
     in this case database would be inconsistent.

commit 6240c3350e8bd86337c7e41722cf6a38881f15e7
Author: Leo Yuriev <leo@yuriev.ru>
Date:   2014-09-12 01:32:13 +0400

     BUGFIX - lmdb: reordering of instructions which update the txn in
a meta-page.

     Without "volatile" or memory-barrier compiler may reorder instructions
     for update the "mm_txnid" field in meta-page in "writemap" mode.

     From the reader's point of view this cause a short
     time interval when the transaction is corrupted.

commit accef62de7fe5660f870f4c5da319a2a8098b2fb
Author: Leo Yuriev <leo@yuriev.ru>
Date:   2014-09-21 02:29:50 +0400

     BUGFIX - lmdb: 'volatile' to important fields, which
               may be updated by readers asynchronously.

     Without 'volatile' compiler may eliminate a mdb_find_oldest() calls.

commit bb83e03cf1b8bceee64550229c3becbdd5400680
Author: Leo Yuriev <leo@yuriev.ru>
Date:   2014-09-19 20:18:17 +0400

     FEATURE - lmdb-backend: support config for 'lifo' and 'coalesce' envflags.

commit 0c168d0e63ed78d13df3fc8a42f3667335678639
Author: Leo Yuriev <leo@yuriev.ru>
Date:   2014-09-20 10:13:28 +0400

     FEATURE - lmdb: MDB_LIFORECLAIM & MDB_COALESCE modes.

     Reclaim FreeDB in LIFO order - this is a main feature.
     Also aim to coalesce small FreeDFB records.

commit 8ddd63161aeb2689822d1a8d27385d62e4e341ae
Author: Leo Yuriev <leo@yuriev.ru>
Date:   2014-09-19 22:47:19 +0400

     BUGFIX - lmdb: properly sync meta-pages in mdb_sync_env().

     Meta-pages may be updated during data-syncing in mdb_sync_env(),
     in this case database would be inconsistent.

     Check-and-retry if lead txn-id changed during flushing data in
mdb_sync_env().

commit 908677f989588d06b9f00620576dea3c5c8675d7
Author: Leo Yuriev <leo@yuriev.ru>
Date:   2014-09-04 16:10:05 +0400

     FEATURE - lmdb-backend: support for "checkpoint kbytes" config-option.

commit 147f41a8110f28456bc32123bde86d47183f9c0a
Author: Leo Yuriev <leo@yuriev.ru>
Date:   2014-09-04 16:01:15 +0400

     FEATURE - lmdb: implementation of "checkpoint kbytes".

     Force flush when volume of the changes reached a configurable threshold.

commit fb82a0b688f4c31313d0790415feda8aaa18651c
Author: Leo Yuriev <leo@yuriev.ru>
Date:   2014-09-04 15:18:16 +0400

     CHANGE - lmdb-backend: checkpoint-interval in seconds instead of minutes.

commit fc409d89e0d9dde20f612e34c2a463c8a81ea000
Author: Leo Yuriev <leo@yuriev.ru>
Date:   2014-09-20 06:51:04 +0400

     EXTENSION - lmdb: more usefull info from mdb_stat tool.

commit ccc7da690ffbff440643295b945fdf7886f48c97
Author: Leo Yuriev <leo@yuriev.ru>
Date:   2014-09-05 00:19:16 +0400

     TRIVIA - lmdb: clean testdb-dir while "make test".