Re: LMDB use of sparse or non-sparse data file

[Quanah Gibson-Mount <quanah@zimbra.com>]

--On Monday, March 23, 2015 8:38 PM +0000 Howard Chu <hyc@symas.com> wrote:

> Geoff Swan wrote:
> > I had to duplicate an LMDB database for replication recently, and used
> > mdb_copy to do so. One server is using the original data.mdb database
> > (which is sparse)
> and the other is using the mdb_copy non-sparse data.mdb file.
>
> If you specified no special options, the file produced by mdb_copy is
> identical to the original - it will also be sparse if the original is.

Well, to be clear: While the DB is sparse, mdb_copy does drop the unused 
map space when using mdb_copy by default.

I.e., if I specified an 80GB maxsize on a 20MB db, then the database copy 
done via mdb_copy will be 20MB not 80GB.  I.e., it's still sparse, but the 
unused portion has been dropped.

[zimbra@zre-ldap003 db]$ ls -l
total 820
-rw-------. 1 zimbra zimbra 52710469632 Mar 23 14:50 data.mdb
-rw-------. 1 zimbra zimbra        8192 Mar 23 14:53 lock.mdb
[zimbra@zre-ldap003 db]$ mkdir -p /tmp/mdb/db
[zimbra@zre-ldap003 db]$ mdb_copy . /tmp/mdb/db
[zimbra@zre-ldap003 db]$ cd /tmp/mdb/db
[zimbra@zre-ldap003 db]$ ls -l
total 816
-rw-r-----. 1 zimbra zimbra 835584 Mar 23 14:54 data.mdb

I think this is the behavior they're referring to.  However, in my 
experience, after starting up slapd with an mdb_copy'd db, where sparse 
files are in use, the size will be set to whatever slapd's configured to 
use after slapd is started.  For example:

[zimbra@zre-ldap003 db]$ cd
[zimbra@zre-ldap003 ~]$ ldap stop
Killing slapd with pid 29463 done.
[zimbra@zre-ldap003 ~]$ cd data/ldap/mdb
[zimbra@zre-ldap003 mdb]$ mv db db.old
[zimbra@zre-ldap003 mdb]$ mv /tmp/mdb/db .
[zimbra@zre-ldap003 mdb]$ cd db
[zimbra@zre-ldap003 db]$ ls -l
total 816
-rw-r-----. 1 zimbra zimbra 835584 Mar 23 14:54 data.mdb
[zimbra@zre-ldap003 db]$ ldap start
Started slapd: pid 28079
[zimbra@zre-ldap003 db]$ ls -l
total 820
-rw-r-----. 1 zimbra zimbra 52710469632 Mar 23 14:55 data.mdb
-rw-------. 1 zimbra zimbra        8192 Mar 23 14:55 lock.mdb

If that is not being seen, then your configurations are not as identical as 
thought.

--Quanah

--

Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
--------------------
Zimbra ::  the leader in open source messaging and collaboration