Re: LMDB - growing the database

[Howard Chu <hyc@symas.com>]

Shmulik Regev wrote:
> Hi,
>
> I understand that the DB size has an upper limit set by the call to
> mdb_env_set_mapsize . I wonder what is the best strategy for growing the size.

The best strategy is to initially pick a large enough size that growing it is 
never an issue. E.g., set it to the amount of free space on the disk partition 
where the DB resides. As the docs state, when the DB is actually in use, 
there's no guarantee that there will be enough free system resources left for 
a resize attempt to even succeed. I.e., if you initially choose a small size, 
by the time you need to deal with it it may be too late.

> From what I read, the put operations of either txn or cursor may fail with
> the MDB_MAP_FULL error but then it is too late to change the DB size. On the
> other hand I didn't find in mdb_env_stat (or perhaps I didn't understand) any
> information suggesting how full is the DB so I can't really implement any
> strategy for preemptively growing the DB based on the used space.
>
> Did I miss anything?

There are 2 sets of information you can use for an estimate.

1) MDB_stat gives you the page size.

2) MDB_envinfo tells the mapsize and the last_pgno. If you divide mapsize by 
pagesize you'll get max pgno. The MAP_FULL error is returned when last_pgno 
reaches max pgno.

This is also tempered by the contents of the freelist. The mdb_stat command 
can tell you how many pages are on the freelist. (mdb_stat -f). The DB isn't 
actually full until the freelist is used up, and last_pgno is maxed out.

--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/