Re: LMDB - growing the database

[btb@bitrate.net]

On Feb 23, 2014, at 16.37, Nat! <nat@mulle-kybernetik.com> wrote:

> I wanted to resurrect this thread:
>>> 
>>> Hi,
>>> 
>>> I understand that the DB size has an upper limit set by the call to
>>> mdb_env_set_mapsize . I wonder what is the best strategy for growing the size.
>>> 
>> The best strategy is to initially pick a large enough size that growing it is never an issue. E.g., set it to the amount of free space on the disk partition where the DB resides. As the docs state, when the DB is actually in use, there's no guarantee that there will be enough free system resources left for a resize attempt to even succeed. I.e., if you initially choose a small size, by the time you need to deal with it it may be too late.
>> 
> I have a problem with the "best strategy" because at least on OS X, the reserved file size of "data.mdb" is just the amount set by mdb_env_set_mapsize. Clearly reserving the amount of free space in my user partition would not be desirable, especially if I want to run more than just one lmdb based program.
> 
> So I tried to grow my db progressively, by setting  mdb_env_set_mapsize first to a small value and the enlarging it. Unfortunately it seems, that whenever I increase the env_mapsize, all the committed data is lost and I start with a clean database (determined by mdb_stat).
> 
> So what gives ? The only thing I have come up with is, that I would need to copy the old database(s) into the new one with a cursor, but that sounds lame.

generally speaking, i’d discourage you from looking at that limit from the perspective of “how large will my data be?”.  instead, consider it a safeguard, for the os/environment.  evaluate your particular environment, and use values amongst your various instances such that, were something unexpected to happen, the entire disk/partition/etc is not consumed to the point of choking out the os [or perhaps other more important processes, etc].

-ben