Re: question about use of buffers in lmdb

[Luke Kenneth Casson Leighton <lkcl@lkcl.net>]

On Fri, Oct 17, 2014 at 6:54 PM, Howard Chu <hyc@symas.com> wrote:
> Luke Kenneth Casson Leighton wrote:
>>
>> howard, et al, hi,
>>
>> i notice that LMDB stores data in an "append only" manner (from the
>> Q&A).  i was thinking of recommending that in python the buffers be
>> handed around with *direct* and persistent pointers to the data within
>> the LMDB shm/memory-map.
>>
>> the reason that i believe this will be successful is because the
>> circumstances are rather unusual: once data is entered (added) it is
>> *never* overwritten, updated, changed, or modified.  there simply is
>> no need [in this application].  so there will only ever be insertion
>> (once and only once) or controlled and monitored deletion (by age of
>> record).
>>
>> under these circumstances is it perfectly reasonable to pass around
>> the pointers to the buffers returned from value look-ups
>> *indefinitely*?  if so that would be awesome.
>
>
> Any time a new value is inserted into an existing page, that page's address
> will change due to COW. Any time an old page gets sufficiently old that no
> other txn points to it, it becomes eligible for reuse - thus completely
> unrelated data could get onto it. So unless you're keeping very close tabs
> on your application's key+data sizes and know how full a page got, you can't
> rely on particular addresses staying valid.

 ok.  that makes sense.  a pity, though, because it would be reaally
cool to refer to data by buffer address forever.  COW means it's
beyond tiblmdb's control, but hey, i tried :)

 l.