[Date Prev][Date Next] [Chronological] [Thread] [Top]

Incremental backup in LMDB



A preliminary branch has been available on gitorious for a few months. It has some issues though, which still need to be resolved.

https://gitorious.org/mdb/mdb/source/a7fc32c0b39c29f9afd2172338060b079d549c59:

The biggest problem is that scanning every page of the DB is unsafe - if a page was an overflow page, and was freed and partially reused, then a scan may walk into intermediate chunks of data (which won't have valid page headers, although the data might coincidentally appear to have them).

Two approaches to solving this are to perform the scan, but ignore pages that are currently listed in the freeDB, or to do a regular tree walk instead. At the moment I'm leaning toward just walking the trees. If the freeDB is large then comparing every scanned page to the freeDB could be expensive. Also, newer pages of the freeDB itself must be copied as well, and this would mean freeDB pages get read twice.

Just thinking out loud here to document what we discussed before in IRC, so it doesn't get lost in the meantime.

--
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/