[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Antw: LMDB performance tip
>>> Howard Chu <hyc@symas.com> schrieb am 06.11.2014 um 14:42 in Nachricht
<545B7AC9.3060207@symas.com>:
> Just recently measured: when doing a bulk-load of a DB that's larger than
> RAM,
> it's faster to turn off WRITEMAP and just use regular writes.
Doesn't that depend on the ratio of change compared to the size of RAM or DB?
>
> When the DB is much larger than RAM, and LMDB is reusing old pages, most
> likely the next page to be written will not currently be in memory. If you
> just access the mapped page (writing to it) the OS will have to page it in
> first. This is an unnecessary I/O operation since you're simply going to
This would mean LMDB has a copy of the mmapped file; otherwise the OS has to page in some memory to provide a buffer for the data to be written anyway. If you copy the mmapped file to/from private buffers, the main advantage of mmapping a file goes away.
> overwrite its contents anyway. If you do a regular write() from a buffer
> instead, the OS just writes it to the target page, no page-in required.
The OS still may have to allocate a buffer for the transfer, causing some paging activity.
>
> Strangely enough, this performance advantage disappears when under an active
>
> random read/write workload. I haven't yet worked out why that is. Perhaps
> the
> cost of multiple memcpy's comes into play.
Singe sync random writes, or async random writes? If sync writes, have you examined using writev() instead to write multiple blocks? When talking about writes and blocks: Do the blocks written match the blocksize and alignmant of the filesystem, and is the blocksize of the filesystem at least one physical sector of the medium, and are the blocks of the filesystem aligned with the medium (talking about now popular 4kB sectors)?
Regards,
Ulrich
> --
> -- Howard Chu
> CTO, Symas Corp. http://www.symas.com
> Director, Highland Sun http://highlandsun.com/hyc/
> Chief Architect, OpenLDAP http://www.openldap.org/project/