[Date Prev][Date Next] [Chronological] [Thread] [Top]

Keeping mdb files opened while mmapped



Hey,
I am using lmdb as a backend to a big database (and its awesome, thanks!). While trying to understand memory consumption of my process I came across something interesting I'd like to get an expert's opinion about. 

I am seeing that the "Virtual" memory is quite large, larger than I expected, and it *might* be a problem in my case, I am being hit with lots of page ins/outs (I have a 500GB RAM machines). 

When I lsof my process, I am seeing that each mdb file is appearing twice (I use many), eg:
_ lsof  -p 59709  | grep 86510580
test    59709 vaknin  mem    REG               8,17  625500160 86510580 /fs/test1.mdb
test    59709 vaknin  239r   REG               8,17  625500160 86510580 /fs/test1.mdb

I know that after mmaping a file, it is not needed to be kept open, and it seems it is in lmdb. I then tried to see if this actually uses memory, and I used memstat for this:
_ memstat -p 59709 -w | grep 86510580
1221680k(1221680k): [08:11]:86510580 59709

From this it seems that the file is indeed using twice as much (virtual) memory as it should.. 2*625500160  1221680000

(this test was repeated for many different files, all have a factor of ≈2)

I know this might simply be an artifact and might not actually be a contributing reason for my swap ins/outs, but I wanted to hear what do you think about it? Why does lmdb keep the file handles open? 

By the way the swap ins/outs are happening later in the process, after querying lmdb and processing its results. 

Thanks!