[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Re: (ITS#9027) LMDB: API to retrieve map address
--000000000000856dab058992e756
Content-Type: text/plain; charset="UTF-8"
The short answer is yes. Our mappings are now bigger than 1TiB (out of 2TiB
total available). Also, it *seems* to be that it is just way faster for
Linux to map things in order all at once. Additionally, we have some boxes
that still have spinning disks, and sequential access is a couple orders of
magnitude faster on those.
I haven't looked at the logic in the Linux pager, but it seems to be fairly
aggressive about taking memory pages back from large memory maps, even
though it doesn't strictly need to. Locking the memory is a bit of a
sledgehammer, but in our experience it is an effective one.
On Thu, May 23, 2019 at 2:23 PM Howard Chu <hyc@symas.com> wrote:
> github@nicwatson.org wrote:
> > Full_Name: Nic Watson
> > Version:
> > OS: Linux
> > URL: ftp://ftp.openldap.org/incoming/
> > Submission from: (NULL) (73.132.68.128)
> >
> >
> > Goal:
> > I'd like a clean way to get at the address of the data memory map in
> LMDB.
> >
> > MDB_envinfo.me_mapaddr only returns the map address if MAP_FIXED is used.
> >
> > Current Workarounds:
> > * Use OS-specific mechanism to retrieve all memory maps (e.g.
> > /proc/<pid>/smaps).
> > * Defeat opaque handle and reach into the MDB_env struct directly and
> grab the
> > me_map field.
> >
> > Justification:
> > In my current application, I notice a significant performance increase
> if I
> > mlock the mapfile. In order to do that cleanly, I need the address of
> the map.
>
> That sounds odd. Is there a lot of memory pressure from other processes on
> the machine?
> Where is the performance loss or gain coming from?
>
> --
> -- Howard Chu
> CTO, Symas Corp. http://www.symas.com
> Director, Highland Sun http://highlandsun.com/hyc/
> Chief Architect, OpenLDAP http://www.openldap.org/project/
>
--000000000000856dab058992e756
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">The short answer is yes. Our mappings are now bigger than =
1TiB (out of 2TiB total available). Also, it *seems* to be that it is just =
way faster for Linux to map things in order all at once. Additionally, we h=
ave some boxes that still have spinning disks, and sequential access is a c=
ouple orders of magnitude faster on those.<div><br></div><div>I haven't=
looked at the logic in the Linux pager, but it seems to be fairly aggressi=
ve about taking memory pages back from large memory maps, even though it do=
esn't strictly need to. Locking the memory is a bit of a sledgehammer, =
but in our experience it is an effective one.</div></div><br><div class=3D"=
gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Thu, May 23, 2019 at =
2:23 PM Howard Chu <<a href=3D"mailto:hyc@symas.com";>hyc@symas.com</a>&g=
t; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0p=
x 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><a hre=
f=3D"mailto:github@nicwatson.org"; target=3D"_blank">github@nicwatson.org</a=
> wrote:<br>
> Full_Name: Nic Watson<br>
> Version: <br>
> OS: Linux<br>
> URL: <a href=3D"ftp://ftp.openldap.org/incoming/"; rel=3D"noreferrer" t=
arget=3D"_blank">ftp://ftp.openldap.org/incoming/</a><br>
> Submission from: (NULL) (73.132.68.128)<br>
> <br>
> <br>
> Goal:<br>
> I'd like a clean way to get at the address of the data memory map =
in LMDB.<br>
> <br>
> MDB_envinfo.me_mapaddr only returns the map address if MAP_FIXED is us=
ed.<br>
> <br>
> Current Workarounds:<br>
> * Use OS-specific mechanism to retrieve all memory maps (e.g.<br>
> /proc/<pid>/smaps). <br>
> * Defeat opaque handle and reach into the MDB_env struct directly and =
grab the<br>
> me_map field.<br>
> <br>
> Justification:<br>
> In my current application, I notice a significant performance increase=
if I<br>
> mlock the mapfile.=C2=A0 In order to do that cleanly, I need the addre=
ss of the map.<br>
<br>
That sounds odd. Is there a lot of memory pressure from other processes on =
the machine?<br>
Where is the performance loss or gain coming from?<br>
<br>
-- <br>
=C2=A0 -- Howard Chu<br>
=C2=A0 CTO, Symas Corp.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<a href=3D"=
http://www.symas.com"; rel=3D"noreferrer" target=3D"_blank">http://www.symas=
.com</a><br>
=C2=A0 Director, Highland Sun=C2=A0 =C2=A0 =C2=A0<a href=3D"http://highland=
sun.com/hyc/" rel=3D"noreferrer" target=3D"_blank">http://highlandsun.com/h=
yc/</a><br>
=C2=A0 Chief Architect, OpenLDAP=C2=A0 <a href=3D"http://www.openldap.org/p=
roject/" rel=3D"noreferrer" target=3D"_blank">http://www.openldap.org/proje=
ct/</a><br>
</blockquote></div>
--000000000000856dab058992e756--