[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Re: (ITS#8975) Calling mdb_env_set_mapsize on a shared DB fails sometimes
- To: openldap-its@OpenLDAP.org
- Subject: Re: (ITS#8975) Calling mdb_env_set_mapsize on a shared DB fails sometimes
- From: kriszyp@gmail.com
- Date: Thu, 21 Feb 2019 16:45:43 +0000
- Auto-submitted: auto-generated (OpenLDAP-ITS)
--0000000000000dacee05826a34a9
Content-Type: text/plain; charset="UTF-8"
Thank you, I appreciate it! Do you know how often or when this will get
into the github mirror https://github.com/LMDB/lmdb/commits/mdb.RE/0.9 (or
if I am misunderstanding how the branches are synced)?
On Sun, Feb 10, 2019 at 3:14 AM Howard Chu <hyc@symas.com> wrote:
> kriszyp@gmail.com wrote:
> > Full_Name: Kristopher William Zyp
> > Version: LMDB 0.9.23
> > OS: Windows Server 2012 R2, Windows 10
> > URL:
> https://github.com/kriszyp/node-lmdb/commit/6df903907f5516320e9a8afce45bd32ab4e8e1f2.patch
> > Submission from: (NULL) (71.199.6.148)
> >
>
> Thanks for the report and patch, added to mdb.RE/0.9
>
> > Calling mdb_env_set_mapsize to increase the map size, when a DB is also
> in use
> > by other processes, when MDB_WRITEMAP is enabled (and the db file has
> been
> > opened with PAGE_READWRITE access), on Windows, will occasionally (seems
> like
> > about 1/100 attempts fails) produce an error "The requested operation
> cannot be
> > performed on a file with a user-mapped section open", or segfaults. The
> error
> > occurs in the SetFilePointer (or SetEndOfFile) call in mdb_env_map that
> is
> > performed to increase the allocated file size to the map size, prior to
> > CreateFileMapping.
> >
> > As it turns out this is pretty easy to solve, because manually expanding
> the
> > file size is not necessary when calling CreateFileMapping with
> PAGE_READWRITE
> > access, as Windows will automatically expand the file size for us, when
> opened
> > with the page write access enabled. Of course, this means all processes
> must be
> > consistent in use of MDB_WRITEMAP, but the documentation already makes
> this
> > explicit and clear.
> >
> > I believe this can be fixed by simply adding a check for MDB_WRITEMAP in
> the if
> > statement that calls SetFilePointer:
> >
> > if (!(flags & MDB_WRITEMAP) && (SetFilePointer(env->me_fd,
> sizelo, &sizehi, 0)
> > != (DWORD)sizelo
> > || !SetEndOfFile(env->me_fd)
> > || SetFilePointer(env->me_fd, 0, NULL, 0) != 0))
> > return ErrCode();
> >
> > The attached URL has the change as a patch/diff as applied to our node
> package.
> >
> > I am certainly happy to just keep this change on our own branches. There
> may be
> > nuances of this that I might not be aware of, but it seems to be working
> great
> > for us and I have tested this with MDB_WRITEMAP enabled and disabled. So
> I
> > thought I would offer/suggest this change, as it seems like it is
> > straightforward change to improve stability. Thank you!
> >
> >
>
>
> --
> -- Howard Chu
> CTO, Symas Corp. http://www.symas.com
> Director, Highland Sun http://highlandsun.com/hyc/
> Chief Architect, OpenLDAP http://www.openldap.org/project/
>
--0000000000000dacee05826a34a9
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div dir=3D"ltr">Thank you, I appreciate it! Do you know h=
ow often or when this will get into the github mirror <a href=3D"https://gi=
thub.com/LMDB/lmdb/commits/mdb.RE/0.9">https://github.com/LMDB/lmdb/commits=
/mdb.RE/0.9</a> (or if I am misunderstanding how the branches are synced)?<=
/div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_a=
ttr">On Sun, Feb 10, 2019 at 3:14 AM Howard Chu <<a href=3D"mailto:hyc@s=
ymas.com">hyc@symas.com</a>> wrote:<br></div><blockquote class=3D"gmail_=
quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,=
204);padding-left:1ex"><a href=3D"mailto:kriszyp@gmail.com"; target=3D"_blan=
k">kriszyp@gmail.com</a> wrote:<br>
> Full_Name: Kristopher William Zyp<br>
> Version: LMDB 0.9.23<br>
> OS: Windows Server 2012 R2, Windows 10<br>
> URL: <a href=3D"https://github.com/kriszyp/node-lmdb/commit/6df903907f=
5516320e9a8afce45bd32ab4e8e1f2.patch" rel=3D"noreferrer" target=3D"_blank">=
https://github.com/kriszyp/node-lmdb/commit/6df903907f5516320e9a8afce45bd32=
ab4e8e1f2.patch</a><br>
> Submission from: (NULL) (71.199.6.148)<br>
> <br>
<br>
Thanks for the report and patch, added to mdb.RE/0.9<br>
<br>
> Calling mdb_env_set_mapsize to increase the map size, when a DB is als=
o in use<br>
> by other processes, when MDB_WRITEMAP is enabled (and the db file has =
been<br>
> opened with PAGE_READWRITE access), on Windows, will occasionally (see=
ms like<br>
> about 1/100 attempts fails) produce an error "The requested opera=
tion cannot be<br>
> performed on a file with a user-mapped section open", or segfault=
s. The error<br>
> occurs in the SetFilePointer (or SetEndOfFile) call in mdb_env_map tha=
t is<br>
> performed to increase the allocated file size to the map size, prior t=
o<br>
> CreateFileMapping.<br>
> <br>
> As it turns out this is pretty easy to solve, because manually expandi=
ng the<br>
> file size is not necessary when calling CreateFileMapping with PAGE_RE=
ADWRITE<br>
> access, as Windows will automatically expand the file size for us, whe=
n opened<br>
> with the page write access enabled. Of course, this means all processe=
s must be<br>
> consistent in use of MDB_WRITEMAP, but the documentation already makes=
this<br>
> explicit and clear.<br>
> <br>
> I believe this can be fixed by simply adding a check for MDB_WRITEMAP =
in the if<br>
> statement that calls SetFilePointer:<br>
> <br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (!(flags &=
; MDB_WRITEMAP) && (SetFilePointer(env->me_fd, sizelo, &size=
hi, 0)<br>
> !=3D (DWORD)sizelo<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0|| !SetEndOfFile(env->me_fd)<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0|| SetFilePointer(env->me_fd, 0, NULL, 0) !=3D 0))<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0return ErrCode();<br>
> <br>
> The attached URL has the change as a patch/diff as applied to our node=
package.<br>
> <br>
> I am certainly happy to just keep this change on our own branches. The=
re may be<br>
> nuances of this that I might not be aware of, but it seems to be worki=
ng great<br>
> for us and I have tested this with MDB_WRITEMAP enabled and disabled. =
So I<br>
> thought I would offer/suggest this change, as it seems like it is<br>
> straightforward change to improve stability. Thank you!<br>
> <br>
> <br>
<br>
<br>
-- <br>
=C2=A0 -- Howard Chu<br>
=C2=A0 CTO, Symas Corp.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<a href=3D"=
http://www.symas.com"; rel=3D"noreferrer" target=3D"_blank">http://www.symas=
.com</a><br>
=C2=A0 Director, Highland Sun=C2=A0 =C2=A0 =C2=A0<a href=3D"http://highland=
sun.com/hyc/" rel=3D"noreferrer" target=3D"_blank">http://highlandsun.com/h=
yc/</a><br>
=C2=A0 Chief Architect, OpenLDAP=C2=A0 <a href=3D"http://www.openldap.org/p=
roject/" rel=3D"noreferrer" target=3D"_blank">http://www.openldap.org/proje=
ct/</a><br>
</blockquote></div>
--0000000000000dacee05826a34a9--