[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Re: (ITS#8975) Calling mdb_env_set_mapsize on a shared DB fails sometimes
- To: openldap-its@OpenLDAP.org
- Subject: Re: (ITS#8975) Calling mdb_env_set_mapsize on a shared DB fails sometimes
- From: quanah@symas.com
- Date: Thu, 21 Feb 2019 17:12:55 +0000
- Auto-submitted: auto-generated (OpenLDAP-ITS)
Hi,
It is literally the first set of commits listed at that URL, so it's
clearly already there.
Hope that helps!
--Quanah
--On Thursday, February 21, 2019 4:45 PM +0000 kriszyp@gmail.com wrote:
> --0000000000000dacee05826a34a9
> Content-Type: text/plain; charset="UTF-8"
>
> Thank you, I appreciate it! Do you know how often or when this will get
> into the github mirror https://github.com/LMDB/lmdb/commits/mdb.RE/0.9 (or
> if I am misunderstanding how the branches are synced)?
>
> On Sun, Feb 10, 2019 at 3:14 AM Howard Chu <hyc@symas.com> wrote:
>
>> kriszyp@gmail.com wrote:
>> > Full_Name: Kristopher William Zyp
>> > Version: LMDB 0.9.23
>> > OS: Windows Server 2012 R2, Windows 10
>> > URL:
>> https://github.com/kriszyp/node-lmdb/commit/6df903907f5516320e9a8afce45b
>> d32ab4e8e1f2.patch
>> > Submission from: (NULL) (71.199.6.148)
>> >
>>
>> Thanks for the report and patch, added to mdb.RE/0.9
>>
>> > Calling mdb_env_set_mapsize to increase the map size, when a DB is also
>> in use
>> > by other processes, when MDB_WRITEMAP is enabled (and the db file has
>> been
>> > opened with PAGE_READWRITE access), on Windows, will occasionally
>> > (seems
>> like
>> > about 1/100 attempts fails) produce an error "The requested operation
>> cannot be
>> > performed on a file with a user-mapped section open", or segfaults. The
>> error
>> > occurs in the SetFilePointer (or SetEndOfFile) call in mdb_env_map that
>> is
>> > performed to increase the allocated file size to the map size, prior to
>> > CreateFileMapping.
>> >
>> > As it turns out this is pretty easy to solve, because manually
>> > expanding
>> the
>> > file size is not necessary when calling CreateFileMapping with
>> PAGE_READWRITE
>> > access, as Windows will automatically expand the file size for us, when
>> opened
>> > with the page write access enabled. Of course, this means all processes
>> must be
>> > consistent in use of MDB_WRITEMAP, but the documentation already makes
>> this
>> > explicit and clear.
>> >
>> > I believe this can be fixed by simply adding a check for MDB_WRITEMAP
>> > in
>> the if
>> > statement that calls SetFilePointer:
>> >
>> > if (!(flags & MDB_WRITEMAP) &&
>> > (SetFilePointer(env->me_fd,
>> sizelo, &sizehi, 0)
>> > != (DWORD)sizelo
>> > || !SetEndOfFile(env->me_fd)
>> > || SetFilePointer(env->me_fd, 0, NULL, 0) != 0))
>> > return ErrCode();
>> >
>> > The attached URL has the change as a patch/diff as applied to our node
>> package.
>> >
>> > I am certainly happy to just keep this change on our own branches.
>> > There
>> may be
>> > nuances of this that I might not be aware of, but it seems to be
>> > working
>> great
>> > for us and I have tested this with MDB_WRITEMAP enabled and disabled.
>> > So
>> I
>> > thought I would offer/suggest this change, as it seems like it is
>> > straightforward change to improve stability. Thank you!
>> >
>> >
>>
>>
>> --
>> -- Howard Chu
>> CTO, Symas Corp. http://www.symas.com
>> Director, Highland Sun http://highlandsun.com/hyc/
>> Chief Architect, OpenLDAP http://www.openldap.org/project/
>>
>
> --0000000000000dacee05826a34a9
> Content-Type: text/html; charset="UTF-8"
> Content-Transfer-Encoding: quoted-printable
>
> <div dir=3D"ltr"><div dir=3D"ltr">Thank you, I appreciate it! Do you know
> h= ow often or when this will get into the github mirror <a
> href=3D"https://gi=
> thub.com/LMDB/lmdb/commits/mdb.RE/0.9">https://github.com/LMDB/lmdb/commi
> ts= /mdb.RE/0.9</a> (or if I am misunderstanding how the branches are
> synced)?<= /div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr"
> class=3D"gmail_a= ttr">On Sun, Feb 10, 2019 at 3:14 AM Howard Chu <<a
> href=3D"mailto:hyc@s= ymas.com">hyc@symas.com</a>>
> wrote:<br></div><blockquote class=3D"gmail_= quote" style=3D"margin:0px
> 0px 0px 0.8ex;border-left:1px solid rgb(204,204,=
> 204);padding-left:1ex"><a href=3D"mailto:kriszyp@gmail.com";
> target=3D"_blan= k">kriszyp@gmail.com</a> wrote:<br>
> > Full_Name: Kristopher William Zyp<br>
> > Version: LMDB 0.9.23<br>
> > OS: Windows Server 2012 R2, Windows 10<br>
> > URL: <a
> href=3D"https://github.com/kriszyp/node-lmdb/commit/6df903907f=
> 5516320e9a8afce45bd32ab4e8e1f2.patch" rel=3D"noreferrer"
> target=3D"_blank">=
> https://github.com/kriszyp/node-lmdb/commit/6df903907f5516320e9a8afce45bd
> 32= ab4e8e1f2.patch</a><br>
> > Submission from: (NULL) (71.199.6.148)<br>
> > <br>
> <br>
> Thanks for the report and patch, added to mdb.RE/0.9<br>
> <br>
> > Calling mdb_env_set_mapsize to increase the map size, when a DB is
> als= o in use<br>
> > by other processes, when MDB_WRITEMAP is enabled (and the db file
> has = been<br>
> > opened with PAGE_READWRITE access), on Windows, will occasionally
> (see= ms like<br>
> > about 1/100 attempts fails) produce an error "The requested
> opera= tion cannot be<br>
> > performed on a file with a user-mapped section open", or
> segfault= s. The error<br>
> > occurs in the SetFilePointer (or SetEndOfFile) call in mdb_env_map
> tha= t is<br>
> > performed to increase the allocated file size to the map size, prior
> t= o<br>
> > CreateFileMapping.<br>
> > <br>
> > As it turns out this is pretty easy to solve, because manually
> expandi= ng the<br>
> > file size is not necessary when calling CreateFileMapping with
> PAGE_RE= ADWRITE<br>
> > access, as Windows will automatically expand the file size for us,
> whe= n opened<br>
> > with the page write access enabled. Of course, this means all
> processe= s must be<br>
> > consistent in use of MDB_WRITEMAP, but the documentation already
> makes= this<br>
> > explicit and clear.<br>
> > <br>
> > I believe this can be fixed by simply adding a check for
> MDB_WRITEMAP = in the if<br>
> > statement that calls SetFilePointer:<br>
> > <br>
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0if (!(flags
> &= ; MDB_WRITEMAP) && (SetFilePointer(env->me_fd, sizelo,
> &size= hi, 0)<br>
> > !=3D (DWORD)sizelo<br>
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
> = =C2=A0 =C2=A0|| !SetEndOfFile(env->me_fd)<br>
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
> = =C2=A0 =C2=A0|| SetFilePointer(env->me_fd, 0, NULL, 0) !=3D 0))<br>
> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0
> = =C2=A0 =C2=A0return ErrCode();<br>
> > <br>
> > The attached URL has the change as a patch/diff as applied to our
> node= package.<br>
> > <br>
> > I am certainly happy to just keep this change on our own branches.
> The= re may be<br>
> > nuances of this that I might not be aware of, but it seems to be
> worki= ng great<br>
> > for us and I have tested this with MDB_WRITEMAP enabled and
> disabled. = So I<br>
> > thought I would offer/suggest this change, as it seems like it is<br>
> > straightforward change to improve stability. Thank you!<br>
> > <br>
> > <br>
> <br>
> <br>
> -- <br>
> =C2=A0 -- Howard Chu<br>
> =C2=A0 CTO, Symas Corp.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<a
> href=3D"= http://www.symas.com"; rel=3D"noreferrer"
> target=3D"_blank">http://www.symas= .com</a><br>
> =C2=A0 Director, Highland Sun=C2=A0 =C2=A0 =C2=A0<a
> href=3D"http://highland= sun.com/hyc/" rel=3D"noreferrer"
> target=3D"_blank">http://highlandsun.com/h= yc/</a><br>
> =C2=A0 Chief Architect, OpenLDAP=C2=A0 <a
> href=3D"http://www.openldap.org/p= roject/" rel=3D"noreferrer"
> target=3D"_blank">http://www.openldap.org/proje= ct/</a><br>
> </blockquote></div>
>
> --0000000000000dacee05826a34a9--
>
>
>
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>