[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: LMDB usage on windows - to much memory needed



Frank Offermanns wrote:
Hello Christian,

thanks for the answer.

What you say is true, but with hdb backend I could use larger databases,
because it has no memory map file, hasn't it?

So I hoped, that I might have missed a configuration parameter which
would have made  mdb be more usefull for 32 bit.

32 bit servers were obsolete at least 5 years ago, maybe 10 years.

But it seems, that I haven't missed such a parameter, so I have to use
64 bit openldap (or use smaller databases).

I am philosophically opposed to supporting 32 bit architectures in LMDB.
http://www.openldap.org/lists/openldap-devel/201405/msg00014.html

However, for the few people who insist, there is an experimental branch available that can support larger DBs on 32 bit CPUs, by unmapping and remapping segments of the DB.

https://gitorious.org/mdb/mdb/source/69d7cb8d44e04f02d8d0c923ae71fbaaa9f42f3a:

Due to the added system calls involved, it is significantly slower. We have plans to merge it into LMDB 1.0 as an optional feature, but for the moment it's only in that branch.

Fundamentally, supporting this feature in LMDB is the wrong thing to do. It adds code/bloat and slows down the overall codebase just in order to support a dying technology. Even in embedded processing 64 bit CPUs are available cheaply now; anyone still pouring money into 32 bit CPUs needs to have their head examined.

Best regards,
Frank




Von: Christian Kratzer <ck-lists@cksoft.de>
An: Frank Offermanns <Frank.Offermanns@caseris.de>,
Kopie: Kristoffer Sjögren <stoffe@gmail.com>,
openldap-technical@openldap.org
Datum: 29.01.2015 14:55
Betreff: Re: Re: LMDB usage on windows - to much memory needed
------------------------------------------------------------------------



Hi,

On Thu, 29 Jan 2015, Frank Offermanns wrote:

 > Hi,
 >
 > thanks for the information, but unfortunately this information is not
 > helpfull for me.
 >
 > Is there a way to configure LMDB on Windows 32 bit, so that a
database can
 > be larger?
 > Or is it mandantory to use 64 bit with lmdb (at least on windows file
 > systems) when handling larger data?

You need 64 bits on any kind of OS to support multi gigabyte databases.

No 32 bit OS will allow a single process to memory map much more than 1
or perhaps 2 GB of address space.

Greetings
Christian



 >
 > Regards,
 > Frank
 >
 >
 >
 >
 > Von:    Kristoffer Sjögren <stoffe@gmail.com>
 > An:     Frank Offermanns <Frank.Offermanns@caseris.de>,
 > Kopie:  openldap-technical@openldap.org
 > Datum:  28.01.2015 16:32
 > Betreff:        Re: LMDB usage on windows - to much memory needed
 >
 >
 >
 > Regarding size.
 >
 > "If an application specifies a size for the file mapping object that is
 > larger than the size of the actual named file on disk and if the page
 > protection allows write access (that is, the flProtect parameter
specifies
 > PAGE_READWRITE or PAGE_EXECUTE_READWRITE), then the file on disk is
 > increased to match the specified size of the file mapping object. If the
 > file is extended, the contents of the file between the old end of the
file
 > and the new end of the file are not guaranteed to be zero; the
behavior is
 > defined by the file system. If the file on disk cannot be increased,
 > CreateFileMapping fails and GetLastError returns ERROR_DISK_FULL."
 >
 >
http://msdn.microsoft.com/en-us/library/windows/desktop/aa366537%28v=vs.85%29.aspx
 >
 > On Wed, Jan 28, 2015 at 3:27 PM, Frank Offermanns <
 > Frank.Offermanns@caseris.de> wrote:
 > Hello,
 >
 > we would like to change our backend from hdb to mdb.
 >
 > I did some initial tests and found out the following:
 > When setting maxsize to 1.4 GB, the size of the database on the file
 > system is about 1.4 GB. I thought I read that the maxsize has nothing to
 > do with the actual physical size needed. (If I have only 10 MB of data my
 > database should be about 10 MB).
 > Even worse is the fact, that slapd.exe then needs about 1.5 GB virtual
 > size memory.
 > And if I use 32 Bit slapd.exe the process will crash when reaching 2 GB
 > virtual size. (as every 32 bit process will do)
 >
 > Now my conclusion is, that the things described in the mdb paper are only
 > valid for unix/linux, because windows uses another memory system. Is this
 > correct? Or is there a way I haven't found to configure OpenLDAP on
 > windows so that it does not need so much RAM.
 > The size on the disk does not matter for us, but since we use only 32 bit
 > slapd.exe the RAM does matter.
 >
 > Best regards,
 > Frank
 >
 >

--
Christian Kratzer   CK Software GmbH
Email:   ck@cksoft.de Wildberger Weg 24/2
Phone:   +49 7032 893 997 - 0       D-71126 Gaeufelden
Fax:     +49 7032 893 997 - 9       HRB 245288, Amtsgericht Stuttgart
Mobile:  +49 171 1947 843           Geschaeftsfuehrer: Christian Kratzer
Web: http://www.cksoft.de/


--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/