Re: quick question

[James Bourne <jbourne@mtroyal.ab.ca>]

On Wednesday 24 July 2002 22:35, James Bourne wrote:
> While I'm investigating other avenues...
>
> Can I confirm with anyone that the file size on Linux 32bit for the
> back-ldbm using gdbm is 2G?

Hi,
I just thought I would update everyone on the list as to what I've found.

Using back-ldbm and gdbm was a problem.  gdbm seems to use the
system off_t for offsets and uses the system lseek.  lseek and off_t under
Linux use only signed 32bit intergers, therefore maximum referencable file
size is only 2GB.

Using back-ldbm and sleepycat db (Berkeley) is better.  Berkeley uses it's
own offset type and they say it's 64bit. (I haven't managed to get to the
2G mark yet on that one, it's only been building for about 6 hours).  

>From the documentation, if you use a 512B database page size, you can
attain a database file size of 2TB under Linux, if you are using a new
enough system to support large files.  If you use a 65K database
page size, this can increase to 256TB, the filesystem used I guess would
be an exercise for the user...

I will post another message once I find out where Berkeley ends up sitting.
>From the documentation, and the compile, and postings to other mailing
lists, it looks as though it should work fine past the 2GB limit of gdbm.

For a bit more back ground on our situation, we hit the limit after using
openldap for about 1 year under gdbm, without rebuilding the databases. It
contained approximately 158000 entries (each with posix info and mail
routing) as well as a few rather large groups.  The id2entry.gdbm file grew
to 2GB yesterday and that's when the problems occured.  Now, during the
rebuild, the gdbm database is only at ~890M with 119000+ entries re-added
to it, I think there was a lot of wasted space in the database.  It has
been rebuilding for 16 hours so far. The berkeley db is a bit smaller, at
~108000 entries, and that rebuild has been running now for about 6 hours
(that machine is slightly faster).

Regards
James Bourne
-- 
James Bourne, Supervisor Data Centre Operations
Mount Royal College, Calgary, AB, CA
www.mtroyal.ab.ca

******************************************************************************
This communication is intended for the use of the recipient to which it is
addressed, and may contain confidential, personal, and or privileged
information. Please contact the sender immediately if you are not the
intended recipient of this communication, and do not copy, distribute, or
take action relying on it. Any communication received in error, or
subsequent reply, should be deleted or destroyed.
******************************************************************************


"There are only 10 types of people in this world: those who
understand binary and those who don't."