Re: Road Map : improved scalability - EFBIG

["Gary Allen (ldap)" <gvldap@corvu.com>]

Again, more conjecture, with a little background, and something to try...

I've seen similar issues on many products that work across platforms.  
Because, for instance, Windows NT 4.0 will function on a FAT filesystem, 
the max file size for many products that support NT 4.0 follows the 2GB 
filesize limit.

This has the effect of cutting down on support calls, "Why won't this 
open on my NT box?"  Thus this is often by design, as the files that are 
available on one platform should work on all platforms.

In the case of OpenLDAP specifically, this should not be the case - as 
OpenLDAP itself doesn't actually deal with the data size (only the size 
of query input and output).  However the slapcat tool (being built with 
BDB specific capabilities) may do this on purpose.

One thing to try (this works with many zip and tar utilities) is to see 
if you can PIPE the file through a program that can handle large files.  
slapcat | gzip

Thank you,
Gary Allen


Donn Cave wrote:

> On Monday, April 12, 2004, at 08:53 AM, Armbrust, Daniel C. wrote:
>
> > This question may fit - I just tried create a database with slapadd 
> > and a 2.1 GB ldif file and all I get is a message saying:
> > Filename: File to large
> >
> > This was with 2.2.5 - I'm going to grab the latest openldap now, and 
> > try again, in hopes that I don't have to split this file.
> >
> > So, what exactly is too large?
>
>
> Depends!  It sounds to me like you're getting error 27, EFBIG,
> from a system call like open(), because the LDIF file is a
> hair over the size supported by old style filesystems.  Right
> around 2.1 Gb you get into a special case of "too large" that
> has to do with the number of bits needed to represent a positive
> integer.
>
> If that's it, you need to find out, for your platform, how to
> avoid building this legacy file stuff into slapadd et al., and
> instead built with `large file support' or whatever they may
> call it.
>
> This is purely speculation and surmise, but I didn't notice
> any other followup (possibly a problem with the list - I don't
> even see your message in the archives at this point.)
>
>     Donn Cave, donn@u.washington.edu