Re: ldapadd: hopelessly slow loading due to high disk iowait

[Yuri Rabover <yurir@3Cube.com>]

"Kurt D. Zeilenga" wrote:

> >flaw either in LDAP or gdbm engine that is turning what should be a
> >linear hashing problem into something that is binomial or cubic.  We
> >are doing some timeing studes with databases of size 100, 1000, and
> >10,000
> >to estimate the degree of non-linearity, but the DEFINITIVE way of
> >addressing this issue would be to rebuild LDAP and gdbm with profiling
> >turned on, and see where in the code all this time is being spent.
> >This is going a bit beyond our immediate brief (to get a snapshot of
> >current art) -- is anyone out there sufficiently challenged by these
> >findings to do the profiling??  Any other ideas about what is
> >happening
> >here?
> 
> Well, there could be a number of reasons... (including bugs), but
> I'd suspect you hit some limit in the file system implementation.
> 
> You might experiment with ldif2ldbm in this case (I generally don't
> recommend ldif2ldbm).  However, since I assume you are programmatically
> generating the LDIF, you can test fragments (using ldapadd) to ensure
> correctness and then (after whipping the database files) use
> ldif2ldbm.
> 
> I suggest you also experiement with Sleepycat BerkeleyDB.  We
> support both their hash and btree implementations each of which
> have different performance characteristics.

Actually I performed similar experiments with Sleepycat both on
Solaris and FreeBSD and noticed the same behavior. File system
implementations are different (although one may say that it is
the same UFS underneath).

	Yuri Rabover
	3Cube, Inc.