Re: Questions...

["Jong" <jongchoi@OpenLDAP.org>]

> > -----Original Message-----
> > From: owner-openldap-devel@OpenLDAP.org
> > [mailto:owner-openldap-devel@OpenLDAP.org]On Behalf Of Quanah Gibson-Mount
>
> > --On Tuesday, January 27, 2004 2:35 PM -0500 Jong
> > <jongchoi@OpenLDAP.org>
> > wrote:
>
> > > The difference between ldbm and bdb became significant when
> > > indexing came into the picture.
> > > So this may be a sign that we should improve the bdb indexing system
> > > further.
> >
> > My data loads always find indexing to be the biggest
> > bottleneck with BDB.
> > Our systems that index multiple things went up two hours in
> > indexing time
> > between 2.1 & 2.2 with BDB (from 7 hours to 9 hours).  I was
> > able to drop
> > this to 2 hours (7 hours less!) by use the new shared memory
> > key.  On our
> > master (which indexes only 3 or 4 things), the load time takes
> > approximately 25 minutes, on both 2.1 and 2.2.
>
> With my recent changes in HEAD, slapindex time has improved quite a bit. I
> have a 400MB database that takes ~52 minutes to slapadd, with 3 indices. It
> used to take several hours (10 or more) to slapindex this database. It now
> takes 1:06 hours, with IDL cache size explicitly set to 1 (effectively
> disabled). With the IDL cache size at its new default (10000, given entry
> cachesize of 1000) it takes only 11 minutes.
>
> The use of the IDL cache should also have a significant improvement on
> slapd's runtime indexing speed, not just slapindex.

Great !
It seems that you've changed the IDL cache from write-through-invalidate to
write-through-update.
I've done this as the write-through-invalidate design, because at the time of design, the
main objective was
to improve the search performance both with / wihtout the presence of intermittent
updates.
It seems that the slapindex access has a repeated read-modify-write pattern whose
performance
can be significantly improved by the write-through-update cache design.
- Jong