Re: ldbm database usage

["Kurt D. Zeilenga" <Kurt@OpenLDAP.org>]

At 08:39 PM 5/17/99 -0700, David Boreham wrote:
>Juan Carlos Gomez wrote:
>> > *Actually the secondary index files probably
>> > will grow unless open-ldap now deletes index keys.
>> > The primary index (id2entry) should not grow
>> > unbounded.
>>
>>   I suggested this to be fixed but got some opposition to it
>> I will try it though and if it works probably it will become an option in our
>> ldbm backend

The LDBM backend doesn't use transactions when interacting
with the underlying database.  Some indexes may be invalid.
As such, the backend must always verify that the index is
valid before using it (the same check it uses when handling
an ALLIDS block).  Luckily, this check is fairly fast.

Though we can attempt to maintain accurate indexes, we must
realize the limitations of the non-transactional backend design.
Hence, we must always error on too many than two few indexes.

As we move towards BerkeleyDB 2 with transactional support,
we can and should maintain accurate indexes.  Of course,
as we'll likely still use an ALLIDS block, such checks
are still needed.

>We just finished fixing all the places where keys need
>to be deleted in the Netscape server code. It took a
>few releases to find them all.

The Netscape server, TTBOMK, uses a Berkeley DB 2
transactional API.  OpenLDAP is headed in the same
direction.

>Once this is done it's possible to speed up some types
>of search by removing the filter test.

Yes.  Also, we should consider adding additional
indexes to speed up one-level and subtree searches...

I should also note that the IDList (ID_BLOCK) code is
terribly inefficient.  It uses for loops to move blocks
of data AND uses linear searches to find items in
sorted arrays. 

Kurt