Re: back-bdb IDL limitations

[Jonghyuk Choi <jongchoi@us.ibm.com>]

During the performance evaluation and
testing of the LDAP component matching in OpenLDAP, we developed an ldif
split program written in C. It was a lot faster than doing it by scripting.
For the second point, isn't it possible
to figure out what the dn of the last committed entry is in a consistent
database by retrieving the last ID and the corresponding entry in the id2entry
database ?

By the "incremental slapadd",
I tried not to rule out a situation where the database is populated incrementally
for example day by day by using slapadd. If there's a system failure during
the current slapadd, it is possible with transactions to avoid the repopulation
of the entire directory which was not populated in the current slapadd
run. It might not be the common case scenario but it shows that it is desirbale
to maintain the transactional operation of slapadd and I guess that's why
the new quick slaptools mode is activated by a command line option.

- Jong-Hyuk

------------------------
Jong Hyuk Choi
IBM Thomas J. Watson Research Center - Enterprise Linux Group
P. O. Box 218, Yorktown Heights, NY 10598
email: jongchoi@us.ibm.com
(phone) 914-945-3979    (fax) 914-945-4425   TL: 862-3979

Howard Chu <hyc@symas.com> 03/04/2005 01:13 PM

To:        Jonghyuk Choi/Watson/IBM@IBMUS         cc:        David Boreham <david_list@boreham.org>, openldap-devel@OpenLDAP.org         Subject:        Re: back-bdb IDL limitations

We've tested ranges from 2 million to 20 million entries,
and it seems 
rather inconvenient to do incremental adds here. First we need to add a
flag to slapadd to tell it to skip the first N entries and start adding
from some offset into the input LDIF, because manually splitting such a
large input LDIF file is a pain. Second we need to add feedback from 
slapadd to tell you exactly where it was in the input when it failed 
(entry number may be enough, byte offset would probably be better) but
of course, if the add is interrupted because the entire machine crashed,
you may never see that feedback. I guess one question to answer is what
kind of failures are you protecting against, and what does it cost 
either way. For the most part, it is simpler to just wipe out the 
database and start over.

Jonghyuk Choi wrote:

>
> Is it so because of the recovery overhead can be huge ? Even so, 
> transactions seem to be indispensable to situations like incremental
> slapadd.
> - Jong-Hyuk
>
> ------------------------
> Jong Hyuk Choi
> IBM Thomas J. Watson Research Center - Enterprise Linux Group
> P. O. Box 218, Yorktown Heights, NY 10598
> email: jongchoi@us.ibm.com
> (phone) 914-945-3979    (fax) 914-945-4425   TL: 862-3979
>
>
>
>
>                  *"David
Boreham" <david_list@boreham.org>*
> Sent by: owner-openldap-devel@OpenLDAP.org
>
> 03/04/2005 12:06 PM
>
>                  
      
>         To:        <openldap-devel@OpenLDAP.org>
>         cc:        
>         Subject:        Re:
back-bdb IDL limitations
>
>
>
>
>  
> 2) there should be non-zero need for transaction protected operations
> of slap tools to cope with a system failure during a lengthy directory
> population.
> I think you will find that it's almost always faster to simply disable
> transactions and
> re-start the slapadd in the event of a failure.
>  
>  



-- 
  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland
Sun
  http://www.symas.com              
http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature