RE: FW: slapd threads on master pegs multiple CPU Sun running Solaris 8 (ITS#2302)

["Kurt D. Zeilenga" <Kurt@OpenLDAP.org>]

At 11:12 AM 2/15/2003, Howard Chu wrote:
>Thinking about this more, my previous post didn't make much sense... In a
>regular (non-transactional) read operation, BDB releases locks as soon as the
>read finishes. I think this is the ideal mode of operation for reads.
>Wrapping reads in a transaction means locks will accumulate until the end of
>the transaction, which increases the probability of a particular read causing
>another write to block. If we just break the search operation into many
>little read transactions, this is no different than the locking behavior of
>non-transactional reads. So, I still don't see any advantage to be gained
>here, and I don't see that the BDB library itself needs any changes.

The concern <http://www.sleepycat.com/docs/ref/lock/notxn.html>:

It is important to realize that concurrent applications that use locking must ensure that two concurrent threads do not block each other. However, because Btree and Hash access method page splits can occur at any time, there is virtually no way to guarantee that an application that writes the database cannot deadlock. Applications running without the protection of transactions may deadlock, and can leave the database in an inconsistent state when they do so. Applications that need concurrent access, but not transactions, are more safely implemented using the Berkeley DB Concurrent Data Store Product. 



>  -- Howard Chu
>  Chief Architect, Symas Corp.       Director, Highland Sun
>  http://www.symas.com               http://highlandsun.com/hyc
>  Symas: Premier OpenSource Development and Support
>
>> -----Original Message-----
>> From: Kurt D. Zeilenga [mailto:Kurt@OpenLDAP.org]
>
>> One possible approach is not to wrap the whole search operation
>> in a single transactions, but use many.  One for baseObject
>> finding, one for candidate determination, and one for each
>> candidate.  However, this would likely cost way more than 10%
>> (with logging).
>>
>> Kurt
>>
>>
>> At 04:11 PM 2/14/2003, Howard Chu wrote:
>> >I guess it's worth exploring. It might be nice if BDB
>> provided a locker
>> >entity for this purpose - i.e., like a transaction, all
>> locks are held by one
>> >locker, but without any of the logging associated with a
>> real transaction. I
>> >haven't convinced myself yet that it really buys us much.
>> >
>> >In the normal case, you slow down the search by 10% and get
>> no other benefit.
>> >
>> >In the case of multiple concurrent operations, you actually decrease
>> >concurrency because locks obtained inside a transaction are
>> held to the end
>> >of the transaction, no matter if you explicitly release them or not.
>> >
>> >In the case of a search spawned while processing another
>> transaction, you are
>> >creating nested transactions, which decreases concurrency
>> even fruther
>> >because all of the locks in each nested transaction are held
>> until the parent
>> >is released. If you don't nest the transactions, then you're just as
>> >vulnerable to deadlock as before.
>> >
>> >The deadlock problem seems to arise from the fact that we
>> can either use
>> >transactions, in which case all locks are tied to that one
>> transaction (this
>> >is good because it prevents us from deadlocking ourself, bad
>> because it keeps
>> >locks longer than they're needed) or we have every
>> individual operation
>> >getting/releasing its own locks on the fly (bad because we
>> can deadlock
>> >ourself easily). What would be ideal is something in the
>> middle - passing a
>> >locker ID to every BDB function, so that it gets used for
>> all the locks, and
>> >they are all released at the exact moment we wish.
>> >
>> >  -- Howard Chu
>> >  Chief Architect, Symas Corp.       Director, Highland Sun
>> >  http://www.symas.com               http://highlandsun.com/hyc
>> >  Symas: Premier OpenSource Development and Support
>> >
>> >-----Original Message-----
>> >From: owner-openldap-bugs@OpenLDAP.org
>> >[mailto:owner-openldap-bugs@OpenLDAP.org]On Behalf Of
>> jongchoi@us.ibm.com
>> >
>> >I think we should restart discussing on the
>> transaction-protected searches.
>> >The overhead of transactional searches does not seem to be
>> prohibitively
>> >significant.
>> >(I remember around 10% when I experimented with it.)
>> >One concern is the semantics of the aborted LDAP search.
>> >1. abort the whole search ?
>> >    In this case, it would be the client's responsibility to
>> resend the
>> >search request.
>> >2. abort a single db->get() of the candidate loop ?
>> >   We should have one transaction per loop iteration.
>> >   The overhead of frequent transaction begin and commit has to be
>> >measured.
>> >- Jong
>> >
>> >BTW, Joseph, what is the version number of BerkeleyDB ? Is it 3.x ?
>> >
>> >------------------------
>> >Jong Hyuk Choi
>> >IBM Thomas J. Watson Research Center - Enterprise Linux Group
>> >P. O. Box 218, Yorktown Heights, NY 10598
>> >email: jongchoi@us.ibm.com
>> >(phone) 914-945-3979    (fax) 914-945-4425   TL: 862-3979
>>