Re: thread pools, performance

[Howard Chu <hyc@symas.com>]

David Boreham wrote:
> Gavin Henry wrote:
> > Thanks for all this Howard. It certainly makes it clear where OpenLDAP
> > lies in the LDAP world (looking down from the top and around at
> > everyone else ;-) ).

> Hmm...not so sure about the 'everyone' part because
> I added this feature (parallel import with correctly implemented
> partent/child entry interlocking) to the product that is now SunDS and
> FedoraDS
> in 1999.

With all due respect David, you've done some nice work on those products but 
OpenLDAP is still significantly faster than either SunDS or FedoraDS, both in 
importing data and in the authentication rates.

IIRC we had this conversation about parallel import before on this list. I had 
tried parallel entry importing before and gave up on it because it showed no 
benefit on my tests. Then based on your hints I tested a variety of other 
approaches. What I eventually settled on for OL 2.3 - serial entry import with 
parallel indexing - has been the fastest approach in my tests.

One obvious reason is that no entry locking is required because only a single 
thread ever operates on any given DB. With parallel writes of entries to the 
same DB, you still need full locking to prevent data corruption. Another 
reason is that BerkeleyDB's B-trees are optimized for sequential writes; 
parallelizing the entry writes defeats that optimization because it partially 
randomizes the I/Os. (I.e., it requires seeks instead of just sequential access.)

It's too bad that SunDS still disallows publishing benchmark results. The 
current license terms just tells the world they have something to hide.
--
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP     http://www.openldap.org/project/