Re: Faster slapadd in presence of multiple indices?

[Jonghyuk Choi <jongchoi@us.ibm.com>]

Actually I did examine this issue too
when I was striving to enhance the performance of slapadd a few months
ago. What I considered at that time is to combine the concurrent slapadding
with the index clustering approach (ITS#3611). The index clustering consists
of two phases: in-memory index creation and batched write of indices to
DB. If the two phases can be arranged to overlap, it should be possible
to keep CPU busy performing in-memory index creation during the time CPU
would have been kept idle waiting for I/O completion. In this way, it would
be possible to improve the performance of slapadd even in a single CPU
system (in addition to multiple CPU system) if I/O bandwidth is abound
(which is true esp. when index clustering approach is used). This has been
in my todo lists before I switched to examining Berkeley DB. I guess this
has a good potential to further improve the performance of slapadd beyond
what has been achieved by the BDB scalability patch.
- Jong-Hyuk

------------------------
Jong Hyuk Choi
IBM Thomas J. Watson Research Center - Enterprise Linux Group
P. O. Box 218, Yorktown Heights, NY 10598
email: jongchoi@us.ibm.com
(phone) 914-945-3979    (fax) 914-945-4425   TL: 862-3979

Howard Chu <hyc@symas.com> Sent by: owner-openldap-devel@OpenLDAP.org 09/07/2005 04:24 AM

To:        ando@sys-net.it         cc:        openldap-devel@OpenLDAP.org         Subject:        Re: Faster slapadd in presence of multiple indices?

Pierangelo Masarati wrote:
> I haven't investigated this issue yet but would exploiting multiple
CPUs
> allow faster slapadds (with -q, i.e. with less consistency checks)
if, for
> instance, the entry and the indices are generated concurrently?  Much
like
> the ancient ldif2index.  This comes from the consideration that
on the
> machine we're testing giant slapadds, we have 75% to 87.5% of the
CPU
> idle...
>
> Does anybody see any big stopper to this approach?
>   
I actually added multi-threading code to slapadd in a previous 
experiment. It's probably still in CVS, but I removed the code because
it yielded no measurable improvement. I think the problem is that this
will be I/O bound regardless of what CPU resources are available.

-- 
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  OpenLDAP Core Team            http://www.openldap.org/project/

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature