RE: FW: Modify performance improvement for back-ldbm and back-bdb (ITS#1526)

["Howard Chu" <hyc@highlandsun.com>]

Looks like what I had in mind. I think there may be an issue with back-ldbm,
since it caches entries; the flags need to be cleared on the save_attrs when
there is an error. I can take care of that with the current patch.

This code has drawn my attention to the fact that back-ldbm doesn't recover
gracefully from indexing errors. I.e., when indexing attributes in a modify, if
there is an error, the indices are not restored to their original state. This
problem doesn't affect back-bdb since the transaction mechanism directly
handles
this issue. I don't know what possible situations could arise to cause this
kind of failure, and whether it is even possible to manually reset an index
when it
has already failed to update. Yet another reason to bang on back-bdb and get it
to production level.

  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support

> -----Original Message-----
> From: Gertjan van Wingerde [mailto:gwingerde@home.nl]
> Sent: Sunday, January 13, 2002 11:04 AM
> To: Howard Chu
> Cc: openldap-devel@openldap.org; masarati@aero.polimi.it
> Subject: Re: FW: Modify performance improvement for back-ldbm and
> back-bdb (ITS#1526)
>
>
> Okay,
>
> I've taken Howard's first alternative and implemented that in the
> attached patch. The patch still has the same effect on the modify
> performance.
>
> Please review and include in the source-base if review is satisfactory.
>
> P.S. Source-patch against release 2.0.19 is available if anyone is
> interested.
>
> Best regards,
>
> 	Gertjan.
>
> Howard Chu wrote:
>
> > I think this is a good idea as well, but I have a couple alternate ideas.
> > First, we could introduce a "flags" field in the Attribute structure, with
> > #define	SLAPD_ATTR_IXADD	0x01
> > #define	SLAPD_ATTR_IXDEL	0x02
> > and set these flags on the attrs as we find them in the modify.
> Seems pretty
> > clean and simple.
> >
> > As another possibility, we can simply break the attr lists onto
> separate lists
> > as the modify proceeds. save_attrs would be broken out onto
> save_attr_del for
> > deleted indexed attrs, and likewise e->e_attrs would be broken out onto
> > e_attr_add for added indexed attrs. The separated lists can be
> easily rejoined
> > at the end
> > of the operation, but their order would not be preserved. Of
> course we claim no
> > guarantees on attribute order as it is, but the current code tends
> to preserve
> > it.
> >
> >   -- Howard Chu
> >   Chief Architect, Symas Corp.       Director, Highland Sun
> >   http://www.symas.com               http://highlandsun.com/hyc
> >   Symas: Premier OpenSource Development and Support
> >
> > -----Original Message-----
> > From: owner-openldap-bugs@OpenLDAP.org
> > [mailto:owner-openldap-bugs@OpenLDAP.org] On Behalf Of
> > masarati@aero.polimi.it
> >
> >
> >>Full_Name: Gertjan van Wingerde
> >>
> >
> >>I have improved the modify performance of OpenLDAP by only reindexing
> >>attributes
> >>that are actually changed in the modify operation instead of all the
> >>
> > attributes
> >
> >>of
> >>an entry. This change improves the performance of a modify operation on a
> >>non-indexed
> >>attribute from about 4 modifications per second to about 38
> modifications per
> >>second.
> >>If an indexed attribute is being modified, the results are a bit less
> >>
> > dramatic,
> >
> >>but
> >>still an improvement of about 200% is observed then.
> >>Note: The changes are only applicable to the LDBM backend and the BDB
> >>
> > backend.
> >
> >>As far
> >>as I can see no other backends can use this optimisation.
> >>
> >
> > Your idea sounds definitely good; I'm not very comfortable with the
> > fixed size array of index add/remove attrs.  This may require a little
> > reworking.  I understand the improvement can be significant, though.
> >
> > Pierangelo
> >
> >
> >
>
>
>
> --
> 	MvG,
>
> 		Gertjan
>
> ----------
>
> Gertjan van Wingerde
> Geessinkweg 177
> 7544 TX Enschede
> The Netherlands
> E-mail: gwingerde@home.nl
>