RE: back-bdb performance

[Pedro Jose Marron <pjmarron@informatik.uni-freiburg.de>]

	Hi Howard,

	I am new to the list, so maybe some of the things I say below have
already been discussed, but I thought I might just give you my take on the
problem you mention.

> As for search/read speed, I don't have a good metric yet. My current runs of
> test008-concurrency all execute in about 52 seconds, no matter if it's
> back-ldbm, back-hdb, or (back-bdb with new index code). All of the
> search/read iterations finish quickly, and the bulk of the time is the
> add/del task.
  ^^^^^^^
  ^^^^^^^

	I am not surprised at all that the bulk of the time is spent in
the add/del task, because it could be implemented much more efficiently. A
while back (OpenLDAP 1.2.10), I designed and implemented a way to
incorporate arbitrary XML documents into LDAP. The whole thing worked very
well, except that loading the XML document was particularly slow. Looking
at the slap_op_add and slap_op_del functions in operation.c, I realized
the problem was in the way new operations were added at the end of the
operation queue.

int slap_op_add(
    Operation           **olist,
        Operation               *op
)
{
        Operation       **tmp;

        for ( tmp = olist; *tmp != NULL; tmp = &(*tmp)->o_next )
                ;       /* NULL */

        *tmp = op;

        return 0;
}

	(This is taken from OpenLDAP 2.0.18, so it is still implemented
like before...)

	The for loop is ok if we have few operations in olist, or if you
only use synchronous operations, but in my case (and I was only adding
3000 - 4000 nodes), it was getting really big, because slapd could not
keep up with the rate at which operations were being sent by the client.
Walking from the first operation to the last one, just to add one more
operation is not the most efficient way to implement the slap_op_add
function, so I changed the structure of the operations queue to also
contain a pointer to the last operation. The result was that insertion
into the LDAP tree improved by a lot (an order of magnitude, or two, I
don't remember anymore).

	I don't know if that is the problem you are having, and whether or
not you are using asynchronous add operations in your tests, but I thought
I just mentioned it in case that solve your problem. The fact that you
spend most of the time adding/deleting operations seems to indicate that
the slap_op_add function needs to be implemented in a different way...

	Other than that, I think your numbers look great! I am looking
forward to seeing the back-dbd and back-hdb code in future releases of
OpenLDAP.

	Later,

	Pedro

P.S. If you want me to send you my modifications to the old OpenLDAP
1.2.10, let me know!