lists/queues (Was: back-bdb performance)

["Kurt D. Zeilenga" <Kurt@OpenLDAP.org>]

I have committed BSD queue(3) support to OpenLDAP.  This provides
a variety of queue/list mechanisms.  These mechanisms should be
used in providing any new (or updated) queue/list capabilities
in OpenLDAP.

http://www.openldap.org/software/man.cgi?query=queue

Kurt

At 05:18 AM 2001-12-06, Pedro Jose Marron wrote:
>        Hi Howard,
>
>        I am new to the list, so maybe some of the things I say below have
>already been discussed, but I thought I might just give you my take on the
>problem you mention.
>
>> As for search/read speed, I don't have a good metric yet. My current runs of
>> test008-concurrency all execute in about 52 seconds, no matter if it's
>> back-ldbm, back-hdb, or (back-bdb with new index code). All of the
>> search/read iterations finish quickly, and the bulk of the time is the
>> add/del task.
>  ^^^^^^^
>  ^^^^^^^
>
>        I am not surprised at all that the bulk of the time is spent in
>the add/del task, because it could be implemented much more efficiently. A
>while back (OpenLDAP 1.2.10), I designed and implemented a way to
>incorporate arbitrary XML documents into LDAP. The whole thing worked very
>well, except that loading the XML document was particularly slow. Looking
>at the slap_op_add and slap_op_del functions in operation.c, I realized
>the problem was in the way new operations were added at the end of the
>operation queue.
>
>int slap_op_add(
>    Operation           **olist,
>        Operation               *op
>)
>{
>        Operation       **tmp;
>
>        for ( tmp = olist; *tmp != NULL; tmp = &(*tmp)->o_next )
>                ;       /* NULL */
>
>        *tmp = op;
>
>        return 0;
>}
>
>        (This is taken from OpenLDAP 2.0.18, so it is still implemented
>like before...)
>
>        The for loop is ok if we have few operations in olist, or if you
>only use synchronous operations, but in my case (and I was only adding
>3000 - 4000 nodes), it was getting really big, because slapd could not
>keep up with the rate at which operations were being sent by the client.
>Walking from the first operation to the last one, just to add one more
>operation is not the most efficient way to implement the slap_op_add
>function, so I changed the structure of the operations queue to also
>contain a pointer to the last operation. The result was that insertion
>into the LDAP tree improved by a lot (an order of magnitude, or two, I
>don't remember anymore).
>
>        I don't know if that is the problem you are having, and whether or
>not you are using asynchronous add operations in your tests, but I thought
>I just mentioned it in case that solve your problem. The fact that you
>spend most of the time adding/deleting operations seems to indicate that
>the slap_op_add function needs to be implemented in a different way...
>
>        Other than that, I think your numbers look great! I am looking
>forward to seeing the back-dbd and back-hdb code in future releases of
>OpenLDAP.
>
>        Later,
>
>        Pedro
>
>P.S. If you want me to send you my modifications to the old OpenLDAP
>1.2.10, let me know!