Re: lists/queues (Was: back-bdb performance)

["Kurt D. Zeilenga" <Kurt@OpenLDAP.org>]

At 07:00 AM 2001-12-06, Kurt D. Zeilenga wrote:
>I have committed BSD queue(3) support to OpenLDAP.

#include <ac/queue.h>

not <sys/queue.h>

>This provides
>a variety of queue/list mechanisms.  These mechanisms should be
>used in providing any new (or updated) queue/list capabilities
>in OpenLDAP.




>http://www.openldap.org/software/man.cgi?query=queue
>
>Kurt
>
>At 05:18 AM 2001-12-06, Pedro Jose Marron wrote:
>>        Hi Howard,
>>
>>        I am new to the list, so maybe some of the things I say below have
>>already been discussed, but I thought I might just give you my take on the
>>problem you mention.
>>
>>> As for search/read speed, I don't have a good metric yet. My current runs of
>>> test008-concurrency all execute in about 52 seconds, no matter if it's
>>> back-ldbm, back-hdb, or (back-bdb with new index code). All of the
>>> search/read iterations finish quickly, and the bulk of the time is the
>>> add/del task.
>>  ^^^^^^^
>>  ^^^^^^^
>>
>>        I am not surprised at all that the bulk of the time is spent in
>>the add/del task, because it could be implemented much more efficiently. A
>>while back (OpenLDAP 1.2.10), I designed and implemented a way to
>>incorporate arbitrary XML documents into LDAP. The whole thing worked very
>>well, except that loading the XML document was particularly slow. Looking
>>at the slap_op_add and slap_op_del functions in operation.c, I realized
>>the problem was in the way new operations were added at the end of the
>>operation queue.
>>
>>int slap_op_add(
>>    Operation           **olist,
>>        Operation               *op
>>)
>>{
>>        Operation       **tmp;
>>
>>        for ( tmp = olist; *tmp != NULL; tmp = &(*tmp)->o_next )
>>                ;       /* NULL */
>>
>>        *tmp = op;
>>
>>        return 0;
>>}
>>
>>        (This is taken from OpenLDAP 2.0.18, so it is still implemented
>>like before...)
>>
>>        The for loop is ok if we have few operations in olist, or if you
>>only use synchronous operations, but in my case (and I was only adding
>>3000 - 4000 nodes), it was getting really big, because slapd could not
>>keep up with the rate at which operations were being sent by the client.
>>Walking from the first operation to the last one, just to add one more
>>operation is not the most efficient way to implement the slap_op_add
>>function, so I changed the structure of the operations queue to also
>>contain a pointer to the last operation. The result was that insertion
>>into the LDAP tree improved by a lot (an order of magnitude, or two, I
>>don't remember anymore).
>>
>>        I don't know if that is the problem you are having, and whether or
>>not you are using asynchronous add operations in your tests, but I thought
>>I just mentioned it in case that solve your problem. The fact that you
>>spend most of the time adding/deleting operations seems to indicate that
>>the slap_op_add function needs to be implemented in a different way...
>>
>>        Other than that, I think your numbers look great! I am looking
>>forward to seeing the back-dbd and back-hdb code in future releases of
>>OpenLDAP.
>>
>>        Later,
>>
>>        Pedro
>>
>>P.S. If you want me to send you my modifications to the old OpenLDAP
>>1.2.10, let me know!