ldap_result() timeout zero returns only one result at a time

[Mark R Bannister <mark@proseconsulting.co.uk>]

Hi,

I've discovered this with the python-ldap module, but having discussed 
it with the author and done some further testing I'm pretty sure this is 
down to the ldap_result() C API.

I have written a daemon that has a number of worker threads.  A worker 
thread can launch an async LDAP search operation, and then it goes off 
to do other work while the search is running.  It adds the FD of the 
socket used for the search operation to its select() call, and if there 
is data to read on that FD it wakes up and runs ldap_result() to grab 
those results.  If the results are incomplete, it goes back around the 
loop again (check for more work to do, call select, wake up, ldap_result 
etc.)

When I call ldap_result(), the
 timeout parameter is 0.  It turns out, however, this is not what I 
want.  When I run an LDAP search operation with say 80,000 entries, 
examining the program with truss (on Solaris) on strace (on RHEL) 
reveals that I end up going around that loop 80,000 times, I call 
select() 80,000 times, I call ldap_result() 80,000 times ... one per 
entry.  The behaviour I actually want is for ldap_result() to return all 
the items currently available in kernel buffers but without waiting for 
any new network packets to arrive.  The best approximation I can get for 
this is to use a timeout value of 0.0001.  Having tried various values, 
this is best for the hardware I'm testing with when OpenLDAP is 
listening on the loopback interface.  It reduces the number of times I 
called select() from 80,000 times to about 3,000 times, and shaved 5 
seconds off the total execution time.  However, I'm acutely aware that 
results will be hardware dependent, and will differ when I'm using a 
physical network interface.

So what I'm asking is, what is the best way of achieving the following 
workflow:

- start LDAP search
- start loop
- select() on FD, wake me up when there is data to read
- ldap.result() gives me ALL available data, and without waiting for 
anything new
- Attend to other unrelated events
- Go round loop again

Thanks & regards,
Mark Bannister.

http://dbis.sf.net
DBIS, a replacement for RFC2307