Re: (ITS#4888) Enhance back-sql with paged results

[ando@sys-net.it]

hyc@symas.com wrote:
> adamson@cmu.edu wrote:
>> Full_Name: Mark Adamson
>> Version: HEAD
>> OS: Solaris 2.8
>> URL: http://null.andrew.cmu.edu/openldap/sql_paged_results.patch
>> Submission from: (NULL) (128.2.234.24)
>>
>>
>> Here at Carnegie Mellon we had a need to use the LDAP_CONTROL_PAGEDRESULTS
>> control with the SQL backend. The patch to add this is not hard, and was mostly
>> copied from the back-bdb/search.c file.
>>
>> http://null.andrew.cmu.edu/openldap/sql_paged_results.patch
>>
>> The only challenge was the contents of the cookie that is used to maintain
>> state. The slap.h file defines the PagedResultsCookie type to be an unsigned
>> long, so I wanted to fit everything into that. The design idea was to keep the
>> ldap_entries.id value of the last returned entry in the cookie so that
>> subsequent searches could add a "AND ldap_entries.id > COOKIEVALUE" clause to
>> the SQL query. However, the gathering of candidate entries is done across all
>> objectClasses with avl_apply(), so if one OC returns an entry with a high
>> ldap_entries.id, objectClasses that come later in the AVL could miss entries.
>>   For this reason, I add the oc_map_id of the last returned entry to the cookie,
>> and I adjusted the avl_apply() call to run through the objectClasses in numeric
>> order. In this way, when subsequent pages of entries are requested, all of the
>> objectClasses that were completed by previous pages can be skipped for the new
>> page. When we get to the objectClass where the previous page left off, we take
>> the ldap_entries.id from the cookie and apply the above mentioned SQL clause.
> 
> A more appropriate change would probably be to change the cookie type to 
> a berval so that arbitrary data may be used as needed by each backend. 
> Then you can set both the oc_map_id and the ldap_entries.id.
> 
> Ultimately this sort of thing should be handled almost entirely by the 
> frontend, with just a single call in the backend to fill a cookie struct 
> for the current search state.

I concur (was about to suggest the same: move parse_paged_cookie() and
send_paged_response() to servers/slapd/result.c, adding as much as
required to customize it from within the backend specific call.

Let me note that the patch doesn't build fine (you need to declare in
advance the static functions you're only defining at the end), and it
seems to work incorrectly.  By running the simple sql-test000 with mysql
(which fails because now the order has changed; my fault: should have
added -S "" to searches), if I search without the control, I correctly
get 6 entries and a referral; if I search with paged results and a page
size less than 6, I only get 5 entries.  The root entry is missing, but
I have no idea of the reason, I couldn't infer anything from the code or
the logs.  Can you check?

p.



Ing. Pierangelo Masarati
OpenLDAP Core Team

SysNet s.n.c.
Via Dossi, 8 - 27100 Pavia - ITALIA
http://www.sys-net.it
------------------------------------------
Office:   +39.02.23998309
Mobile:   +39.333.4963172
Email:    pierangelo.masarati@sys-net.it
------------------------------------------