RE: server side sorting & paging?

["Armbrust, Daniel C." <Armbrust.Daniel@mayo.edu>]

That is a problem we have run into as well.  The lack of server side sorting has been a disappointment to us as well.  Unlike most users who post here, we are not using openldap for a user directory.  We are using it as a  hierarchal database, where some of our datasets have millions of entries.  Until the server side paging was implemented, we had pretty much exceeded the capacity of openldap.  Now we have a bit of breathing room again (though I still need to try to finish tracking down the last of the bugs in the paging methods).

I don't believe that the argument that results don't need to be sorted holds any water - in every single user application I have ever written, whenever a list of data is shown to the end user, if it is not sorted, this is the first thing that will be asked for.  And if you have to go through a usability lab with you applications, you will never be allowed through without it.

I understand the arguments for not doing server side sorting, and indeed, we have many use cases where we don't need sorting ourselves.  But when we are presenting the data to a user application, sorting is a requirement, full stop.  So, now it becomes a question of how to do it (and where - client or server)

Currently with data sets of our size, trying to write applications in such a way that they client side sort is highly painful - as the number of characters you must search with has to be >3 (in order to maintain any performance) and we much dynamically choose the number of characters we search with due to varying data sizes, and we will either end up with far to many results to sort in the client (or even return in a reasonable amount of time, for that matter) or, if we hit a sparse area, we have to do hundreds of searches to get enough results to display.  All of this results in very slow response times - and it creates server dependant code - as when we move over to IBM's Tivoli, none of it is needed.

Another way to look at the argument of whether or not the server should support sorting, is to ask the question, would you even think about using a sql database that didn't support sorting?  I can jump over to MySql right now, and given an indexed column, I can pull up 100 sorted results from a 10 million row table in a fraction of a second.  If I try to do that with the same data in openldap, I will have to go through a complex client side process, and I know I can't do it in subsecond time (I've tried).  There simply is no fast way to handle large amounts of data like this without indices.  The same arguments given here against making server side sorting available in openldap could be applied to sql - yet nearly every database out there supports sorting.

For our use case, even the simplest server side sort would be useful to us.  Meaning, one that only worked on the local server, and simply maintained a set of pointers that weren't updated when changes happen on the database.  

Issues like this, plus other scalability problems that we are encountering are all things we are having to take into account as we evaluate whether or not we should continue using a ldap database, or move to something else.  It's not a simple decision for us, as there are a lot of advantages with ldap for us.  And the fact that Tivoli doesn't give us the problems with scalability that we have encountered so far with openldap gives us reason to stay with ldap.  But we also want to stay with a open and free server (much of our work is or will be open sourced in the future).  

Just my 2c (maybe 3, I got kind of wordy)

Of course, if I really wanted to be useful, I should learn C better, and try implementing it myself instead of just complaining about it... But I keep running into that problem that I can't add hours to the day :)

Dan