Re: efficiency of queries returning many entries

[Randy Kunkee <randy@randallkunkee.com>]

Mark Whitehouse wrote:

> Hi,
>
> I have the following configuration:
>
>    RH Linux 6.2
>    openldap 2.0.7
>    berkley db 3.2.9
>
> I have a DB with approx 100K entries.  Most of my queries are fast.
> However, whenever I submit a query which contains a large result set (i.e.
> ~10,000 entries) the query takes approx 80 secs.
>
> The query in question is simple and all the attributes in the query are
> indexed, so I assume the reason the query takes so long is the volume of
> data being processed??
>
> My question is....  How do I make queries which will return large numbers of
> entries efficient?  How do search engines (Yahoo / Google) do this, I have
> noticed that broad queries to search engine sites (which return many
> thousands of queries) are very fast.  Do they use some sort of paging
> mechanism where the results of the query are retrieved one part at a time?
> Does any similar mechanism exist in OpenLDAP?
>
> Any help would be appreciated.
>
> Thanks,
> Mark

I don't think LDAP would be a good way to implement a search engine.  Something
like a WAIS database would be more appropriate.  I doubt the vendors you refer
to use LDAP for the search engine, even if they use it to hold certain kinds of
information.

Randy