Slapd frontend performance issues

["Eric T. Blue" <openldap@star-cs.com>]

I've been working with Dmitry over the last couple months in trying to fine
tune back-sql.  One avenue I'm looking at is implementing a backend
connection pool for the RDBMS, and possibly porting the ODBC code to Oracle
specific OCI.  I've done a good bit of benchmarking so far with the ldbm
backend, as well as back-sql.  The other day I decided to start looking at
the performance of the OpenLDAP server as a whole, eliminating the backend,
just to see what type of numbers I could expect.  Here are some numbers
benchmarked on a SUN E220 2/450Mhz procs w/ 2GB RAM:

This is done via an in-house tool I wrote, but the numbers are almost
identical to those reported from Mindcraft's DirectoryMark.
non-threaded single client:

BerkleyDB backend: 217 queries / second
BACK-SQL backend: 6 queries / second

Now, I first tried to make a dummy backend myself which just defined the
initialize routine from backend.c.  In the init code, I assigned 0 to all
fcns except for the search fcn.  In the search fcn all I did was either
return an LDAP_SUCCESS message, or an LDAP_OTHER with some text to see what
was going on.  The result was about 300 q / s.  Now, to give a more accurate
stress test, I ran 20 clients in parallel which did a bind, search, and
unbind for every operation.  Combining the results of all 20 clients
increases the number to show about 570 q /s with no backend.  I also tried
returning the LDAP_SUCCESS directly from search.c in servers/slapd.  This
number seems low to me, and I was wondering if there is anything I can do to
increase performance?  For some reason, I made an assumption that the
"slowness" in throughput was due to disk or protocol IO for the backend
processing... maybe not.  My ultimate goal, which may be unrealistic at this
point, is to achieve a MAX number of 1,000 q / s while using OCI.  What I'd
like to do is establish a connection pool of x number of persistent OCI
connections to Oracle on DB startup and assign each LDAP handle to a
particular OCI handle for the connection.  I'd appreciate any insight
anybody can offer.  You guys have been very helpful so far.  Thanks!

-----Original Message-----
From: mitya@samovaar.dot.ru [mailto:mitya@samovaar.dot.ru]On Behalf Of
Dmitry Kovalev
Sent: Friday, April 06, 2001 5:55 AM
To: Eric T. Blue
Subject: Re: back-sql / oc question


Hello Eric!

"Eric T. Blue" wrote:

> Dmitry,
>         I haven't had a chance to profile it yet, but I may have some
dissapointing
> news.  Up till now, I've assumed the a great deal of processing is spent
on
> the backend (regardless if it is ldbm, or back-sql).  So, I decided to
make
> dummy backend that really does nothing... just inits function pointers to
0.
> I've also changed search.c in the servers/slapd directory to give an
> LDAP_SUCCESS message and return instantly.  If I do this and then give a
> stress test to the server (ie. 10 clients connecting simultaneously with
> 1,000 searches each) I only can achieve 300 queries per second.  If this
is
> the case, then obviously I can never expect to achieve a number greater
than
> this(or even 25% of this assuming that an OCI backend is put in place).
> This number is very surprising to me, and I can't believe that the core
> OpenLDAP code is this sluggish.. what do you think?
>

Well, this is too strange (all it have to do is choose a working thread,
parse the
request, and send OK), but after all - I have no knowledge of slapd
frontend - I just
plugged in my backend, thats all...


There are a number of tunings that can be done - worker thread pool size
etc., of
which I don't know much...

We should contact somebody from the real core - because my core membership
is limited
to back-sql support - I don't have much time even for this nowadays :(

Namely, Kurt D. Zeilenga appears to be the chief architect - we could
forward this to
him, with questions regarding slapd frontend...

WBW, Dmitry