RE: Slapd frontend performance issues

[Sam Drake <drake@timesten.com>]

I've also been doing quite a bit of work on the Back-sql code to optimize
it, and now have Back-sql running at 100 queries/second against our
in-memory relational database, TimesTen.  Your 6 queries/sec number pretty
much matches what I saw  using Oracle with back-sql out-of-the-box as well.
I'm using similar Sun hardware to Eric's case.

I have a long list of additional enhancements to make, including using
connection pools as Eric suggests.  I haven't got around to that one yet.

I'll send additional email to the list in the next day or two describing the
changes we've made, none of which are specific to TimesTen (they should
improve the performance with most backend ODBC databases).  I'd be happy to
make the changes public to Dmitry and/or anyone else who would like to try
them.  If anyone's interested in details sooner, please let me know.

...Sam Drake / Manager, Product Integration / TimesTen Performance Software
...email: drake@timesten.com / www.timesten.com




> -----Original Message-----
> From: Eric T. Blue [mailto:openldap@star-cs.com]
> Sent: Friday, April 06, 2001 1:50 PM
> To: openldap-devel@OpenLDAP.org
> Subject: Slapd frontend performance issues
> 
> 
> I've been working with Dmitry over the last couple months in 
> trying to fine
> tune back-sql.  One avenue I'm looking at is implementing a backend
> connection pool for the RDBMS, and possibly porting the ODBC 
> code to Oracle
> specific OCI.  I've done a good bit of benchmarking so far 
> with the ldbm
> backend, as well as back-sql.  The other day I decided to 
> start looking at
> the performance of the OpenLDAP server as a whole, 
> eliminating the backend,
> just to see what type of numbers I could expect.  Here are 
> some numbers
> benchmarked on a SUN E220 2/450Mhz procs w/ 2GB RAM:
> 
> This is done via an in-house tool I wrote, but the numbers are almost
> identical to those reported from Mindcraft's DirectoryMark.
> non-threaded single client:
> 
> BerkleyDB backend: 217 queries / second
> BACK-SQL backend: 6 queries / second
> 
> Now, I first tried to make a dummy backend myself which just 
> defined the
> initialize routine from backend.c.  In the init code, I 
> assigned 0 to all
> fcns except for the search fcn.  In the search fcn all I did 
> was either
> return an LDAP_SUCCESS message, or an LDAP_OTHER with some 
> text to see what
> was going on.  The result was about 300 q / s.  Now, to give 
> a more accurate
> stress test, I ran 20 clients in parallel which did a bind, 
> search, and
> unbind for every operation.  Combining the results of all 20 clients
> increases the number to show about 570 q /s with no backend.  
> I also tried
> returning the LDAP_SUCCESS directly from search.c in 
> servers/slapd.  This
> number seems low to me, and I was wondering if there is 
> anything I can do to
> increase performance?  For some reason, I made an assumption that the
> "slowness" in throughput was due to disk or protocol IO for 
> the backend
> processing... maybe not.  My ultimate goal, which may be 
> unrealistic at this
> point, is to achieve a MAX number of 1,000 q / s while using 
> OCI.  What I'd
> like to do is establish a connection pool of x number of 
> persistent OCI
> connections to Oracle on DB startup and assign each LDAP handle to a
> particular OCI handle for the connection.  I'd appreciate any insight
> anybody can offer.  You guys have been very helpful so far.  Thanks!
> 
> -----Original Message-----
> From: mitya@samovaar.dot.ru [mailto:mitya@samovaar.dot.ru]On Behalf Of
> Dmitry Kovalev
> Sent: Friday, April 06, 2001 5:55 AM
> To: Eric T. Blue
> Subject: Re: back-sql / oc question
> 
> 
> Hello Eric!
> 
> "Eric T. Blue" wrote:
> 
> > Dmitry,
> >         I haven't had a chance to profile it yet, but I may 
> have some
> dissapointing
> > news.  Up till now, I've assumed the a great deal of 
> processing is spent
> on
> > the backend (regardless if it is ldbm, or back-sql).  So, I 
> decided to
> make
> > dummy backend that really does nothing... just inits 
> function pointers to
> 0.
> > I've also changed search.c in the servers/slapd directory to give an
> > LDAP_SUCCESS message and return instantly.  If I do this 
> and then give a
> > stress test to the server (ie. 10 clients connecting 
> simultaneously with
> > 1,000 searches each) I only can achieve 300 queries per 
> second.  If this
> is
> > the case, then obviously I can never expect to achieve a 
> number greater
> than
> > this(or even 25% of this assuming that an OCI backend is 
> put in place).
> > This number is very surprising to me, and I can't believe 
> that the core
> > OpenLDAP code is this sluggish.. what do you think?
> >
> 
> Well, this is too strange (all it have to do is choose a 
> working thread,
> parse the
> request, and send OK), but after all - I have no knowledge of slapd
> frontend - I just
> plugged in my backend, thats all...
> 
> 
> There are a number of tunings that can be done - worker 
> thread pool size
> etc., of
> which I don't know much...
> 
> We should contact somebody from the real core - because my 
> core membership
> is limited
> to back-sql support - I don't have much time even for this nowadays :(
> 
> Namely, Kurt D. Zeilenga appears to be the chief architect - we could
> forward this to
> him, with questions regarding slapd frontend...
> 
> WBW, Dmitry
>