[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: performance issue behind a a load balancer 2.3.32

Philip Guenther wrote:

On Tue, Jul 21, 2009 at 01:54:25PM -0700, Quanah Gibson-Mount wrote:

--On Tuesday, July 21, 2009 4:51 PM -0400 "Clowser, Jeff"<jeff_clowser@fanniemae.com>  wrote:

Do you have any facts/numbers to back this up?  I've never seen F5's
slow things down noticably.


We've had F5's be the root of the problem with several clients who load
balanced their LDAP servers, and pointed postfix at the F5 for delivery.
They added just a few milliseconds of time to each LDAP query, but that
was enough to completely back up their mail delivery system.<...>


Given the reported log message, this (latency) is very likely to be the
cause of the problem.  "connection deferred: binding" means that the
server received a request on a connection that was in the middle of
processing a bind.  This means that the client sends a bind and then
additional request(s) without waiting for the bind result.  That's a
violation by the client of the LDAP protocol specification, RFC 4511,
section 4.2.1, paragraph 2:

    After sending a BindRequest, clients MUST NOT send further LDAP PDUs
    until receiving the BindResponse.  Similarly, servers SHOULD NOT
    process or respond to requests received while processing a
    BindRequest.

The log message is slapd saying "I'm obeying that SHOULD NOT for this
connection, loser".  It should be obvious now why the conn_max_pending*
options have no effect.

Understanding _why_ clients are violating the spec by sending further
requests while a bind is outstanding may help you understand how the F5 or
the clients should be tuned (or beaten with sticks, etc).
All true, but in certain versions of OpenLDAP, slapd would send the Bind result to the client before it was done with its internal bookkeeping. So it's possible that, on a very busy slapd, a very fast well-behaved client could get the Bind result and send its next request before slapd was finished marking the connection as "no longer Binding". (See ITS#3850 and #6189). Still, none of this will result in much additional latency within slapd (beyond any latency already imposed by the CPU load, number of available threads, etc...) 


You presumably don't notice this under normal circumstances or with RR DNS
because the server completes the BIND before the next request is received.
My understanding (perhaps suspect) is that the F5 will increase the
'bunching' of packets on individual connections (because the first packet
after a pause will see a higher latency than the succeeding packets).

So, are you measuring latency through the F5?  I would *strongly* suggest
doing so *before* tuning the F5 in any way, such as by the VIP type
mentioned by John Morrissey, so that you can wave that in front of
management (and under the nose of the F5 saleman when negotiating your
next support renewal...)


--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/