Re: Setting TCP_NODELAY on TCP sockets

[dboreham@netscape.com (David Boreham)]

Gaël Roualland wrote:

> I am setting up a FreeBSD LDAP server and I was experiencing very poor
> performance (5 ldap_search/sec using only one connection and a small
> database) when querying the LDAP server (remotely and localy as well). I
> however noticed that I could get better performance (around 30 req/s)
> running my test client on a linux system.
> 
> After some experimentation (tcpdumping and so on) it camed out that the
> slow performance was due to the fact that ip paquets between the server
> and the client were smaller than the mss, and such the following ip
> paquets were not sent until the previous one was ack'ed. (thus the
> difference between freebsd and linux came from the difference in delay
> between acks in the tcp stack of both systems). Setting TCP_NODELAY on
> the socket in the server and the client solves the problem by disabling
> nagle algorithm, and improves performance a lot (up to 300 req/s now).

Indeed. Here's a mail from Nagle from a few years ago
when we discovered the same thing (the problem doesn't
happen between any two arbitrary nodes, you need one
which implements delayed ACK, and not all do):

> You write:
> >I (Nagle) wrote in comp.protocols.tcp-ip:
> >>     The interaction between the "Nagle algorithm"
> >>(actually I called it tinygram prevention) and delayed ACKs was badly
> >>thought out.  Delayed ACKs were put in after I got out of protocol
> >>design; I would never have done them with a fixed timer.
> 
> >My colleagues designed a protocol which fits your model exactly
> >(LDAP), and I've been wondering why we see some 300 searches
> >per second when the target is not present in the directory, but
> >5 per second (not 4.9 or 5.1, mind you) when it is. Turns out that
> >the second case involves two sends from the server, where the
> >first has one.
> >
> >Now I'm off to re-code the server so it spots when there's
> >more than one search result packet to send, and coalesces them.
> 
>      The "send,send,wait for reply" case kills performance.  Arguably,
> TCP should be fixed, but it's rather late for that.  On the other hand,
> application protocols that do "send,send,wait for reply" are doing something
> inefficient; coalescing at the application protocol level will provide
> better performance over the Internet.
> 
>      I understand that the new "persistent TCP connection" variant on HTTP
> currently under development may be prone to this problem.  Your people
> working on that might want to check.
> 
>                                            John Nagle
> 
>