Re: (ITS#4640) slapd lockup

[quanah@stanford.edu]

--On Wednesday, August 16, 2006 7:29 PM -0700 Quanah Gibson-Mount 
<quanah@stanford.edu> wrote:

>
>
> --On Wednesday, August 16, 2006 8:33 AM +0000 openldap-its@OpenLDAP.org
> wrote:
>
>
> This has happened again, and now I'm seeing slapd grow at crazy rates
> even though I stopped incoming connections to it by dropping it out of
> the load balance pool:

I applied Howard's patches from HEAD to daemon.c and connection.c, and then 
deployed that build to ldap4.  Once again, the system went haywire, but 
this time something interesting was logged:

Aug 17 05:30:03 ldap4 slapd[15598]: slapd(79): setsockopt(SO_KEEPALIVE) 
failed errno=9 (Bad file descriptor)
Aug 17 05:30:03 ldap4 slapd[15598]: slapd(79): setsockopt(TCP_NODELAY) 
failed errno=9 (Bad file descriptor)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=16915 fd=79 closed
Aug 17 05:30:03 ldap4 slapd[15598]: conn=16917 fd=94 closed (connection 
lost)
Aug 17 05:30:03 ldap4 slapd[15598]: connection_init(79, 
IP=171.64.15.82:58841): set nonblocking failed
Aug 17 05:30:03 ldap4 slapd[15598]: daemon: epoll_ctl(ADD,fd=79) failed, 
errno=9, shutting down
Aug 17 05:30:03 ldap4 slapd[15598]: conn=16918 fd=79 ACCEPT from 
IP=171.64.15.82:58841 (IP=0.0.0.0:389)
Aug 17 05:30:03 ldap4 slapd[15598]: daemon: abnormal condition, shutdown 
initiated.
Aug 17 05:30:03 ldap4 slapd[15598]: conn=13177 fd=16 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=8025 fd=23 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=16062 fd=24 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=12907 fd=25 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=97 fd=26 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=9104 fd=27 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=14248 fd=28 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=80 fd=29 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=11106 fd=30 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=16013 fd=31 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=408 fd=32 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=5514 fd=33 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=16164 fd=35 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=419 fd=36 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=16534 fd=37 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=529 fd=38 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=7884 fd=39 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=11107 fd=40 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=508 fd=41 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=7258 fd=42 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=12348 fd=43 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=8096 fd=45 closed (slapd shutdown)
Aug 17 05:30:03 ldap4 slapd[15598]: conn=256 fd=46 closed (slapd shutdown)


Now I thought that we'd fixed the bad file descriptor problem a few 
releases ago?  I wonder if there is a problem with epoll.

--Quanah

--
Quanah Gibson-Mount
Principal Software Developer
ITS/Shared Application Services
Stanford University
GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html