[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: slapd hang once a day (fwd)

To: antidotez@catcha.com, sergio.pereira@edb.ericsson.se
Subject: Re: slapd hang once a day (fwd)
From: kunkee@neosoft.com (Randy Kunkee)
Date: Thu, 13 Jul 2000 16:14:20 -0500 (CDT)
Cc: "S.T. Wong" <st@hp735c.csc.cuhk.edu.hk>, openldap-software@OpenLDAP.org
In-reply-to: <20000713170817.33212.qmail@nicole.catcha.com> from "antidotez@catcha.com" at "Jul 13, 2000 05:08:17 pm"

I wonder if it is coincidence that both of you are reporting a similar
problem (auth services cease after a while) and that both of you are
using ldap to authenticate squid.  Maybe I've way off base tying the
two together.  It is quite likely that one of your clients (perhaps
a client the two of you have in common) is "brain-damaged", as Kurt
suggested, and is failing to close its connection once the authentication
is done.  The display below should be re-run every few hours of operation
to see what grows.  Another possibility is to take applications out
of service one at a time for an hour or so, while carefully monitor the
data below to check for file descriptor leaks.  When the leak stops, you
know you have the culprit.

For my own experience with OpenLDAP, I can assure you that there are no
such leaks in OpenLDAP 1.2.x.

Randy


> Quoting Randy Kunkee <kunkee@neosoft.com>:
> 
> Thanks, we've done as adviced and following is the result. How should we read 
> the information below? Most of them are sockets rather than files. Your help 
> in analysing the result is greatly appreciated.
> 
> [Fri Jul 14 00:52:45]root@aquarius:/proc# ls -l 20665/fd | less
> lrwx------    1 root     root           64 Jul 14 00:52 0 -> /dev/null
> l-wx------    1 root     root           64 Jul 14 00:52 1 -> 
> pipe:[7406361]
> lrwx------    1 root     root           64 Jul 14 00:52 10 -> 
> /etc/openldap/user
> db/dn2id.dbb
> lrwx------    1 root     root           64 Jul 14 00:52 100 -> 
> socket:[7413858]
> lrwx------    1 root     root           64 Jul 14 00:52 101 -> 
> socket:[7415379]
> lrwx------    1 root     root           64 Jul 14 00:52 102 -> 
> socket:[7416352]
> lrwx------    1 root     root           64 Jul 14 00:52 103 -> 
> socket:[7417422]
> lrwx------    1 root     root           64 Jul 14 00:52 104 -> 
> socket:[7419554]
> lrwx------    1 root     root           64 Jul 14 00:52 105 -> 
> socket:[7421151]
> lrwx------    1 root     root           64 Jul 14 00:52 106 -> 
> socket:[7420639]
> lrwx------    1 root     root           64 Jul 14 00:52 107 -> 
> socket:[7417866]
> lrwx------    1 root     root           64 Jul 14 00:52 108 -> 
> socket:[7421024]
> lrwx------    1 root     root           64 Jul 14 00:52 109 -> 
> socket:[7420488]
> lrwx------    1 root     root           64 Jul 14 00:52 11 -> 
> /etc/openldap/user
> db/id2entry.dbb
> lrwx------    1 root     root           64 Jul 14 00:52 110 -> 
> socket:[7419017]
> lrwx------    1 root     root           64 Jul 14 00:52 111 -> 
> socket:[7421187]
> lrwx------    1 root     root           64 Jul 14 00:52 112 -> 
> socket:[7420445]
> lrwx------    1 root     root           64 Jul 14 00:52 113 -> 
> socket:[7417884]
> lrwx------    1 root     root           64 Jul 14 00:52 114 -> 
> socket:[7418489]
> lrwx------    1 root     root           64 Jul 14 00:52 115 -> 
> socket:[7420490]
> 
> 
> > You may have a new bug.
> >
> > Most linux systems have a proc filesystem that you can go to as root
> > to examine the open files a process has.  For example, if the pid of
> > slapd is 100, you should be able to "ls -l /proc/100/fd", which will
> > give you a listing of devices and inode numbers that are open by the
> > process.
> >
> > You can then do a "find / -inum #" where # is the inode number of
> > the file.  In the case of slapd, the number of places the file could
> > be open is rather small, so you can drasticly reduce the find runs
> > by narrowing down the directories, for example:
> >
> > find /etc /dev /(path to ldbm files) /(path to log files) -inum #
> >
> > Once you know what file is not getting closed, you'll know where to go
> > to fix the problem.  Just because /etc/hosts.allow is the one you get
> > the error on does not necessarily mean that it's the one not getting
> > closed.
> >
> > > Unable to apply the patch as the link given in ITS is no longer valid.
> > Can
> > > someone pls advise if the patch is needed for OpenLDAP-1.2.9-5 and
> > OpenLDAP
> > > 1.2.11? If needed, where else can I find the patch or someone can email
> > me
> > > pls.
> > >
> > > Below is a detailed configuration of our server setup and hope it'll help
> > to
> > > diagnose the problem better.
> > >
> > > 1. Version of OpenLDAP
> > > OpenLDAP-1.2.9-5
> > >
> > > 2. Operating System
> > > RedHat Linux 6.2 with kernel 2.2.15
> > >
> > > 3. Hardware specifications
> > > Intel Pentium III 550 MHz 512MB RAM
> > >
> > > 4. User population
> > > Approximately 4500
> > >
> > > 5. Applications/services running on server
> > > LDAP, BIND8, SSH, APACHE, SSL, PHP, SENDMAIL, IMAP, POP,
> > > SQUID, MySQL
> > >
> > > 6. Applications used by users to access above mentioned services
> > > MS Outlook/Outlook Express, pine, Netscape mail/navigator, IE
> > >
> > > 7. Error found in /var/log/messages as:
> > >
> > > Jul 12 12:40:05 server1 slapd[32521]: warning: cannot open
> > > /etc/hosts.allow: Too many open files
> > > Jul 12 12:40:05 server1 slapd[32521]: warning: cannot open
> > /etc/hosts.deny:
> > > Too many open files
> > >
> > > Quoting "S.T. Wong" <st@hp735c.csc.cuhk.edu.hk>:
> > >
> > > > Hello,
> > > >
> > > > We're running OpenLDAP 1.2.11 on Solaris 2.6, and have the same
> > problem.
> > > > Did the patches mentioned in the ITS cases mentioned incorporated in
> > 1.2.11
> > > > ?
> > > >
> > > > Thanks a lot.
> > > > Regards,
> > > > ST Wong
> > > >
> > > > > You likely are using brain damaged clients.
> > > > >
> > > > > See ITS#237, ITS#169, ITS#142, and ITS#30.
> > > > >
> > > > > Kurt
> > > > >
> > > > > At 05:09 PM 7/11/00 +0000, antidotez@catcha.com wrote:
> > > > > >recently, ldap (slapd) has been giving problem. it will
> > > > > >hang at least once every day and needs manual killing
> > > > > >and restaring. below is an entry from log file:
> > > > > >"Jul 10 09:21:55 testserver slapd[19503]: warning:
> > > > > >cannot open /etc/hosts.allow: Toomany open files"
> > > > > >
> > > > > >would appreciate if someone could advise on the
> > > > > >cause for "cannot open /etc/hosts.allow" and "too many
> > > > > >open files" and the solution to the hanging problem.
> > > > > >
> > > > > >is there any tool/prog that can be used to monitor slapd
> > > > > >and email/page when the process hang?
> > > > > >
> > > > > >your help is greatly appreciated.
> > > > > >
> > > > > >thank you very much
> > > > > >
> > > > > >regards
> > > > > >
> > > > > >
> > > > > >-----------------------------------------------------
> > > > > >Get your FREE email from Catcha! http://mail.catcha.com
> > > >
> > >
> > >
> > > -----------------------------------------------------
> > > Get your FREE email from Catcha! http://mail.catcha.com
> > >
> >
> > 
> 
> 
> -----------------------------------------------------
> Get your FREE email from Catcha! http://mail.catcha.com
>

Follow-Ups:
- Re: slapd hang once a day (fwd)
  - From: "S.T. Wong" <st@hp735c.csc.cuhk.edu.hk>

References:
- Re: slapd hang once a day (fwd)
  - From: antidotez@catcha.com

Prev by Date: squid_ldap_auth authentication script
Next by Date: Re: squid_ldap_auth authentication script
Index(es):
- Chronological
- Thread