[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: slapd memory and performance problems





--On Tuesday, April 01, 2003 5:25 PM -0800 Mike Denka <mdenk@whidbey.net> wrote:

After experimenting with threads (max number and idletimeout), upgrading
to v.2.1.16 and applying Howard's new cvs version of tpool.c, I am still
experiencing some performance issues with slapd.  The most immediate
problem I see is that the response time for a query under peak loads is
very slow.  The most obvious oddity is that the slapd threads are
growing in size over time.

I am currently running four load balanced mail servers which query the
ldap server once for each incoming message to extract the user's mail
quota.  The user's entry is indexed by uid and a user defined attribute
in the user's entry stores the mail quota.  At peak times I am getting a
combined total of between 1 and 4 queries per second for these quotas
from the mail servers.  A search on this server during peak load times
takes several seconds (10 to 20) to complete.  If I try to add
additional loads on this same server (e.g., radius authentication making
maybe one query every two to 4 seconds), queries start timing out and
applications (like radius) making those queries start to fail.

Upon starting slapd, each thread has a size of about 4800KB.  This
thread size seems to double after 8 hours or so and grows to about 15M.
Also, the total memory available on the box seems to degrade over time,
not just from the increased thread size, but from what appears to be a
memory leak somewhere since just restarting slapd does not increase the
available amount of system memory.  Only a reboot recovers system RAM.

I'm running RedHat 8.0 on a platform with an Intel 1G processor and
512MB of RAM.  Processor idle time never dips below 90%, no swap space
is ever used but memory, as I said, diminishes over time.  However,
performance stays constant and doesn't change as memory degrades (until
memory drops to below 5M or so, at which time the server must be
rebooted).

Each query for a mail quota does a simple bind via the ldap:// url using
the entire dn as the base and '(objectclass=*)' as the filter.  This
base and filter is composed internally by Exim (our MTA).

I am considering using Unix sockets as was suggested in an earlier
thread but I don't understand that approach enough to engage it yet.

Could anyone comment on the performance of this box?  Is 1 to 4
queries/second the best I  can expect on a system of this kind?  Any
ideas what might be causing the memory in the slapd threads to grow over
time?  Slapd is the only user application running on this machine.  I
can't be sure that slapd is causing the memory leak, but I don't see
this problem on any of our other similar machines.  Are there any known
issues involving memory leaks in this latest version of slapd?  Also,
could someone comment on the efficacy of using unix sockets and point me
to some docs somewhere explaining them?

Mike,

I can't comment on the redhat portion of this, as we are running under Solaris 8, but I can offer the following:

We began a production rollout of OpenLDAP today, using 2.1.16 with the tpool.c patch. So far, only 1 mail router has been cut over to one machine. It is doing about 10,000 queries/hour. So far, I have seen no degradation of performance, and the memory footprint is remaining constant.

A big difference between our setups, is that we have sendmail using GSSAPI to bind, as we hacked our sendmail to use GSSAPI authentication. In load testing of these systems, I consistently saw 66 queries/second as the peak for the particular hardware we are using, and that was leaving the test running overnight, with over 1 million queries made in about 6 hours against the machine (18 querying clients). The hardware is:

SunFire v120 with 650MHz CPU & 4GB of memory.

I will keep you posted if we encounter any problems with threads. Maybe Redhat's thread implementation is buggy. I have had performance problems using Solaris 9, due to a change in Sun's thread implementation.

For indexing, we index the filter (essentially uid) and the attribute we are searching for (maildrop) with eq as well. Our searches continue to take less than a second to complete.

Software wise, we are running:

OpenLDAP 2.1.16 + tpool.c patch
Cyrus-SASL 2.1.12
Heimdal-0.5.2
BDB 4.1.25
OpenSSL 0.9.6i

--Quanah

--
Quanah Gibson-Mount
Senior Systems Administrator
ITSS/TSS/Computing Systems
Stanford University
GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html