(ITS#6660) paged result searches fail to deallocate memory until slapd shutdown

[quanah@zimbra.com]

Full_Name: Quanah Gibson-Mount
Version: 2.4.23
OS: Linux 2.6
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (75.111.45.108)


When using paged result searches with large databases, it quickly becomes
apparent that slapd is failing to deallocate memory allocated during those
searches once they complete.  The example database I am using has 6,326,513
entries, and is some 8.5GB on disk (id2entry.bdb + dn2id.bdb).

What has been found is that over time when using multiple paged results
searches, slapd's memory continually grows until it eventually consumes all
memory on the system.  Once slapd is stopped, the allocated memory is freed. 
This behavior can be seen in a more pronounced fashion by running multiple paged
results searches and searches of the full database simultaneously.

For example, I start slapd, and preload all the caches by doing 4 full searches
of the database.  At this point, top reports (V/R/S):

12.2G/10G/8.4G

I then start 10 ldap searches dumping the full database (using root dn), and 10
paged result searches that only return some 2.6 million entries.  After these
all complete, I run this sequence again.  Instead of slapd's memory remaining
stable (i.e., re-using all the memory it had previously allocated), it instead
continues to grow.  At the end of just 2 of these runs, we end up at:

22.5G/20G/8.4G

Or a 10.3GB growth in virtual memory and 10GB growth in resident memory.

As soon as slapd is halted, all of this memory is freed (Making it impossible to
track this via valgrind).

I'm currently using the heap profiler shipped with tcmalloc to gather further
information and will follow up when that is complete.

--Quanah