[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Re: Memory consumption issue
On Mon, Nov 17, 2008 at 02:43:48PM +0100, tkohlhepp wrote:
> I've discovered a major memory issue. We have an OpenLDAP server
> (2.4.11) running with almost 2 million entries. When I do a ldapsearch
> to retrieve the entire tree the memory consumption grows and grows and
> will never stop before it ate the entire RAM and swap.
The figures you give suggest that the maximum size reached is under 1GB.
That does not seem bad for 2M entries.
Retrieving 2M entries in a single operation is going to tax any LDAP
server, especially if you do not request paged results. Consider what
it must do:
1) Make a list of every entry ID
2) Retrieve the data for every entry
3) Build a message containing 2M entries
4) Send the message
Steps 2 and 3 are going to consume a *lot* of memory. Just looking at
the number of attributes you are indexing, I suspect that your entries
are over 1kb each. That gives a result message of 2M * 1kb = 2GB (at least).
All of this must fit into memory before it can be sent.
> #dbcachesize 1000000
> cachesize 1000
> idlcachesize 3000
> cachefree 500
> searchstack 8
> The DB_CONFIG look like this
>
> set_cachesize 0 8435456 1
> set_lg_regionmax 262144
> set_lg_bsize 2097152
All of those numbers seem very small for a directory with 2M entries.
It depends how you use it of course, but 'fetch every entry' is going
to kill it.
> When I start slapd is consumes that much RAM
> total kB 478476
>
> After doing the search it looks like this
> total kB 937316
If that represents 'all of memory and all of swap' then you need to
add more of both!
If the normal service runs OK with these numbers but it is just
the 'retrieve all' operation that gives trouble, then why not find
a different way to do that one job? slapcat perhaps?
Andrew
--
-----------------------------------------------------------------------
| From Andrew Findlay, Skills 1st Ltd |
| Consultant in large-scale systems, networks, and directory services |
| http://www.skills-1st.co.uk/ +44 1628 782565 |
-----------------------------------------------------------------------