Re: slapd problems under stress

[Stephan Siano <Stephan.Siano@suse.de>]

Hi,

Mark Valence wrote:

> >Our test program (written by using libldap) starts 80 async
> >request and waits for their results. This is done every
> >10 seconds. In case of a time-out (10 seconds) all pending
> >operations will be abandoned. Sometimes the slapd crashes
> >with a SEGV, sometimes it hangs, sometimes it aborts (from
> >assert() during a memory allocation)!
>
> Can you send the source for the test program?  Does the problem
> happen after replying to a bunch of search requests, or does it
> happen as soon as you start the test program?  Do the three cases
> (SEGV, hang, abort) happen with equal likelihood, or is one case more
> likely than the others?

The problem occurs after sending a quite big number of async requests
but before the slapd manages to answer any of them.

I can send you the test program (I don't think it is interesting enough
to send it to the list...)

[...]

> Can you try a test?  In libraries/libldap_r/tpool.c, add the
> following line after line 161:
>
>    return ldap_pvt_thread_create( &thr, 1, start_routine, arg );
>
> Then rebuild and try running your tests and see if the problems still happen.

OK, I can't test our backends at the moment (they require a customer's
database) but the behaviour seems to have changed with the ldbm backend.

The behaviour seems to have changed a little but I still managed to
crash the slapd with an access violation. I could still observe up to
960 instances of slapd in the process list. 
What does this line 161 do?

> >I think that the problems are thread related.
> >Are there any known problems? Is it possible to set a limit
> >on the number of concurrent threads?
>
> You can change line 97 in servers/slapd/init.c to something like:
>
>    ldap_pvt_thread_pool_init(&connection_pool, 5, 0);
>
> where the second parameter is the maximum number of concurrent threads.

That seems to improve things a lot! I couldn't crash the system with the
ldbm-backend anymore if I apply that patch. I even get answers to all
the requests :-). I'll try our backends on monday.

Shouldn't the maximum number of concurrent be a configurable parameter?

Yours,
Stephan Siano

-- 
Stephan Siano                           Mail:  Stephan.Siano@suse.de
SuSE Linux Solutions AG                 Phone: 06196 50951 31
Mergenthalerallee 45-47			Fax:   06196 409607
D-65760 Eschborn