Re: (ITS#6090) slapd locks up; all slapd worker threads blocking on mutex acquisition in bdb_cache_lru_link()

[hyc@symas.com]

hyc@symas.com wrote:
> jwm@horde.net wrote:
>> Full_Name: John Morrissey
>> Version: 2.4.16
>> OS: Linux
>> URL: ftp://ftp.openldap.org/incoming/
>> Submission from: (NULL) (2001:4978:194:0:21f:5bff:fee9:da92)
>>
>>
>> After a couple days of uptime, slapd no longer responds to incoming connections
>> (the connection would be accepted, but all LDAP operations would block). All
>> worker threads seem to be blocking on mutex acquisition in bdb_cache_lru_link().
>> One thread was chewing lots of CPU.
>>
>> Backtrace is below. I also have a ~1.7GB core if it's deemed useful; I'll keep
>> it around for a week or two. This is with BDB 4.7.25+all three patches.
>>
> Interesting trace, it looks like all the active threads are waiting for the
> mutex but apparently none of them owns it. Can you please provide the contents
> of the mutex? e.g.
> 	thread 14
> 	frame 3
> 	print *mutex

Ah, I missed this before, your thread 3 is inside a BerkeleyDB lock function. 
There's nothing useful in the trace for thread 3 though. It seems you may need 
to recompile BerkeleyDB with debugging enabled (and with 
-fno-omit-frame-pointer) to get a useful trace from this. This is looking more 
like a BDB locking issue than an OpenLDAP issue. If you still have the 
environment, db_stat -CA would be helpful.

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/