Re: (ITS#6029) slapadd lockup

[quanah@zimbra.com]

--On Saturday, March 21, 2009 7:09 PM -0700 Howard Chu <hyc@symas.com> 
wrote:

> quanah@zimbra.com wrote:
>> Full_Name: Quanah Gibson-Mount
>> Version: RE24 3/19/2009
>> OS: Linux 2.6
>> URL: ftp://ftp.openldap.org/incoming/
>> Submission from: (NULL) (75.111.29.239)
>>
>>
>> While testing various load scenarios, slapadd deadlocked on me.
>
> The trace for thread 1 and 2 look identical, so I'm assuming they're the
> same thread. In that case I see 3 tool threads running. How many did you
> have configured? If 1 and 2 are really different threads (certainly their
> thread IDs look quite different), then there's something very wrong in
> the thread library...
>
> print bdb_tool_index_tcount
>

(gdb) print bdb_tool_index_tcount
$1 = 1


> It looks like one thread has exited early. I'm not sure how that can
> happen since the only way for any index thread to exit is if
> slapd_shutdown is set. Is it set?

(gdb) print slapd_shutdown
$2 = 0


It's configured with 4 tool threads.

--Quanah

>> (gdb) thr apply all bt
>>
>> Thread 5 (Thread 1106889024 (LWP 9361)):
>> # 0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> # 1  0x00002b7819f27de9 in bdb_tool_trickle_task (ctx=<value optimized
>> # out>,
>> ptr=<value optimized out>) at tools.c:1124
>> # 2  0x00002b781715a2ca in ldap_int_thread_pool_wrapper
>> # (xpool=0x17e07fd0) at
>> ../../../libraries/libldap_r/tpool.c:663
>> # 3  0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0
>> # 4  0x000000395e0d1e3d in clone () from /lib64/libc.so.6
>>
>> Thread 4 (Thread 1115281728 (LWP 9362)):
>> # 0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> # 1  0x00002b7819f28163 in bdb_tool_index_task (ctx=<value optimized
>> # out>,
>> ptr=<value optimized out>) at tools.c:1147
>> # 2  0x00002b781715a2ca in ldap_int_thread_pool_wrapper
>> # (xpool=0x17e07fd0) at
>> ../../../libraries/libldap_r/tpool.c:663
>> # 3  0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0
>> # 4  0x000000395e0d1e3d in clone () from /lib64/libc.so.6
>>
>> Thread 3 (Thread 1123674432 (LWP 9363)):
>> # 0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> # 1  0x00002b7819f28163 in bdb_tool_index_task (ctx=<value optimized
>> # out>,
>> ptr=<value optimized out>) at tools.c:1147
>> # 2  0x00002b781715a2ca in ldap_int_thread_pool_wrapper
>> # (xpool=0x17e07fd0) at
>> ../../../libraries/libldap_r/tpool.c:663
>> # 3  0x00002b78178fe2f7 in start_thread () from /lib64/libpthread.so.0
>> # 4  0x000000395e0d1e3d in clone () from /lib64/libc.so.6
>>
>> Thread 2 (Thread 1132067136 (LWP 9360)):
>> # 0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> # 1  0x00002b7819f28014 in bdb_tool_index_add (op=0x7fff93950310,
>> # txn=<value
>> optimized out>, e=0x2b7b521d2068) at tools.c:475
>> # 2  0x00002b7819f28970 in hdb_tool_entry_put (be=0x17f39240,
>> # e=0x2b7b521d2068,
>> text=0x7fff93960cc0) at tools.c:545
>> # 3  0x00000000004906f9 in slapadd (argc=<value optimized out>,
>> argv=0x7fff93960ec8) at ../../../servers/slapd/slapadd.c:350
>> # 4  0x0000000000416c59 in main (argc=6, argv=0x7fff93960ec8) at
>> ../../../servers/slapd/main.c:391
>>
>> Thread 1 (Thread 47794824997056 (LWP 9360)):
>> # 0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>> # 1  0x00002b7819f28014 in bdb_tool_index_add (op=0x7fff93950310,
>> # txn=<value
>> optimized out>, e=0x2b7b521d2068) at tools.c:475
>> # 2  0x00002b7819f28970 in hdb_tool_entry_put (be=0x17f39240,
>> # e=0x2b7b521d2068,
>> text=0x7fff93960cc0) at tools.c:545
>> # 3  0x00000000004906f9 in slapadd (argc=<value optimized out>,
>> argv=0x7fff93960ec8) at ../../../servers/slapd/slapadd.c:350
>> # 4  0x0000000000416c59 in main (argc=6, argv=0x7fff93960ec8) at
>> ../../../servers/slapd/main.c:391
>> # 0  0x00002b7817902496 in pthread_cond_wait@@GLIBC_2.3.2 () from
>> /lib64/libpthread.so.0
>
>
> --
>    -- Howard Chu
>    CTO, Symas Corp.           http://www.symas.com
>    Director, Highland Sun     http://highlandsun.com/hyc/
>    Chief Architect, OpenLDAP  http://www.openldap.org/project/



--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra ::  the leader in open source messaging and collaboration