[Date Prev][Date Next] [Chronological] [Thread] [Top]
Re: (ITS#5161) delta-syncrepl mutex lockup

To: openldap-its@OpenLDAP.org
Subject: Re: (ITS#5161) delta-syncrepl mutex lockup
From: quanah@zimbra.com
Date: Wed, 5 Dec 2007 01:55:44 GMT

Just hit this again today.

(gdb) info threads
  10 Thread 1082132800 (LWP 15704)  0xffffffffff600410 in ?? ()
* 9 Thread 1090525504 (LWP 15705)  0x00000038dcc0c758 in 
__lll_mutex_lock_wait () from /lib64/libpthread.so.0
  8 Thread 1098918208 (LWP 15706)  0x00000038dcc0c758 in 
__lll_mutex_lock_wait () from /lib64/libpthread.so.0
  7 Thread 1107310912 (LWP 15707)  0x00000038dcc0c758 in 
__lll_mutex_lock_wait () from /lib64/libpthread.so.0
  6 Thread 1115703616 (LWP 32142)  0x00000038dcc0c758 in 
__lll_mutex_lock_wait () from /lib64/libpthread.so.0
  5 Thread 1124096320 (LWP 1167)  0x00000038dcc0c758 in 
__lll_mutex_lock_wait () from /lib64/libpthread.so.0
  4 Thread 1132489024 (LWP 1168)  0x00000038dcc0c758 in 
__lll_mutex_lock_wait () from /lib64/libpthread.so.0
  3 Thread 1140881728 (LWP 1431)  0x00000038dcc0c758 in 
__lll_mutex_lock_wait () from /lib64/libpthread.so.0
  2 Thread 1149274432 (LWP 1432)  0x00000038dcc0c758 in 
__lll_mutex_lock_wait () from /lib64/libpthread.so.0
  1 Thread 46912515736208 (LWP 15700)  0x00000038dcc074e5 in pthread_join 
() from /lib64/libpthread.so.0
(gdb) thread 10
[Switching to thread 10 (Thread 1082132800 (LWP 15704))]#0 
0xffffffffff600410 in ?? ()
(gdb) bt
#0  0xffffffffff600410 in ?? ()
#1  0x00000038dc0862dd in time () from /lib64/libc.so.6
#2  0x0000000000428530 in slapd_daemon_task (ptr=0x0) at daemon.c:2033
#3  0x00000038dcc061b5 in start_thread () from /lib64/libpthread.so.0
#4  0x00000038dc0cd36d in clone () from /lib64/libc.so.6
#5  0x0000000000000000 in ?? ()
(gdb) thread 9
[Switching to thread 9 (Thread 1090525504 (LWP 15705))]#0 
0x00000038dcc0c758 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00000038dcc0c758 in __lll_mutex_lock_wait () from 
/lib64/libpthread.so.0
#1  0x00000038dcc087c4 in _L_mutex_lock_107 () from /lib64/libpthread.so.0
#2  0x00000038dcc08263 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002aaaaad1685e in ldap_pvt_thread_mutex_lock (mutex=0x7f7fa58) at 
thr_posix.c:333
#4  0x00002aaaae660500 in accesslog_op_mod (op=0xc48fc00, rs=0x41000d00) at 
accesslog.c:1236
#5  0x00000000004a0cfe in overlay_op_walk (op=0xc48fc00, rs=0x41000d00, 
which=op_delete, oi=0x7f86c40, on=0x7f868c0) at backover.c:640
#6  0x00000000004a0f90 in over_op_func (op=0xc48fc00, rs=0x41000d00, 
which=op_delete) at backover.c:702
#7  0x00000000004a10da in over_op_delete (op=0xc48fc00, rs=0x41000d00) at 
backover.c:754
#8  0x000000000044c4f9 in fe_op_delete (op=0xc48fc00, rs=0x41000d00) at 
delete.c:179
#9  0x000000000044c126 in do_delete (op=0xc48fc00, rs=0x41000d00) at 
delete.c:91
#10 0x000000000042c2da in connection_operation (ctx=0x41000de0, 
arg_v=0xc48fc00) at connection.c:1133
#11 0x00002aaaaad154ba in ldap_int_thread_pool_wrapper (xpool=0x7c0ff00) at 
tpool.c:478
#12 0x00000038dcc061b5 in start_thread () from /lib64/libpthread.so.0
#13 0x00000038dc0cd36d in clone () from /lib64/libc.so.6
#14 0x0000000000000000 in ?? ()
(gdb) thread 8
[Switching to thread 8 (Thread 1098918208 (LWP 15706))]#0 
0x00000038dcc0c758 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00000038dcc0c758 in __lll_mutex_lock_wait () from 
/lib64/libpthread.so.0
#1  0x00000038dcc087c4 in _L_mutex_lock_107 () from /lib64/libpthread.so.0
#2  0x00000038dcc08263 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002aaaaad1685e in ldap_pvt_thread_mutex_lock (mutex=0x7f7fa58) at 
thr_posix.c:333
#4  0x00002aaaae660500 in accesslog_op_mod (op=0xd5f7600, rs=0x41801ca0) at 
accesslog.c:1236
#5  0x00000000004a0cfe in overlay_op_walk (op=0xd5f7600, rs=0x41801ca0, 
which=op_delete, oi=0x7f86c40, on=0x7f868c0) at backover.c:640
#6  0x00000000004a0f90 in over_op_func (op=0xd5f7600, rs=0x41801ca0, 
which=op_delete) at backover.c:702
#7  0x00000000004a10da in over_op_delete (op=0xd5f7600, rs=0x41801ca0) at 
backover.c:754
#8  0x000000000044c4f9 in fe_op_delete (op=0xd5f7600, rs=0x41801ca0) at 
delete.c:179
#9  0x000000000044c126 in do_delete (op=0xd5f7600, rs=0x41801ca0) at 
delete.c:91
#10 0x000000000042c2da in connection_operation (ctx=0x41801de0, 
arg_v=0xd5f7600) at connection.c:1133
#11 0x000000000042c7b0 in connection_read_thread (ctx=0x41801de0, 
argv=0x1d) at connection.c:1261
#12 0x00002aaaaad154ba in ldap_int_thread_pool_wrapper (xpool=0x7c0ff00) at 
tpool.c:478
#13 0x00000038dcc061b5 in start_thread () from /lib64/libpthread.so.0
#14 0x00000038dc0cd36d in clone () from /lib64/libc.so.6
#15 0x0000000000000000 in ?? ()
(gdb) thread 7
[Switching to thread 7 (Thread 1107310912 (LWP 15707))]#0 
0x00000038dcc0c758 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00000038dcc0c758 in __lll_mutex_lock_wait () from 
/lib64/libpthread.so.0
#1  0x00000038dcc087c4 in _L_mutex_lock_107 () from /lib64/libpthread.so.0
#2  0x00000038dcc08263 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002aaaaad1685e in ldap_pvt_thread_mutex_lock (mutex=0x7f7fa58) at 
thr_posix.c:333
#4  0x00002aaaae660500 in accesslog_op_mod (op=0xc48f900, rs=0x42002ca0) at 
accesslog.c:1236
#5  0x00000000004a0cfe in overlay_op_walk (op=0xc48f900, rs=0x42002ca0, 
which=op_delete, oi=0x7f86c40, on=0x7f868c0) at backover.c:640
#6  0x00000000004a0f90 in over_op_func (op=0xc48f900, rs=0x42002ca0, 
which=op_delete) at backover.c:702
#7  0x00000000004a10da in over_op_delete (op=0xc48f900, rs=0x42002ca0) at 
backover.c:754
#8  0x000000000044c4f9 in fe_op_delete (op=0xc48f900, rs=0x42002ca0) at 
delete.c:179
#9  0x000000000044c126 in do_delete (op=0xc48f900, rs=0x42002ca0) at 
delete.c:91
#10 0x000000000042c2da in connection_operation (ctx=0x42002de0, 
arg_v=0xc48f900) at connection.c:1133
#11 0x000000000042c7b0 in connection_read_thread (ctx=0x42002de0, 
argv=0x1c) at connection.c:1261
#12 0x00002aaaaad154ba in ldap_int_thread_pool_wrapper (xpool=0x7c0ff00) at 
tpool.c:478
#13 0x00000038dcc061b5 in start_thread () from /lib64/libpthread.so.0
#14 0x00000038dc0cd36d in clone () from /lib64/libc.so.6
#15 0x0000000000000000 in ?? ()
(gdb) thread 6
[Switching to thread 6 (Thread 1115703616 (LWP 32142))]#0 
0x00000038dcc0c758 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00000038dcc0c758 in __lll_mutex_lock_wait () from 
/lib64/libpthread.so.0
#1  0x00000038dcc087c4 in _L_mutex_lock_107 () from /lib64/libpthread.so.0
#2  0x00000038dcc08263 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002aaaaad1685e in ldap_pvt_thread_mutex_lock (mutex=0x7f7fa58) at 
thr_posix.c:333
#4  0x00002aaaae660500 in accesslog_op_mod (op=0xb20ac00, rs=0x42803d00) at 
accesslog.c:1236
#5  0x00000000004a0cfe in overlay_op_walk (op=0xb20ac00, rs=0x42803d00, 
which=op_delete, oi=0x7f86c40, on=0x7f868c0) at backover.c:640
#6  0x00000000004a0f90 in over_op_func (op=0xb20ac00, rs=0x42803d00, 
which=op_delete) at backover.c:702
#7  0x00000000004a10da in over_op_delete (op=0xb20ac00, rs=0x42803d00) at 
backover.c:754
#8  0x000000000044c4f9 in fe_op_delete (op=0xb20ac00, rs=0x42803d00) at 
delete.c:179
#9  0x000000000044c126 in do_delete (op=0xb20ac00, rs=0x42803d00) at 
delete.c:91
#10 0x000000000042c2da in connection_operation (ctx=0x42803de0, 
arg_v=0xb20ac00) at connection.c:1133
#11 0x00002aaaaad154ba in ldap_int_thread_pool_wrapper (xpool=0x7c0ff00) at 
tpool.c:478
#12 0x00000038dcc061b5 in start_thread () from /lib64/libpthread.so.0
#13 0x00000038dc0cd36d in clone () from /lib64/libc.so.6
#14 0x0000000000000000 in ?? ()
(gdb) thread 5
[Switching to thread 5 (Thread 1124096320 (LWP 1167))]#0 
0x00000038dcc0c758 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00000038dcc0c758 in __lll_mutex_lock_wait () from 
/lib64/libpthread.so.0
#1  0x00000038dcc087c4 in _L_mutex_lock_107 () from /lib64/libpthread.so.0
#2  0x00000038dcc08263 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002aaaaad1685e in ldap_pvt_thread_mutex_lock (mutex=0x7f7fa58) at 
thr_posix.c:333
#4  0x00002aaaae660500 in accesslog_op_mod (op=0xd5f7c00, rs=0x43004d00) at 
accesslog.c:1236
#5  0x00000000004a0cfe in overlay_op_walk (op=0xd5f7c00, rs=0x43004d00, 
which=op_delete, oi=0x7f86c40, on=0x7f868c0) at backover.c:640
#6  0x00000000004a0f90 in over_op_func (op=0xd5f7c00, rs=0x43004d00, 
which=op_delete) at backover.c:702
#7  0x00000000004a10da in over_op_delete (op=0xd5f7c00, rs=0x43004d00) at 
backover.c:754
#8  0x000000000044c4f9 in fe_op_delete (op=0xd5f7c00, rs=0x43004d00) at 
delete.c:179
#9  0x000000000044c126 in do_delete (op=0xd5f7c00, rs=0x43004d00) at 
delete.c:91
#10 0x000000000042c2da in connection_operation (ctx=0x43004de0, 
arg_v=0xd5f7c00) at connection.c:1133
#11 0x00002aaaaad154ba in ldap_int_thread_pool_wrapper (xpool=0x7c0ff00) at 
tpool.c:478
#12 0x00000038dcc061b5 in start_thread () from /lib64/libpthread.so.0
#13 0x00000038dc0cd36d in clone () from /lib64/libc.so.6
#14 0x0000000000000000 in ?? ()
(gdb) thread 4
[Switching to thread 4 (Thread 1132489024 (LWP 1168))]#0 
0x00000038dcc0c758 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00000038dcc0c758 in __lll_mutex_lock_wait () from 
/lib64/libpthread.so.0
#1  0x00000038dcc087c4 in _L_mutex_lock_107 () from /lib64/libpthread.so.0
#2  0x00000038dcc08263 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002aaaaad1685e in ldap_pvt_thread_mutex_lock (mutex=0x7f7fa58) at 
thr_posix.c:333
#4  0x00002aaaae660500 in accesslog_op_mod (op=0xd5f7900, rs=0x43805ca0) at 
accesslog.c:1236
#5  0x00000000004a0cfe in overlay_op_walk (op=0xd5f7900, rs=0x43805ca0, 
which=op_delete, oi=0x7f86c40, on=0x7f868c0) at backover.c:640
#6  0x00000000004a0f90 in over_op_func (op=0xd5f7900, rs=0x43805ca0, 
which=op_delete) at backover.c:702
#7  0x00000000004a10da in over_op_delete (op=0xd5f7900, rs=0x43805ca0) at 
backover.c:754
#8  0x000000000044c4f9 in fe_op_delete (op=0xd5f7900, rs=0x43805ca0) at 
delete.c:179
#9  0x000000000044c126 in do_delete (op=0xd5f7900, rs=0x43805ca0) at 
delete.c:91
#10 0x000000000042c2da in connection_operation (ctx=0x43805de0, 
arg_v=0xd5f7900) at connection.c:1133
#11 0x000000000042c7b0 in connection_read_thread (ctx=0x43805de0, 
argv=0x1d) at connection.c:1261
#12 0x00002aaaaad154ba in ldap_int_thread_pool_wrapper (xpool=0x7c0ff00) at 
tpool.c:478
#13 0x00000038dcc061b5 in start_thread () from /lib64/libpthread.so.0
#14 0x00000038dc0cd36d in clone () from /lib64/libc.so.6
#15 0x0000000000000000 in ?? ()
(gdb) thread 3
[Switching to thread 3 (Thread 1140881728 (LWP 1431))]#0 
0x00000038dcc0c758 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00000038dcc0c758 in __lll_mutex_lock_wait () from 
/lib64/libpthread.so.0
#1  0x00000038dcc087c4 in _L_mutex_lock_107 () from /lib64/libpthread.so.0
#2  0x00000038dcc08263 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002aaaaad1685e in ldap_pvt_thread_mutex_lock (mutex=0x7f7fa58) at 
thr_posix.c:333
#4  0x00002aaaae660500 in accesslog_op_mod (op=0xd955c00, rs=0x44006d00) at 
accesslog.c:1236
#5  0x00000000004a0cfe in overlay_op_walk (op=0xd955c00, rs=0x44006d00, 
which=op_delete, oi=0x7f86c40, on=0x7f868c0) at backover.c:640
#6  0x00000000004a0f90 in over_op_func (op=0xd955c00, rs=0x44006d00, 
which=op_delete) at backover.c:702
#7  0x00000000004a10da in over_op_delete (op=0xd955c00, rs=0x44006d00) at 
backover.c:754
#8  0x000000000044c4f9 in fe_op_delete (op=0xd955c00, rs=0x44006d00) at 
delete.c:179
#9  0x000000000044c126 in do_delete (op=0xd955c00, rs=0x44006d00) at 
delete.c:91
#10 0x000000000042c2da in connection_operation (ctx=0x44006de0, 
arg_v=0xd955c00) at connection.c:1133
#11 0x00002aaaaad154ba in ldap_int_thread_pool_wrapper (xpool=0x7c0ff00) at 
tpool.c:478
#12 0x00000038dcc061b5 in start_thread () from /lib64/libpthread.so.0
#13 0x00000038dc0cd36d in clone () from /lib64/libc.so.6
#14 0x0000000000000000 in ?? ()
(gdb) thread 2
[Switching to thread 2 (Thread 1149274432 (LWP 1432))]#0 
0x00000038dcc0c758 in __lll_mutex_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00000038dcc0c758 in __lll_mutex_lock_wait () from 
/lib64/libpthread.so.0
#1  0x00000038dcc087c4 in _L_mutex_lock_107 () from /lib64/libpthread.so.0
#2  0x00000038dcc08263 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002aaaaad1685e in ldap_pvt_thread_mutex_lock (mutex=0x7f7fa58) at 
thr_posix.c:333
#4  0x00002aaaae660500 in accesslog_op_mod (op=0xd955900, rs=0x44807ca0) at 
accesslog.c:1236
#5  0x00000000004a0cfe in overlay_op_walk (op=0xd955900, rs=0x44807ca0, 
which=op_delete, oi=0x7f86c40, on=0x7f868c0) at backover.c:640
#6  0x00000000004a0f90 in over_op_func (op=0xd955900, rs=0x44807ca0, 
which=op_delete) at backover.c:702
#7  0x00000000004a10da in over_op_delete (op=0xd955900, rs=0x44807ca0) at 
backover.c:754
#8  0x000000000044c4f9 in fe_op_delete (op=0xd955900, rs=0x44807ca0) at 
delete.c:179
#9  0x000000000044c126 in do_delete (op=0xd955900, rs=0x44807ca0) at 
delete.c:91
#10 0x000000000042c2da in connection_operation (ctx=0x44807de0, 
arg_v=0xd955900) at connection.c:1133
#11 0x000000000042c7b0 in connection_read_thread (ctx=0x44807de0, 
argv=0x1e) at connection.c:1261
#12 0x00002aaaaad154ba in ldap_int_thread_pool_wrapper (xpool=0x7c0ff00) at 
tpool.c:478
#13 0x00000038dcc061b5 in start_thread () from /lib64/libpthread.so.0
#14 0x00000038dc0cd36d in clone () from /lib64/libc.so.6
#15 0x0000000000000000 in ?? ()
(gdb) thread 1
[Switching to thread 1 (Thread 46912515736208 (LWP 15700))]#0 
0x00000038dcc074e5 in pthread_join () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00000038dcc074e5 in pthread_join () from /lib64/libpthread.so.0
#1  0x00002aaaaad16767 in ldap_pvt_thread_join (thread=1082132800, 
thread_return=0x0) at thr_posix.c:193
#2  0x0000000000429bd3 in slapd_daemon () at daemon.c:2579
#3  0x0000000000412113 in main (argc=10, argv=0x7fffd07d4fe8) at main.c:859

--Quanah


--

Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra ::  the leader in open source messaging and collaboration
Prev by Date: Re: (ITS#5257) test008 hanging in bdb
Next by Date: Re: (ITS#5257) test008 hanging in bdb
Index(es):
- Chronological
- Thread