[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
(ITS#7010) Deadlock in Producer/Consumer Nodes
Full_Name: Duncan Idaho
Version: 2.4.25
OS: Centos 5.5
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (216.148.0.72)
We are running a cluster of "synchronization hosts" that act as Producers for a
large number of Consumers and additionally Consume from a single global
read/write master. After several weeks of stable operation synchronization came
to a halt on both nodes in the Producer/Consumer cluster simultaneously.
Stack-trace:
Thread 15 (Thread 10745):
#0 0x000000322ea07b35 in pthread_join () from /lib64/libpthread.so.0
#1 0x000000000041929d in slapd_daemon () at daemon.c:2922
#2 0x000000000040730e in main (argc=<value optimized out>, argv=0x7fffd2567cb8)
at main.c:983
Thread 14 (Thread 8866):
#0 0x000000322ea0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00000000004ec695 in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
tpool.c:672
#2 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#3 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#4 0x0000000000000000 in ?? ()
Thread 13 (Thread 14876):
#0 0x000000322ea0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000322ea08e1a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x000000322ea08cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000004d7e48 in syncprov_op_search (op=0x2aaac06ab4a0, rs=0x471a9c10)
at syncprov.c:2476
#4 0x000000000047e64a in overlay_op_walk (op=0x2aaac06ab4a0, rs=0x471a9c10,
which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
#5 0x000000000047ec27 in over_op_func (op=0x2aaac06ab4a0, rs=0x471a9c10,
which=op_search) at backover.c:721
#6 0x0000000000420a79 in fe_op_search (op=0x2aaac06ab4a0, rs=0x471a9c10) at
search.c:372
#7 0x0000000000421217 in do_search (op=0x2aaac06ab4a0, rs=0x471a9c10) at
search.c:217
#8 0x000000000041e965 in connection_operation (ctx=0x471a9d60, arg_v=<value
optimized out>) at connection.c:1113
#9 0x000000000041f23c in connection_read_thread (ctx=0x471a9d60, argv=<value
optimized out>) at connection.c:1249
#10 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
tpool.c:685
#11 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#12 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#13 0x0000000000000000 in ?? ()
Thread 12 (Thread 14875):
#0 0x000000322ea0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00000000004ec695 in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
tpool.c:672
#2 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#3 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#4 0x0000000000000000 in ?? ()
Thread 11 (Thread 14874):
#0 0x000000322ea0d605 in __lll_unlock_wake () from /lib64/libpthread.so.0
#1 0x000000322ea0a0e7 in _L_unlock_766 () from /lib64/libpthread.so.0
#2 0x000000322ea0a04e in pthread_mutex_unlock () from /lib64/libpthread.so.0
#3 0x00000000004d7e5e in syncprov_op_search (op=0x2aaac0897eb0, rs=0x461a7c10)
at syncprov.c:2471
#4 0x000000000047e64a in overlay_op_walk (op=0x2aaac0897eb0, rs=0x461a7c10,
which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
#5 0x000000000047ec27 in over_op_func (op=0x2aaac0897eb0, rs=0x461a7c10,
which=op_search) at backover.c:721
#6 0x0000000000420a79 in fe_op_search (op=0x2aaac0897eb0, rs=0x461a7c10) at
search.c:372
#7 0x0000000000421217 in do_search (op=0x2aaac0897eb0, rs=0x461a7c10) at
search.c:217
#8 0x000000000041e965 in connection_operation (ctx=0x461a7d60, arg_v=<value
optimized out>) at connection.c:1113
#9 0x000000000041f23c in connection_read_thread (ctx=0x461a7d60, argv=<value
optimized out>) at connection.c:1249
#10 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
tpool.c:685
#11 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#12 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#13 0x0000000000000000 in ?? ()
Thread 10 (Thread 14873):
#0 0x000000322ea0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000322ea08e1a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x000000322ea08cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000004d7e48 in syncprov_op_search (op=0x2aaab82b63c0, rs=0x459a6c10)
at syncprov.c:2476
#4 0x000000000047e64a in overlay_op_walk (op=0x2aaab82b63c0, rs=0x459a6c10,
which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
#5 0x000000000047ec27 in over_op_func (op=0x2aaab82b63c0, rs=0x459a6c10,
which=op_search) at backover.c:721
#6 0x0000000000420a79 in fe_op_search (op=0x2aaab82b63c0, rs=0x459a6c10) at
search.c:372
#7 0x0000000000421217 in do_search (op=0x2aaab82b63c0, rs=0x459a6c10) at
search.c:217
...back 1 page
Thread 15 (Thread 10745):
#0 0x000000322ea07b35 in pthread_join () from /lib64/libpthread.so.0
#1 0x000000000041929d in slapd_daemon () at daemon.c:2922
#2 0x000000000040730e in main (argc=<value optimized out>, argv=0x7fffd2567cb8)
at main.c:983
Thread 14 (Thread 8866):
#0 0x000000322ea0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00000000004ec695 in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
tpool.c:672
#2 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#3 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#4 0x0000000000000000 in ?? ()
Thread 13 (Thread 14876):
#0 0x000000322ea0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000322ea08e1a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x000000322ea08cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000004d7e48 in syncprov_op_search (op=0x2aaac06ab4a0, rs=0x471a9c10)
at syncprov.c:2476
#4 0x000000000047e64a in overlay_op_walk (op=0x2aaac06ab4a0, rs=0x471a9c10,
which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
#5 0x000000000047ec27 in over_op_func (op=0x2aaac06ab4a0, rs=0x471a9c10,
which=op_search) at backover.c:721
#6 0x0000000000420a79 in fe_op_search (op=0x2aaac06ab4a0, rs=0x471a9c10) at
search.c:372
#7 0x0000000000421217 in do_search (op=0x2aaac06ab4a0, rs=0x471a9c10) at
search.c:217
#8 0x000000000041e965 in connection_operation (ctx=0x471a9d60, arg_v=<value
optimized out>) at connection.c:1113
#9 0x000000000041f23c in connection_read_thread (ctx=0x471a9d60, argv=<value
optimized out>) at connection.c:1249
#10 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
tpool.c:685
#11 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#12 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#13 0x0000000000000000 in ?? ()
Thread 12 (Thread 14875):
#0 0x000000322ea0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00000000004ec695 in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
tpool.c:672
#2 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#3 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#4 0x0000000000000000 in ?? ()
Thread 11 (Thread 14874):
#0 0x000000322ea0d605 in __lll_unlock_wake () from /lib64/libpthread.so.0
#1 0x000000322ea0a0e7 in _L_unlock_766 () from /lib64/libpthread.so.0
#2 0x000000322ea0a04e in pthread_mutex_unlock () from /lib64/libpthread.so.0
#3 0x00000000004d7e5e in syncprov_op_search (op=0x2aaac0897eb0, rs=0x461a7c10)
at syncprov.c:2471
#4 0x000000000047e64a in overlay_op_walk (op=0x2aaac0897eb0, rs=0x461a7c10,
which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
#5 0x000000000047ec27 in over_op_func (op=0x2aaac0897eb0, rs=0x461a7c10,
which=op_search) at backover.c:721
#6 0x0000000000420a79 in fe_op_search (op=0x2aaac0897eb0, rs=0x461a7c10) at
search.c:372
#7 0x0000000000421217 in do_search (op=0x2aaac0897eb0, rs=0x461a7c10) at
search.c:217
#8 0x000000000041e965 in connection_operation (ctx=0x461a7d60, arg_v=<value
optimized out>) at connection.c:1113
#9 0x000000000041f23c in connection_read_thread (ctx=0x461a7d60, argv=<value
optimized out>) at connection.c:1249
#10 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
tpool.c:685
#11 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#12 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#13 0x0000000000000000 in ?? ()
Thread 10 (Thread 14873):
#0 0x000000322ea0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000322ea08e1a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x000000322ea08cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000004d7e48 in syncprov_op_search (op=0x2aaab82b63c0, rs=0x459a6c10)
at syncprov.c:2476
#4 0x000000000047e64a in overlay_op_walk (op=0x2aaab82b63c0, rs=0x459a6c10,
which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
#5 0x000000000047ec27 in over_op_func (op=0x2aaab82b63c0, rs=0x459a6c10,
which=op_search) at backover.c:721
#6 0x0000000000420a79 in fe_op_search (op=0x2aaab82b63c0, rs=0x459a6c10) at
search.c:372
#7 0x0000000000421217 in do_search (op=0x2aaab82b63c0, rs=0x459a6c10) at
search.c:217
[root@su1-admin-syncrepl01 ldap]# cat gdb.txt
Thread 15 (Thread 10745):
#0 0x000000322ea07b35 in pthread_join () from /lib64/libpthread.so.0
#1 0x000000000041929d in slapd_daemon () at daemon.c:2922
#2 0x000000000040730e in main (argc=<value optimized out>, argv=0x7fffd2567cb8)
at main.c:983
Thread 14 (Thread 8866):
#0 0x000000322ea0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00000000004ec695 in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
tpool.c:672
#2 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#3 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#4 0x0000000000000000 in ?? ()
Thread 13 (Thread 14876):
#0 0x000000322ea0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000322ea08e1a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x000000322ea08cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000004d7e48 in syncprov_op_search (op=0x2aaac06ab4a0, rs=0x471a9c10)
at syncprov.c:2476
#4 0x000000000047e64a in overlay_op_walk (op=0x2aaac06ab4a0, rs=0x471a9c10,
which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
#5 0x000000000047ec27 in over_op_func (op=0x2aaac06ab4a0, rs=0x471a9c10,
which=op_search) at backover.c:721
#6 0x0000000000420a79 in fe_op_search (op=0x2aaac06ab4a0, rs=0x471a9c10) at
search.c:372
#7 0x0000000000421217 in do_search (op=0x2aaac06ab4a0, rs=0x471a9c10) at
search.c:217
#8 0x000000000041e965 in connection_operation (ctx=0x471a9d60, arg_v=<value
optimized out>) at connection.c:1113
#9 0x000000000041f23c in connection_read_thread (ctx=0x471a9d60, argv=<value
optimized out>) at connection.c:1249
#10 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
tpool.c:685
#11 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#12 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#13 0x0000000000000000 in ?? ()
Thread 12 (Thread 14875):
#0 0x000000322ea0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00000000004ec695 in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
tpool.c:672
#2 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#3 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#4 0x0000000000000000 in ?? ()
Thread 11 (Thread 14874):
#0 0x000000322ea0d605 in __lll_unlock_wake () from /lib64/libpthread.so.0
#1 0x000000322ea0a0e7 in _L_unlock_766 () from /lib64/libpthread.so.0
#2 0x000000322ea0a04e in pthread_mutex_unlock () from /lib64/libpthread.so.0
#3 0x00000000004d7e5e in syncprov_op_search (op=0x2aaac0897eb0, rs=0x461a7c10)
at syncprov.c:2471
#4 0x000000000047e64a in overlay_op_walk (op=0x2aaac0897eb0, rs=0x461a7c10,
which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
#5 0x000000000047ec27 in over_op_func (op=0x2aaac0897eb0, rs=0x461a7c10,
which=op_search) at backover.c:721
#6 0x0000000000420a79 in fe_op_search (op=0x2aaac0897eb0, rs=0x461a7c10) at
search.c:372
#7 0x0000000000421217 in do_search (op=0x2aaac0897eb0, rs=0x461a7c10) at
search.c:217
#8 0x000000000041e965 in connection_operation (ctx=0x461a7d60, arg_v=<value
optimized out>) at connection.c:1113
#9 0x000000000041f23c in connection_read_thread (ctx=0x461a7d60, argv=<value
optimized out>) at connection.c:1249
#10 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
tpool.c:685
#11 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#12 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#13 0x0000000000000000 in ?? ()
Thread 10 (Thread 14873):
#0 0x000000322ea0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000322ea08e1a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x000000322ea08cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000004d7e48 in syncprov_op_search (op=0x2aaab82b63c0, rs=0x459a6c10)
at syncprov.c:2476
#4 0x000000000047e64a in overlay_op_walk (op=0x2aaab82b63c0, rs=0x459a6c10,
which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
#5 0x000000000047ec27 in over_op_func (op=0x2aaab82b63c0, rs=0x459a6c10,
which=op_search) at backover.c:721
#6 0x0000000000420a79 in fe_op_search (op=0x2aaab82b63c0, rs=0x459a6c10) at
search.c:372
#7 0x0000000000421217 in do_search (op=0x2aaab82b63c0, rs=0x459a6c10) at
search.c:217
#8 0x000000000041e965 in connection_operation (ctx=0x459a6d60, arg_v=<value
optimized out>) at connection.c:1113
#9 0x000000000041f23c in connection_read_thread (ctx=0x459a6d60, argv=<value
optimized out>) at connection.c:1249
#10 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
tpool.c:685
#11 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#12 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#13 0x0000000000000000 in ?? ()
Thread 9 (Thread 14872):
#0 0x000000322ea0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000322ea08e1a in _L_lock_1034 () from /lib64/libpthread.so.0
#2 0x000000322ea08cdc in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000004d7e48 in syncprov_op_search (op=0x2aaab89e58c0, rs=0x451a5c10)
at syncprov.c:2476
#4 0x000000000047e64a in overlay_op_walk (op=0x2aaab89e58c0, rs=0x451a5c10,
which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
#5 0x000000000047ec27 in over_op_func (op=0x2aaab89e58c0, rs=0x451a5c10,
which=op_search) at backover.c:721
#6 0x0000000000420a79 in fe_op_search (op=0x2aaab89e58c0, rs=0x451a5c10) at
search.c:372
#7 0x0000000000421217 in do_search (op=0x2aaab89e58c0, rs=0x451a5c10) at
search.c:217
#8 0x000000000041e965 in connection_operation (ctx=0x451a5d60, arg_v=<value
optimized out>) at connection.c:1113
#9 0x000000000041f23c in connection_read_thread (ctx=0x451a5d60, argv=<value
optimized out>) at connection.c:1249
#10 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
tpool.c:685
#11 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#12 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#13 0x0000000000000000 in ?? ()
Thread 8 (Thread 16034):
#0 0x000000322ea0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x000000000042db1c in send_ldap_ber (op=<value optimized out>,
ber=0x449a43f0) at result.c:372
#2 0x00000000004308d8 in slap_send_search_entry (op=0x449a45a0, rs=0x449a4af0)
at result.c:1410
#3 0x00000000004da80d in syncprov_qplay (ctx=<value optimized out>, arg=<value
optimized out>) at syncprov.c:884
#4 syncprov_qtask (ctx=<value optimized out>, arg=<value optimized out>) at
syncprov.c:1006
#5 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
tpool.c:685
#6 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#7 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#8 0x0000000000000000 in ?? ()
Thread 7 (Thread 16033):
#0 0x000000322ea0d605 in __lll_unlock_wake () from /lib64/libpthread.so.0
#1 0x000000322ea0a0e7 in _L_unlock_766 () from /lib64/libpthread.so.0
#2 0x000000322ea0a04e in pthread_mutex_unlock () from /lib64/libpthread.so.0
#3 0x00000000004d7e5e in syncprov_op_search (op=0x2d554b70, rs=0x441a3c10) at
syncprov.c:2471
#4 0x000000000047e64a in overlay_op_walk (op=0x2d554b70, rs=0x441a3c10,
which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
#5 0x000000000047ec27 in over_op_func (op=0x2d554b70, rs=0x441a3c10,
which=op_search) at backover.c:721
#6 0x0000000000420a79 in fe_op_search (op=0x2d554b70, rs=0x441a3c10) at
search.c:372
#7 0x0000000000421217 in do_search (op=0x2d554b70, rs=0x441a3c10) at
search.c:217
#8 0x000000000041e965 in connection_operation (ctx=0x441a3d60, arg_v=<value
optimized out>) at connection.c:1113
#9 0x000000000041f23c in connection_read_thread (ctx=0x441a3d60, argv=<value
optimized out>) at connection.c:1249
#10 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
tpool.c:685
#11 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#12 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#13 0x0000000000000000 in ?? ()
Thread 6 (Thread 11937):
#0 0x000000322e2baa27 in sched_yield () from /lib64/libc.so.6
#1 0x00000000004d7e7f in syncprov_op_search (op=0x2aaab802af20, rs=0x418f7c10)
at syncprov.c:2475
#2 0x000000000047e64a in overlay_op_walk (op=0x2aaab802af20, rs=0x418f7c10,
which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
#3 0x000000000047ec27 in over_op_func (op=0x2aaab802af20, rs=0x418f7c10,
which=op_search) at backover.c:721
#4 0x0000000000420a79 in fe_op_search (op=0x2aaab802af20, rs=0x418f7c10) at
search.c:372
#5 0x0000000000421217 in do_search (op=0x2aaab802af20, rs=0x418f7c10) at
search.c:217
#6 0x000000000041e965 in connection_operation (ctx=0x418f7d60, arg_v=<value
optimized out>) at connection.c:1113
#7 0x000000000041f23c in connection_read_thread (ctx=0x418f7d60, argv=<value
optimized out>) at connection.c:1249
#8 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
tpool.c:685
#9 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#10 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#11 0x0000000000000000 in ?? ()
Thread 5 (Thread 10755):
#0 0x000000322ea0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1 0x00002b1d8ae85bd1 in __db_pthread_mutex_lock () from
/usr/local/openldap/lib/libdb-5.1.so
#2 0x00002b1d8ae8528a in __db_tas_mutex_lock () from
/usr/local/openldap/lib/libdb-5.1.so
#3 0x00002b1d8af1804c in __lock_get_internal () from
/usr/local/openldap/lib/libdb-5.1.so
#4 0x00002b1d8af189ea in __lock_vec () from
/usr/local/openldap/lib/libdb-5.1.so
#5 0x00002b1d8af1969b in __lock_vec_pp () from
/usr/local/openldap/lib/libdb-5.1.so
#6 0x00000000004bc6c1 in hdb_cache_entry_db_relock (bdb=0xe2bc780, txn=<value
optimized out>, ei=0x2aaadaefc6a0, rw=1, tryOnly=<value optimized out>,
lock=0x439a1640) at cache.c:198
#7 0x00000000004bcb70 in hdb_cache_modify (bdb=0x2b1d8f0fa00c,
e=0x2b1df490e138, newAttrs=0x2b1e071d6918, txn=0x0, lock=0x0) at cache.c:1231
#8 0x0000000000495d13 in hdb_modify (op=0x439a2120, rs=0x439a1cb0) at
modify.c:662
#9 0x000000000047e6c2 in overlay_op_walk (op=0x439a2120, rs=0x439a1cb0,
which=op_modify, oi=0xe2be010, on=0x0) at backover.c:669
#10 0x000000000047ec27 in over_op_func (op=0x439a2120, rs=0x439a1cb0,
which=op_modify) at backover.c:721
#11 0x000000000047240f in syncrepl_entry (si=0xe2c2800, op=0x439a2120,
entry=0x2b1df490e408, modlist=0x439a2c68, syncstate=2, syncUUID=<value optimized
out>, syncCSN=0x2aaade6b6ab0) at syncrepl.c:2694
#12 0x0000000000479cf7 in do_syncrep2 (ctx=<value optimized out>, arg=<value
optimized out>) at syncrepl.c:955
#13 do_syncrepl (ctx=<value optimized out>, arg=<value optimized out>) at
syncrepl.c:1455
#14 0x000000000041f7ca in connection_read_thread (ctx=0x439a2d60, argv=<value
optimized out>) at connection.c:1251
#15 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
tpool.c:685
#16 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#17 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#18 0x0000000000000000 in ?? ()
Thread 4 (Thread 10752):
#0 0x000000322e2baa27 in sched_yield () from /lib64/libc.so.6
#1 0x00000000004d7e7f in syncprov_op_search (op=0xe6f0990, rs=0x410f6c10) at
syncprov.c:2475
#2 0x000000000047e64a in overlay_op_walk (op=0xe6f0990, rs=0x410f6c10,
which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
#3 0x000000000047ec27 in over_op_func (op=0xe6f0990, rs=0x410f6c10,
which=op_search) at backover.c:721
#4 0x0000000000420a79 in fe_op_search (op=0xe6f0990, rs=0x410f6c10) at
search.c:372
#5 0x0000000000421217 in do_search (op=0xe6f0990, rs=0x410f6c10) at
search.c:217
#6 0x000000000041e965 in connection_operation (ctx=0x410f6d60, arg_v=<value
optimized out>) at connection.c:1113
#7 0x000000000041f23c in connection_read_thread (ctx=0x410f6d60, argv=<value
optimized out>) at connection.c:1249
#8 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
tpool.c:685
#9 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#10 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#11 0x0000000000000000 in ?? ()
Thread 3 (Thread 10749):
#0 0x000000322e2baa27 in sched_yield () from /lib64/libc.so.6
#1 0x00000000004d7e7f in syncprov_op_search (op=0xe5eefa0, rs=0x431a1c10) at
syncprov.c:2475
#2 0x000000000047e64a in overlay_op_walk (op=0xe5eefa0, rs=0x431a1c10,
which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
#3 0x000000000047ec27 in over_op_func (op=0xe5eefa0, rs=0x431a1c10,
which=op_search) at backover.c:721
#4 0x0000000000420a79 in fe_op_search (op=0xe5eefa0, rs=0x431a1c10) at
search.c:372
#5 0x0000000000421217 in do_search (op=0xe5eefa0, rs=0x431a1c10) at
search.c:217
#6 0x000000000041e965 in connection_operation (ctx=0x431a1d60, arg_v=<value
optimized out>) at connection.c:1113
#7 0x000000000041f23c in connection_read_thread (ctx=0x431a1d60, argv=<value
optimized out>) at connection.c:1249
#8 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
tpool.c:685
#9 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#10 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#11 0x0000000000000000 in ?? ()
Thread 2 (Thread 10748):
#0 0x000000322e2baa27 in sched_yield () from /lib64/libc.so.6
#1 0x00000000004d7e7f in syncprov_op_search (op=0xe5ebbf0, rs=0x429a0c10) at
syncprov.c:2475
#2 0x000000000047e64a in overlay_op_walk (op=0xe5ebbf0, rs=0x429a0c10,
which=op_search, oi=0xe2be010, on=0xe2c30b0) at backover.c:659
#3 0x000000000047ec27 in over_op_func (op=0xe5ebbf0, rs=0x429a0c10,
which=op_search) at backover.c:721
#4 0x0000000000420a79 in fe_op_search (op=0xe5ebbf0, rs=0x429a0c10) at
search.c:372
#5 0x0000000000421217 in do_search (op=0xe5ebbf0, rs=0x429a0c10) at
search.c:217
#6 0x000000000041e965 in connection_operation (ctx=0x429a0d60, arg_v=<value
optimized out>) at connection.c:1113
#7 0x000000000041f23c in connection_read_thread (ctx=0x429a0d60, argv=<value
optimized out>) at connection.c:1249
#8 0x00000000004ec63c in ldap_int_thread_pool_wrapper (xpool=0xe232120) at
tpool.c:685
#9 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#10 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#11 0x0000000000000000 in ?? ()
Thread 1 (Thread 10747):
#0 0x000000322e2d4108 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000041bc8b in slapd_daemon_task (ptr=<value optimized out>) at
daemon.c:2528
#2 0x000000322ea0673d in start_thread () from /lib64/libpthread.so.0
#3 0x000000322e2d3d1d in clone () from /lib64/libc.so.6
#4 0x0000000000000000 in ?? ()
Logs at the time of the lock-up:
Producer:
Jul 28 22:27:08 admin02 slapd[5178]: conn=422733 op=92 MOD
dn="thing=thing1,ou=things,dc=example,dc=net"
Jul 28 22:27:08 admin02 slapd[5178]: conn=422733 op=92 MOD attr=stuff
Jul 28 22:27:08 admin02 slapd[5178]: slap_queue_csn: queing 0x4b0883f0
20110728222708.058275Z#000000#000#000000
Jul 28 22:27:08 admin02 slapd[5178]: syncprov_sendresp: cookie=rid=001
,csn=20110728222708.058275Z#000000#000#000000
Jul 28 22:27:08 admin02 slapd[5178]: syncprov_sendresp: cookie=rid=001
Producer later:
Jul 29 07:04:55 admin02 slapd[5178]: Entry
thing=thing1,ou=things,dc=example,dc=net CSN
20110728222708.058275Z#000000#000#000000 older or equal to ctx
20110728222708.058275Z#000000#000#000000
Producer/Consumer:
Jul 28 22:27:08 su1-admin-syncrepl01 slapd[10745]: do_syncrep2: rid=001
cookie=rid=001,csn=20110728222708.058275Z#000000#000#000000
Jul 28 22:27:08 su1-admin-syncrepl01 slapd[10745]: slap_queue_csn: queing
0x2aaadecf4b00 20110728222708.058275Z#000000#000#000000
^^ No more syncrepl logs after this
Help!