[Date Prev][Date Next] [Chronological] [Thread] [Top]
(ITS#7296) slapd stuck (in async mode) after client disconnection

To: openldap-its@OpenLDAP.org
Subject: (ITS#7296) slapd stuck (in async mode) after client disconnection
From: gerard.gobillard@alcatel-lucent.com
Date: Fri, 8 Jun 2012 07:06:21 GMT
Auto-submitted: auto-generated (OpenLDAP-ITS)
Full_Name: gerard gobillard
Version: 2.4.30
OS: RHEL 5.2
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (92.139.226.97)


Hello all,

Summary : on an Intel basic box (dual core) we create search base traffic with
slamd in asynchronous mode (30 threads). When, after 1 or 2 seconds, Ctrl-C is
sent to slamd (SIGINT), slapd gets stuck with main thread in epoll_wait() and
all other threads in pthread_cond_wait() (in send_ldap_ber()). It may be
necessary to repeat a few times before slapd gets stuck

 Notes : no change to slapd.conf except RootDN.

We build OpenLDAP as follows :

CPPFLAGS="-I/usr/local/BerkeleyDB.4.7/include -g"

export CPPFLAGS

LDFLAGS="-L/usr/local/lib -L/usr/local/BerkeleyDB.4.7/lib
-R/usr/local/BerkeleyDB.4.7/lib"

export LDFLAGS

LD_LIBRARY_PATH=/usr/local/BerkeleyDB.4.7/lib

./configure --prefix=/usr/slapd.2.4.23.Dbg --enable-spasswd=yes
--enable-crypt=yes --disable-syslog --enable-debug --enable-ppolicy

make depend

make

make install

gstack:

[root@lalxsdmcomp db-5.2.36]# gstack 19444
Thread 18 (Thread 1099237696 (LWP 19445)):
#0  0x0000003cf08d21d8 in epoll_wait () from /lib64/libc.so.6
#1  0x000000000042bf11 in slapd_daemon_task ()
#2  0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0
#3  0x0000003cf08d1ded in clone () from /lib64/libc.so.6
Thread 17 (Thread 1112529216 (LWP 19463)):
#0  0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x0000000000444428 in send_ldap_ber ()
#2  0x00000000004477da in slap_send_search_entry ()
#3  0x00000000004cdbe6 in bdb_search ()
#4  0x00000000004334e4 in fe_op_search ()
#5  0x0000000000432d9b in do_search ()
#6  0x000000000042fec7 in connection_operation ()
#7  0x000000000056562c in ldap_int_thread_pool_wrapper ()
#8  0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0
#9  0x0000003cf08d1ded in clone () from /lib64/libc.so.6
Thread 16 (Thread 1120921920 (LWP 19464)):
#0  0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x0000000000444428 in send_ldap_ber ()
#2  0x00000000004477da in slap_send_search_entry ()
#3  0x00000000004cdbe6 in bdb_search ()
#4  0x00000000004334e4 in fe_op_search ()
#5  0x0000000000432d9b in do_search ()
#6  0x000000000042fec7 in connection_operation ()
#7  0x000000000056562c in ldap_int_thread_pool_wrapper ()
#8  0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0
#9  0x0000003cf08d1ded in clone () from /lib64/libc.so.6
Thread 15 (Thread 1129314624 (LWP 19465)):
#0  0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x0000000000444428 in send_ldap_ber ()
#2  0x00000000004477da in slap_send_search_entry ()
#3  0x00000000004cdbe6 in bdb_search ()
#4  0x00000000004334e4 in fe_op_search ()
#5  0x0000000000432d9b in do_search ()
#6  0x000000000042fec7 in connection_operation ()
#7  0x000000000056562c in ldap_int_thread_pool_wrapper ()
#8  0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0
#9  0x0000003cf08d1ded in clone () from /lib64/libc.so.6
Thread 14 (Thread 1137707328 (LWP 19466)):
#0  0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x0000000000444428 in send_ldap_ber ()
#2  0x00000000004477da in slap_send_search_entry ()
#3  0x00000000004cdbe6 in bdb_search ()
#4  0x00000000004334e4 in fe_op_search ()
#5  0x0000000000432d9b in do_search ()
#6  0x000000000042fec7 in connection_operation ()
#7  0x000000000056562c in ldap_int_thread_pool_wrapper ()
#8  0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0
#9  0x0000003cf08d1ded in clone () from /lib64/libc.so.6
Thread 13 (Thread 1146100032 (LWP 19467)):
#0  0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x0000000000444428 in send_ldap_ber ()
#2  0x00000000004477da in slap_send_search_entry ()
#3  0x00000000004cdbe6 in bdb_search ()
#4  0x00000000004334e4 in fe_op_search ()
#5  0x0000000000432d9b in do_search ()
#6  0x000000000042fec7 in connection_operation ()
#7  0x000000000056562c in ldap_int_thread_pool_wrapper ()
#8  0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0
#9  0x0000003cf08d1ded in clone () from /lib64/libc.so.6
Thread 12 (Thread 1154492736 (LWP 19468)):
#0  0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x0000000000444428 in send_ldap_ber ()
#2  0x00000000004477da in slap_send_search_entry ()
#3  0x00000000004cdbe6 in bdb_search ()
#4  0x00000000004334e4 in fe_op_search ()
#5  0x0000000000432d9b in do_search ()
#6  0x000000000042fec7 in connection_operation ()
#7  0x000000000056562c in ldap_int_thread_pool_wrapper ()
#8  0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0
#9  0x0000003cf08d1ded in clone () from /lib64/libc.so.6
Thread 11 (Thread 1162885440 (LWP 19469)):
#0  0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x00000000004446a0 in send_ldap_ber ()
#2  0x00000000004477da in slap_send_search_entry ()
#3  0x00000000004cdbe6 in bdb_search ()
#4  0x00000000004334e4 in fe_op_search ()
#5  0x0000000000432d9b in do_search ()
#6  0x000000000042fec7 in connection_operation ()
#7  0x000000000056562c in ldap_int_thread_pool_wrapper ()
#8  0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0
#9  0x0000003cf08d1ded in clone () from /lib64/libc.so.6
Thread 10 (Thread 1171278144 (LWP 19470)):
#0  0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x0000000000444428 in send_ldap_ber ()
#2  0x00000000004477da in slap_send_search_entry ()
#3  0x00000000004cdbe6 in bdb_search ()
#4  0x00000000004334e4 in fe_op_search ()
#5  0x0000000000432d9b in do_search ()
#6  0x000000000042fec7 in connection_operation ()
#7  0x000000000056562c in ldap_int_thread_pool_wrapper ()
#8  0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0
#9  0x0000003cf08d1ded in clone () from /lib64/libc.so.6
Thread 9 (Thread 1087490368 (LWP 19471)):
#0  0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x0000000000444428 in send_ldap_ber ()
#2  0x00000000004477da in slap_send_search_entry ()
#3  0x00000000004cdbe6 in bdb_search ()
#4  0x00000000004334e4 in fe_op_search ()
#5  0x0000000000432d9b in do_search ()
#6  0x000000000042fec7 in connection_operation ()
#7  0x000000000056562c in ldap_int_thread_pool_wrapper ()
#8  0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0
#9  0x0000003cf08d1ded in clone () from /lib64/libc.so.6
Thread 8 (Thread 1179670848 (LWP 19472)):
#0  0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x0000000000444428 in send_ldap_ber ()
#2  0x00000000004477da in slap_send_search_entry ()
#3  0x00000000004cdbe6 in bdb_search ()
#4  0x00000000004334e4 in fe_op_search ()
#5  0x0000000000432d9b in do_search ()
#6  0x000000000042fec7 in connection_operation ()
#7  0x000000000056562c in ldap_int_thread_pool_wrapper ()
#8  0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0
#9  0x0000003cf08d1ded in clone () from /lib64/libc.so.6
Thread 7 (Thread 1188063552 (LWP 19473)):
#0  0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x0000000000444428 in send_ldap_ber ()
#2  0x00000000004477da in slap_send_search_entry ()
#3  0x00000000004cdbe6 in bdb_search ()
#4  0x00000000004334e4 in fe_op_search ()
#5  0x0000000000432d9b in do_search ()
#6  0x000000000042fec7 in connection_operation ()
#7  0x000000000056562c in ldap_int_thread_pool_wrapper ()
#8  0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0
#9  0x0000003cf08d1ded in clone () from /lib64/libc.so.6
Thread 6 (Thread 1196456256 (LWP 19474)):
#0  0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x00000000004446a0 in send_ldap_ber ()
#2  0x00000000004477da in slap_send_search_entry ()
#3  0x00000000004cdbe6 in bdb_search ()
#4  0x00000000004334e4 in fe_op_search ()
#5  0x0000000000432d9b in do_search ()
#6  0x000000000042fec7 in connection_operation ()
#7  0x000000000056562c in ldap_int_thread_pool_wrapper ()
#8  0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0
#9  0x0000003cf08d1ded in clone () from /lib64/libc.so.6
Thread 5 (Thread 1204848960 (LWP 19475)):
#0  0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x0000000000444428 in send_ldap_ber ()
#2  0x00000000004477da in slap_send_search_entry ()
#3  0x00000000004cdbe6 in bdb_search ()
#4  0x00000000004334e4 in fe_op_search ()
#5  0x0000000000432d9b in do_search ()
#6  0x000000000042fec7 in connection_operation ()
#7  0x000000000056562c in ldap_int_thread_pool_wrapper ()
#8  0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0
#9  0x0000003cf08d1ded in clone () from /lib64/libc.so.6
Thread 4 (Thread 1213241664 (LWP 19476)):
#0  0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x0000000000444428 in send_ldap_ber ()
#2  0x00000000004477da in slap_send_search_entry ()
#3  0x00000000004cdbe6 in bdb_search ()
#4  0x00000000004334e4 in fe_op_search ()
#5  0x0000000000432d9b in do_search ()
#6  0x000000000042fec7 in connection_operation ()
#7  0x000000000056562c in ldap_int_thread_pool_wrapper ()
#8  0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0
#9  0x0000003cf08d1ded in clone () from /lib64/libc.so.6
Thread 3 (Thread 1221634368 (LWP 19477)):
#0  0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x0000000000444428 in send_ldap_ber ()
#2  0x00000000004477da in slap_send_search_entry ()
#3  0x00000000004cdbe6 in bdb_search ()
#4  0x00000000004334e4 in fe_op_search ()
#5  0x0000000000432d9b in do_search ()
#6  0x000000000042fec7 in connection_operation ()
#7  0x000000000056562c in ldap_int_thread_pool_wrapper ()
#8  0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0
#9  0x0000003cf08d1ded in clone () from /lib64/libc.so.6
Thread 2 (Thread 1230027072 (LWP 19478)):
#0  0x0000003cf140a4b6 in pthread_cond_wait@@GLIBC_2.3.2 ()
#1  0x00000000004446a0 in send_ldap_ber ()
#2  0x00000000004477da in slap_send_search_entry ()
#3  0x00000000004cdbe6 in bdb_search ()
#4  0x00000000004334e4 in fe_op_search ()
#5  0x0000000000432d9b in do_search ()
#6  0x000000000042fec7 in connection_operation ()
#7  0x000000000056562c in ldap_int_thread_pool_wrapper ()
#8  0x0000003cf1406307 in start_thread () from /lib64/libpthread.so.0
#9  0x0000003cf08d1ded in clone () from /lib64/libc.so.6
Thread 1 (Thread 47813429243440 (LWP 19444)):
#0  0x0000003cf14075c5 in pthread_join () from /lib64/libpthread.so.0
#1  0x000000000042cf32 in slapd_daemon ()
#2  0x000000000040d2a0 in main ()
#0  0x0000003cf14075c5 in pthread_join () from /lib64/libpthread.so.0

netstat:

[root@lalxsdmcomp db-5.2.36]# netstat -pan | grep :389
tcp        0      0 0.0.0.0:389                 0.0.0.0:*                  
LISTEN      19444/slapd
tcp    28071      0 172.25.198.71:389           172.25.195.198:38743       
CLOSE_WAIT  19444/slapd
tcp    14026      0 172.25.198.71:389           172.25.195.198:38749       
CLOSE_WAIT  19444/slapd
tcp    57221      0 172.25.198.71:389           172.25.195.198:38751       
CLOSE_WAIT  19444/slapd
tcp    31304      0 172.25.198.71:389           172.25.195.198:38750       
CLOSE_WAIT  19444/slapd
tcp    38300      0 172.25.198.71:389           172.25.195.198:38745       
CLOSE_WAIT  19444/slapd
tcp    83048      0 172.25.198.71:389           172.25.195.198:38744       
ESTABLISHED 19444/slapd
tcp     3433      0 172.25.198.71:389           172.25.195.198:38747       
CLOSE_WAIT  19444/slapd
tcp    11058      0 172.25.198.71:389           172.25.195.198:38746       
CLOSE_WAIT  19444/slapd
tcp    47204      0 172.25.198.71:389           172.25.195.198:38757       
CLOSE_WAIT  19444/slapd
tcp    36392      0 172.25.198.71:389           172.25.195.198:38756       
CLOSE_WAIT  19444/slapd
tcp     2341      0 172.25.198.71:389           172.25.195.198:38759       
CLOSE_WAIT  19444/slapd
tcp    73183      0 172.25.198.71:389           172.25.195.198:38758       
ESTABLISHED 19444/slapd
tcp    71796      0 172.25.198.71:389           172.25.195.198:38753       
CLOSE_WAIT  19444/slapd
tcp    78209      0 172.25.198.71:389           172.25.195.198:38752       
CLOSE_WAIT  19444/slapd
tcp    35915      0 172.25.198.71:389           172.25.195.198:38755       
CLOSE_WAIT  19444/slapd
tcp    12701      0 172.25.198.71:389           172.25.195.198:38754       
CLOSE_WAIT  19444/slapd
tcp    20333      0 172.25.198.71:389           172.25.195.198:38765       
CLOSE_WAIT  19444/slapd
tcp    74817      0 172.25.198.71:389           172.25.195.198:38764       
CLOSE_WAIT  19444/slapd
tcp    73757      0 172.25.198.71:389           172.25.195.198:38767       
CLOSE_WAIT  19444/slapd
tcp     2705      0 172.25.198.71:389           172.25.195.198:38766       
CLOSE_WAIT  19444/slapd
tcp    36180      0 172.25.198.71:389           172.25.195.198:38761       
CLOSE_WAIT  19444/slapd
tcp    75029      0 172.25.198.71:389           172.25.195.198:38760       
CLOSE_WAIT  19444/slapd
tcp    49112      0 172.25.198.71:389           172.25.195.198:38763       
CLOSE_WAIT  19444/slapd
tcp     8726      0 172.25.198.71:389           172.25.195.198:38762       
CLOSE_WAIT  19444/slapd
tcp       44      0 172.25.198.71:389           172.25.195.198:38772       
ESTABLISHED -
tcp     4005      0 172.25.198.71:389           172.25.195.198:38769       
CLOSE_WAIT  19444/slapd
tcp    17789      0 172.25.198.71:389           172.25.195.198:38768       
CLOSE_WAIT  19444/slapd
tcp     9945      0 172.25.198.71:389           172.25.195.198:38770       
CLOSE_WAIT  19444/slapd
tcp        0      0 :::389                      :::*                       
LISTEN      19444/slapd

Provides attachments at https://online-storage.yuuwaa.com/s/HkSdr/94bc6ba666

slapd_stuck.log - openldap logs
populBlits.ldif - ldif file to get a blitz population (that's what we used but
it works with any other data)
testWriteTO.pl - a Perl script to loop thru client disconnection
Prev by Date: (ITS#7295) Problem with TLS: threads and OpenSSL
Next by Date: Re: (ITS#7292) Memory leak with MMR + delta syncrepl
Index(es):
- Chronological
- Thread