[Date Prev][Date Next] [Chronological] [Thread] [Top]
Re: (ITS#5354) slapd repeatedly hangs and stops reponding

To: openldap-its@OpenLDAP.org
Subject: Re: (ITS#5354) slapd repeatedly hangs and stops reponding
From: orenl@cs.columbia.edu
Date: Fri, 8 Feb 2008 01:36:47 GMT
This is a multi-part message in MIME format.
--------------080905060502070406050804
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

sometimes it happens faster than other times ... here is another
backtrace, this time with the symbols.

Oren.


hyc@symas.com wrote:
> quanah@zimbra.com wrote:
>> --On Thursday, February 07, 2008 6:56 PM -0500 Oren Laadan
>> <orenl@cs.columbia.edu>  wrote:
>>
>>> More threads, less threads -- it still happens :(
>>>
>>> Attached is the last part of the log before it stopped responding
>>> (you can see towards the end the time gap).
>>>
>>> Also attached is the backtrace of all threads (thanks to gdb).
>> Your backtrace is fairly useless.  You need to do a make install STRIP=""
>> so it doesn't strip the binaries on installation.  Or copy over the slapd
>> from your build area, as that is pre-stripping.  Additionally, I'd add the
>> patch from ITS#5341 to your build.
>>
>> Reading symbols from /usr/local/opt/ldap-2.4.7/libexec/slapd...(no
>> debugging symbols found)...done.
> 
> It shows enough; back-meta is hanging waiting for responses from some other 
> LDAP server. This is a pretty bad configuration; you should not use back-meta 
> (or back-ldap) to redirect queries back into the same slapd. You should use 
> back-relay instead.

--------------080905060502070406050804
Content-Type: text/plain;
 name="ccc"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="ccc"

LOCALHOST_ROOT:/var/log # gdb -p 982
GNU gdb 6.7.1-debian
Copyright (C) 2007 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i486-linux-gnu".
Attaching to process 982
Reading symbols from /usr/local/opt/ldap-2.3.39/libexec/slapd...done.
Using host libthread_db library "/lib/i686/cmov/libthread_db.so.1".
Reading symbols from /usr/local/opt/ldap-2.3.39/lib/libldap_r-2.3.so.0...done.
Loaded symbols for /opt/ldap-2.3.39/lib/libldap_r-2.3.so.0
Reading symbols from /usr/local/opt/ldap-2.3.39/lib/liblber-2.3.so.0...done.
Loaded symbols for /opt/ldap-2.3.39/lib/liblber-2.3.so.0
Reading symbols from /usr/lib/libdb-4.5.so...done.
Loaded symbols for /usr/lib/libdb-4.5.so
Reading symbols from /lib/i686/cmov/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 0xf7a5d6b0 (LWP 982)]
[New Thread 0xf03ffb90 (LWP 998)]
[New Thread 0xf1ed2b90 (LWP 997)]
[New Thread 0xf27d4b90 (LWP 996)]
[New Thread 0xf30d6b90 (LWP 995)]
[New Thread 0xf39d8b90 (LWP 994)]
[New Thread 0xf5addb90 (LWP 986)]
[New Thread 0xf63dfb90 (LWP 985)]
[New Thread 0xf6ce1b90 (LWP 984)]
[New Thread 0xf74e2b90 (LWP 983)]
Loaded symbols for /lib/i686/cmov/libpthread.so.0
Reading symbols from /usr/lib/libsasl2.so.2...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /lib/i686/cmov/libdl.so.2...done.
Loaded symbols for /lib/i686/cmov/libdl.so.2
Reading symbols from /usr/lib/i686/cmov/libssl.so.0.9.8...done.
Loaded symbols for /usr/lib/i686/cmov/libssl.so.0.9.8
Reading symbols from /usr/lib/i686/cmov/libcrypto.so.0.9.8...done.
Loaded symbols for /usr/lib/i686/cmov/libcrypto.so.0.9.8
Reading symbols from /lib/i686/cmov/libresolv.so.2...done.
Loaded symbols for /lib/i686/cmov/libresolv.so.2
Reading symbols from /lib/i686/cmov/libc.so.6...done.
Loaded symbols for /lib/i686/cmov/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/lib/libz.so.1...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /lib/i686/cmov/libnss_files.so.2...done.
Loaded symbols for /lib/i686/cmov/libnss_files.so.2
Reading symbols from /usr/lib/sasl2/libsasldb.so.2...done.
Loaded symbols for /usr/lib/sasl2/libsasldb.so.2
Reading symbols from /usr/lib/libdb-4.4.so...done.
Loaded symbols for /usr/lib/libdb-4.4.so
Reading symbols from /usr/lib/sasl2/libntlm.so.2...done.
Loaded symbols for /usr/lib/sasl2/libntlm.so.2
Reading symbols from /usr/lib/sasl2/libplain.so.2...done.
Loaded symbols for /usr/lib/sasl2/libplain.so.2
Reading symbols from /lib/i686/cmov/libcrypt.so.1...done.
Loaded symbols for /lib/i686/cmov/libcrypt.so.1
Reading symbols from /usr/lib/sasl2/liblogin.so.2...done.
Loaded symbols for /usr/lib/sasl2/liblogin.so.2
Reading symbols from /usr/lib/sasl2/libdigestmd5.so.2...done.
Loaded symbols for /usr/lib/sasl2/libdigestmd5.so.2
Reading symbols from /usr/lib/sasl2/libcrammd5.so.2...done.
Loaded symbols for /usr/lib/sasl2/libcrammd5.so.2
Reading symbols from /usr/lib/sasl2/libanonymous.so.2...done.
Loaded symbols for /usr/lib/sasl2/libanonymous.so.2

warning: Lowest section in system-supplied DSO at 0xffffe000 is .hash at ffffe0b4
0xffffe405 in __kernel_vsyscall ()
(gdb) thread apply all bt

Thread 10 (Thread 0xf74e2b90 (LWP 983)):
#0  0xffffe405 in __kernel_vsyscall ()
#1  0xf7b4af76 in epoll_wait () from /lib/i686/cmov/libc.so.6
#2  0x0805e8e6 in slapd_daemon_task (ptr=0x0) at daemon.c:2174
#3  0xf7d814fb in start_thread () from /lib/i686/cmov/libpthread.so.0
#4  0xf7b4a75e in clone () from /lib/i686/cmov/libc.so.6

Thread 9 (Thread 0xf6ce1b90 (LWP 984)):
#0  0xffffe405 in __kernel_vsyscall ()
#1  0xf7b404c7 in poll () from /lib/i686/cmov/libc.so.6
#2  0xf7ef7555 in ldap_int_select (ld=0x84c9710, timeout=0xf6cdfda0)
    at os-ip.c:930
#3  0xf7ee3b41 in ldap_result (ld=0x84c9710, msgid=42, all=2, 
    timeout=0xf6ce00a0, result=0xf6ce00b4) at result.c:299
#4  0x080dbbe5 in meta_back_search (op=0x878c2c8, rs=0xf6ce11c4)
    at search.c:1007
#5  0x08063c39 in fe_op_search (op=0x878c2c8, rs=0xf6ce11c4) at search.c:355
#6  0x0806455a in do_search (op=0x878c2c8, rs=0xf6ce11c4) at search.c:217
#7  0x08061ced in connection_operation (ctx=0xf6ce1248, arg_v=0x878c2c8)
    at connection.c:1133
#8  0xf7ee09ae in ldap_int_thread_pool_wrapper (xpool=0x81f3da0) at tpool.c:478
#9  0xf7d814fb in start_thread () from /lib/i686/cmov/libpthread.so.0
#10 0xf7b4a75e in clone () from /lib/i686/cmov/libc.so.6

Thread 8 (Thread 0xf63dfb90 (LWP 985)):
#0  0xffffe405 in __kernel_vsyscall ()
#1  0xf7b404c7 in poll () from /lib/i686/cmov/libc.so.6
#2  0xf7ef7555 in ldap_int_select (ld=0x87c00b8, timeout=0xf63ddda0)
    at os-ip.c:930
#3  0xf7ee3b41 in ldap_result (ld=0x87c00b8, msgid=42, all=2, 
    timeout=0xf63de0a0, result=0xf63de0b4) at result.c:299
#4  0x080dbbe5 in meta_back_search (op=0x878c0e8, rs=0xf63df1c4)
    at search.c:1007
#5  0x08063c39 in fe_op_search (op=0x878c0e8, rs=0xf63df1c4) at search.c:355
#6  0x0806455a in do_search (op=0x878c0e8, rs=0xf63df1c4) at search.c:217
#7  0x08061ced in connection_operation (ctx=0xf63df248, arg_v=0x878c0e8)
    at connection.c:1133
#8  0xf7ee09ae in ldap_int_thread_pool_wrapper (xpool=0x81f3da0) at tpool.c:478
#9  0xf7d814fb in start_thread () from /lib/i686/cmov/libpthread.so.0
#10 0xf7b4a75e in clone () from /lib/i686/cmov/libc.so.6

Thread 7 (Thread 0xf5addb90 (LWP 986)):
#0  0xffffe405 in __kernel_vsyscall ()
#1  0xf7b404c7 in poll () from /lib/i686/cmov/libc.so.6
#2  0xf7ef7555 in ldap_int_select (ld=0x87e05d0, timeout=0xf5adbda0)
    at os-ip.c:930
#3  0xf7ee3b41 in ldap_result (ld=0x87e05d0, msgid=42, all=2, 
    timeout=0xf5adc0a0, result=0xf5adc0b4) at result.c:299
#4  0x080dbbe5 in meta_back_search (op=0x8333730, rs=0xf5add1c4)
    at search.c:1007
#5  0x08063c39 in fe_op_search (op=0x8333730, rs=0xf5add1c4) at search.c:355
#6  0x0806455a in do_search (op=0x8333730, rs=0xf5add1c4) at search.c:217
#7  0x08061ced in connection_operation (ctx=0xf5add248, arg_v=0x8333730)
    at connection.c:1133
#8  0xf7ee09ae in ldap_int_thread_pool_wrapper (xpool=0x81f3da0) at tpool.c:478
---Type <return> to continue, or q <return> to quit--- 
#9  0xf7d814fb in start_thread () from /lib/i686/cmov/libpthread.so.0
#10 0xf7b4a75e in clone () from /lib/i686/cmov/libc.so.6

Thread 6 (Thread 0xf39d8b90 (LWP 994)):
#0  0xffffe405 in __kernel_vsyscall ()
#1  0xf7b404c7 in poll () from /lib/i686/cmov/libc.so.6
#2  0xf7ef7555 in ldap_int_select (ld=0x829eb88, timeout=0xf39d6da0)
    at os-ip.c:930
#3  0xf7ee3b41 in ldap_result (ld=0x829eb88, msgid=46, all=2, 
    timeout=0xf39d70a0, result=0xf39d70b4) at result.c:299
#4  0x080dbbe5 in meta_back_search (op=0x8356748, rs=0xf39d81c4)
    at search.c:1007
#5  0x08063c39 in fe_op_search (op=0x8356748, rs=0xf39d81c4) at search.c:355
#6  0x0806455a in do_search (op=0x8356748, rs=0xf39d81c4) at search.c:217
#7  0x08061ced in connection_operation (ctx=0xf39d8248, arg_v=0x8356748)
    at connection.c:1133
#8  0xf7ee09ae in ldap_int_thread_pool_wrapper (xpool=0x81f3da0) at tpool.c:478
#9  0xf7d814fb in start_thread () from /lib/i686/cmov/libpthread.so.0
#10 0xf7b4a75e in clone () from /lib/i686/cmov/libc.so.6

Thread 5 (Thread 0xf30d6b90 (LWP 995)):
#0  0xffffe405 in __kernel_vsyscall ()
#1  0xf7b404c7 in poll () from /lib/i686/cmov/libc.so.6
#2  0xf7ef7555 in ldap_int_select (ld=0x83788b8, timeout=0xf30d4da0)
    at os-ip.c:930
#3  0xf7ee3b41 in ldap_result (ld=0x83788b8, msgid=46, all=2, 
    timeout=0xf30d50a0, result=0xf30d50b4) at result.c:299
#4  0x080dbbe5 in meta_back_search (op=0x8333910, rs=0xf30d61c4)
    at search.c:1007
#5  0x08063c39 in fe_op_search (op=0x8333910, rs=0xf30d61c4) at search.c:355
#6  0x0806455a in do_search (op=0x8333910, rs=0xf30d61c4) at search.c:217
#7  0x08061ced in connection_operation (ctx=0xf30d6248, arg_v=0x8333910)
    at connection.c:1133
#8  0xf7ee09ae in ldap_int_thread_pool_wrapper (xpool=0x81f3da0) at tpool.c:478
#9  0xf7d814fb in start_thread () from /lib/i686/cmov/libpthread.so.0
#10 0xf7b4a75e in clone () from /lib/i686/cmov/libc.so.6

Thread 4 (Thread 0xf27d4b90 (LWP 996)):
#0  0xffffe405 in __kernel_vsyscall ()
#1  0xf7b404c7 in poll () from /lib/i686/cmov/libc.so.6
#2  0xf7ef7555 in ldap_int_select (ld=0xf04a5d60, timeout=0xf27d2da0)
    at os-ip.c:930
#3  0xf7ee3b41 in ldap_result (ld=0xf04a5d60, msgid=15, all=2, 
    timeout=0xf27d30a0, result=0xf27d30b4) at result.c:299
#4  0x080dbbe5 in meta_back_search (op=0x828c140, rs=0xf27d41c4)
    at search.c:1007
#5  0x08063c39 in fe_op_search (op=0x828c140, rs=0xf27d41c4) at search.c:355
#6  0x0806455a in do_search (op=0x828c140, rs=0xf27d41c4) at search.c:217
#7  0x08061ced in connection_operation (ctx=0xf27d4248, arg_v=0x828c140)
    at connection.c:1133
#8  0xf7ee09ae in ldap_int_thread_pool_wrapper (xpool=0x81f3da0) at tpool.c:478
#9  0xf7d814fb in start_thread () from /lib/i686/cmov/libpthread.so.0
#10 0xf7b4a75e in clone () from /lib/i686/cmov/libc.so.6

Thread 3 (Thread 0xf1ed2b90 (LWP 997)):
#0  0xffffe405 in __kernel_vsyscall ()
---Type <return> to continue, or q <return> to quit---
#1  0xf7b404c7 in poll () from /lib/i686/cmov/libc.so.6
#2  0xf7ef7555 in ldap_int_select (ld=0x83d3078, timeout=0xf1ed0da0)
    at os-ip.c:930
#3  0xf7ee3b41 in ldap_result (ld=0x83d3078, msgid=42, all=2, 
    timeout=0xf1ed10a0, result=0xf1ed10b4) at result.c:299
#4  0x080dbbe5 in meta_back_search (op=0x83b6900, rs=0xf1ed21c4)
    at search.c:1007
#5  0x08063c39 in fe_op_search (op=0x83b6900, rs=0xf1ed21c4) at search.c:355
#6  0x0806455a in do_search (op=0x83b6900, rs=0xf1ed21c4) at search.c:217
#7  0x08061ced in connection_operation (ctx=0xf1ed2248, arg_v=0x83b6900)
    at connection.c:1133
#8  0xf7ee09ae in ldap_int_thread_pool_wrapper (xpool=0x81f3da0) at tpool.c:478
#9  0xf7d814fb in start_thread () from /lib/i686/cmov/libpthread.so.0
#10 0xf7b4a75e in clone () from /lib/i686/cmov/libc.so.6

Thread 2 (Thread 0xf03ffb90 (LWP 998)):
#0  0xffffe405 in __kernel_vsyscall ()
#1  0xf7b404c7 in poll () from /lib/i686/cmov/libc.so.6
#2  0xf7ef7555 in ldap_int_select (ld=0xf04a5820, timeout=0xf03fdda0)
    at os-ip.c:930
#3  0xf7ee3b41 in ldap_result (ld=0xf04a5820, msgid=15, all=2, 
    timeout=0xf03fe0a0, result=0xf03fe0b4) at result.c:299
#4  0x080dbbe5 in meta_back_search (op=0x83451a0, rs=0xf03ff1c4)
    at search.c:1007
#5  0x08063c39 in fe_op_search (op=0x83451a0, rs=0xf03ff1c4) at search.c:355
#6  0x0806455a in do_search (op=0x83451a0, rs=0xf03ff1c4) at search.c:217
#7  0x08061ced in connection_operation (ctx=0xf03ff248, arg_v=0x83451a0)
    at connection.c:1133
#8  0xf7ee09ae in ldap_int_thread_pool_wrapper (xpool=0x81f3da0) at tpool.c:478
#9  0xf7d814fb in start_thread () from /lib/i686/cmov/libpthread.so.0
#10 0xf7b4a75e in clone () from /lib/i686/cmov/libc.so.6

Thread 1 (Thread 0xf7a5d6b0 (LWP 982)):
#0  0xffffe405 in __kernel_vsyscall ()
#1  0xf7d82775 in pthread_join () from /lib/i686/cmov/libpthread.so.0
#2  0xf7ee1294 in ldap_pvt_thread_join (thread=4149095312, thread_return=0x0)
    at thr_posix.c:193
#3  0x0805cc23 in slapd_daemon () at daemon.c:2579
#4  0x0804faa8 in main (argc=9, argv=0xff8a5594) at main.c:859
#0  0xffffe405 in __kernel_vsyscall ()
(gdb) 

--------------080905060502070406050804--
Prev by Date: Re: (ITS#5354) slapd repeatedly hangs and stops reponding
Next by Date: Re: (ITS#5354) slapd repeatedly hangs and stops reponding
Index(es):
- Chronological
- Thread