[Date Prev][Date Next] [Chronological] [Thread] [Top]
RE: slapd stopping with no error message

To: "'Ralf Haferkamp'" <rhafer@suse.de>, <openldap-software@openldap.org>
Subject: RE: slapd stopping with no error message
From: "Douglas B. Jones" <douglas@gpc.edu>
Date: Thu, 15 Mar 2007 13:05:56 -0400
Cc: douglas@gpc.edu
In-reply-to: <200703141610.38731.rhafer@suse.de>
Organization: Georgia Perimeter College
References: <PHEKJLKKDLGCAEJLJPHMEEAPKCAA.douglas@gpc.edu> <200703141610.38731.rhafer@suse.de>
Thread-index: AcdmS4MDIc0RJExWQ/2q/e1GZOgSiQA2Emgg
Thank you for the link. re-ran it and got the following:

------------------------------------------------------------------------------------
(previous -d 1 output cut)
>>> dnPrettyNormal: <cn=Write,cn=Waiters,cn=Monitor>
<<< dnPrettyNormal: <cn=Write,cn=Waiters,cn=Monitor>, <cn=write,cn=waiters,cn=monitor>
>>> dnNormalize: <>
<<< dnNormalize: <>
>>> dnNormalize: <>
<<< dnNormalize: <>
<= str2entry(cn=Write,cn=Waiters,cn=Monitor) -> 0x260d9fd8
backend_startup_one: starting "o=gpc,c=us"
slapd starting
[New Thread 635861936 (LWP 10061)]
[New Thread 631663536 (LWP 10062)]
[New Thread 627465136 (LWP 10063)]
[New Thread 623266736 (LWP 10064)]
[New Thread 619068336 (LWP 10065)]
>>> slap_listener(ldap://:389/)
connection_get(33): got connid=0
connection_read(33): checking for input on id=0
ber_get_next
ber_get_next: tag 0x30 len 16 contents:
ber_get_next
do_bind
ber_scanf fmt ({imt) ber:
ber_scanf fmt (m}) ber:
>>> dnPrettyNormal: <>
<<< dnPrettyNormal: <>, <>
do_bind: version=3 dn="" method=128
send_ldap_result: conn=0 op=0 p=3
send_ldap_response: msgid=30 tag=97 err=0
ber_flush: 14 bytes to sd 33
connection_get(33): got connid=0
connection_read(33): checking for input on id=0
ber_get_next

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 635861936 (LWP 10061)]
ber_get_next (sb=0x25f2ffe8, len=0x25e6427c, ber=0x25f65fd4) at io.c:551
551                     if (*ber->ber_ptr & 0x80) {     /* multi-byte */
(gdb) bt full
#0  ber_get_next (sb=0x25f2ffe8, len=0x25e6427c, ber=0x25f65fd4) at io.c:551
        l = Variable "l" is not available.
(gdb) thread apply all bt

Thread 6 (Thread 619068336 (LWP 10065)):
#0  0x0028e7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x004e7b26 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/libpthread.so.0
#2  0x0812d25b in ldap_int_thread_pool_wrapper (xpool=0xb725df58)
    at tpool.c:490
#3  0x004e5371 in start_thread () from /lib/tls/libpthread.so.0
#4  0x0036effe in clone () from /lib/tls/libc.so.6

Thread 5 (Thread 623266736 (LWP 10064)):
#0  0x0028e7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x004e7b26 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/libpthread.so.0
#2  0x0812d25b in ldap_int_thread_pool_wrapper (xpool=0xb725df58)
    at tpool.c:490
#3  0x004e5371 in start_thread () from /lib/tls/libpthread.so.0
#4  0x0036effe in clone () from /lib/tls/libc.so.6

Thread 4 (Thread 627465136 (LWP 10063)):
#0  0x0028e7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x004e7b26 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib/tls/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#2  0x0812d25b in ldap_int_thread_pool_wrapper (xpool=0xb725df58)
    at tpool.c:490
#3  0x004e5371 in start_thread () from /lib/tls/libpthread.so.0
#4  0x0036effe in clone () from /lib/tls/libc.so.6

Thread 3 (Thread 631663536 (LWP 10062)):
#0  0x0028e7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x0036ba19 in mprotect () from /lib/tls/libc.so.6
#2  0x0814fb3f in Page_AllowAccess (address=0x25fa8000, size=475136)
    at page.c:149
#3  0x0814f6be in free (address=0xb7ea4ff0) at efence.c:694
#4  0x003232b5 in tzset_internal () from /lib/tls/libc.so.6
#5  0x00323d0e in tzset () from /lib/tls/libc.so.6
#6  0x003288ae in strftime_l () from /lib/tls/libc.so.6
#7  0x0036af6b in vsyslog () from /lib/tls/libc.so.6
#8  0x0036b4df in syslog () from /lib/tls/libc.so.6
#9  0x0806ab08 in slap_send_ldap_result (op=0x25f61e58, rs=0x25a66230)
    at result.c:585
#10 0x08075952 in fe_op_bind (op=0x25f61e58, rs=0x25a66230) at bind.c:327
#11 0x0807523d in do_bind (op=0x25f61e58, rs=0x25a66230) at bind.c:200
#12 0x0805eac4 in connection_operation (ctx=0x25a662b0, arg_v=0x25f61e58)
    at connection.c:1133
#13 0x0812d1df in ldap_int_thread_pool_wrapper (xpool=0xb725df58)
---Type <return> to continue, or q <return> to quit---
    at tpool.c:478
#14 0x004e5371 in start_thread () from /lib/tls/libpthread.so.0
#15 0x0036effe in clone () from /lib/tls/libc.so.6

Thread 2 (Thread 635861936 (LWP 10061)):
#0  ber_get_next (sb=0x25f2ffe8, len=0x25e6427c, ber=0x25f65fd4) at io.c:551
#1  0x0805eed0 in connection_read (s=33) at connection.c:1528
#2  0x0805baa2 in slapd_daemon_task (ptr=0x0) at daemon.c:2461
#3  0x004e5371 in start_thread () from /lib/tls/libpthread.so.0
#4  0x0036effe in clone () from /lib/tls/libc.so.6

Thread 1 (Thread -1208731424 (LWP 10058)):
#0  0x0028e7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x004e60fd in pthread_join () from /lib/tls/libpthread.so.0
#2  0x0805c621 in slapd_daemon () at daemon.c:2572
#3  0x0804dc9f in main (argc=13, argv=0xbff6ac14) at main.c:860
------------------------------------------------------------------------------------

It did return the names to my client, which was good, but just crashed after (during)
that. Thanks for all the help!

-----Original Message-----
From: openldap-software-bounces+douglas=gpc.edu@OpenLDAP.org [mailto:openldap-software-bounces+douglas=gpc.edu@OpenLDAP.org] On
Behalf Of Ralf Haferkamp
Sent: Wednesday, March 14, 2007 11:11 AM
To: openldap-software@openldap.org
Subject: Re: slapd stopping with no error message

On Sunday 11 March 2007 20:44, Douglas B. Jones wrote:
> If there is anymore gdb commands I can do, please
> let me know. Thanks!
This might help:
http://www.openldap.org/faq/index.cgi?_highlightWords=backtrace&file=59


> -----Original Message-----
> From: Douglas B. Jones [mailto:douglas@gpc.edu]
> Sent: Friday, March 09, 2007 3:15 PM
> To: Howard Chu
> Cc: openldap-software@openldap.org; Aaron Richton; Douglas B Jones
> Subject: RE: slapd stopping with no error message
>
>
>
> Thanks, I did not think about them being stripped! I reran it with:
>
>
>
> and got:
> %gdb ./servers/slapd/slapd
> (gdb)r -d 1 -u ldap3 -g ldap3 -h ldap://:389/ -l local6 -F \
> 					/usr/local/openldap-2.3.34/etc/openldap
> ( output skipped)
>
> >>> dnPrettyNormal: <>
>
> <<< dnPrettyNormal: <>, <>
> do_bind: version=3 dn="" method=128
> send_ldap_result: conn=0 op=0 p=3
> send_ldap_response: msgid=1 tag=97 err=0
> ber_flush: 14 bytes to sd 34
> do_bind: v3 anonymous bind
> connection_get(34): got connid=0
> connection_read(34): checking for input on id=0
> ber_get_next
>
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 635632560 (LWP 14158)]
> ber_get_next (sb=0x25efbfe8, len=0x25e2c27c, ber=0x2452cfd4) at io.c:551
> 551                     if (*ber->ber_ptr & 0x80) {     /* multi-byte */
> (gdb) where
> #0  ber_get_next (sb=0x25efbfe8, len=0x25e2c27c, ber=0x2452cfd4) at
> io.c:551 #1  0x0805eed0 in connection_read (s=34) at connection.c:1528
> #2  0x0805baa2 in slapd_daemon_task (ptr=0x0) at daemon.c:2461
> #3  0x004e5371 in start_thread () from /lib/tls/libpthread.so.0
> #4  0x0036effe in clone () from /lib/tls/libc.so.6
>
> If I can provide any more information, please let me know. Thanks!
> Thanks again for all the help!
>
> -----Original Message-----
> From: openldap-software-bounces+douglas=gpc.edu@OpenLDAP.org
> [mailto:openldap-software-bounces+douglas=gpc.edu@OpenLDAP.org]On Behalf
> Of Howard Chu
> Sent: Friday, March 09, 2007 3:05 PM
> To: Douglas B. Jones
> Cc: openldap-software@openldap.org; Aaron Richton
> Subject: Re: slapd stopping with no error message
>
> Douglas B. Jones wrote:
> > Thank you for the help, I thought that might be the case. I re-compiled
> > slapd (and all) with -g (I put those in the CFLAGS and exported it). I
> > then ran gdb with:
> >
> > gdb \
> >         -d /var/lib/ldap3/src/openldap-2.3.34/servers/slapd \
> >         -d /var/lib/ldap3/src/openldap-2.3.34/servers/slapd/back-bdd \
> >         -d /var/lib/ldap3/src/openldap-2.3.34/libraries/liblber \
> >         -d /var/lib/ldap3/src/openldap-2.3.34/libraries/libldap \
> >         -d /var/lib/ldap3/src/openldap-2.3.34/libraries/libldap_r\
> >         -d /var/lib/ldap3/src/openldap-2.3.34/libraries/libunicode\
> >         -d /var/lib/ldap3/src/openldap-2.3.34/libraries/libutil\
> >         -d /var/lib/ldap3/src/openldap-2.3.34/libraries/librewrite\
>
> None of those options are necessary.
>
> >         /usr/local/openldap-2.3.34/libexec/slapd
>
> By default the installed executables are stripped. You could just debug
> the original binary in the build tree. (Or you can redo the install in
> the slapd directory with "make install STRIP=" )
>
> > and run it with:
> >
> > (gdb) r -d 1 -u ldap3 -h ldap://:389/ -l LOCAL6 -f \
> > 	/usr/local/openldap-2.3.34/etc/openldap/slapd.conf -F \
> > 	/usr/local/openldap-2.3.34/etc/openldap -u ldap3 -g ldap3
>
> Have you read the slapd(8) manpage? Why are you providing both -f and
> -F? You have -u twice.
>
> > and when in the debugger it crashes on the first search and I type
> > in where in gdb:
> >
> > (first part here last of output from -d 1 for slapd)
> >
> >>>> dnNormalize: <>
> >
> > <<< dnNormalize: <>
> >
> >>>> dnNormalize: <>
> >
> > <<< dnNormalize: <>
> > <= str2entry(cn=Write,cn=Waiters,cn=Monitor) -> 0x25eaafd8
> > backend_startup_one: starting "o=gpc,c=us"
> > slapd starting
> > [New Thread 634641328 (LWP 14073)]
> > [New Thread 630442928 (LWP 14074)]
> > [New Thread 626244528 (LWP 14075)]
> > [New Thread 622046128 (LWP 14076)]
> >
> >>>> slap_listener(ldap://:389/)
> >
> > connection_get(34): got connid=0
> > connection_read(34): checking for input on id=0
> > ber_get_next
> > ber_get_next: tag 0x30 len 12 contents:
> > ber_get_next
> > do_bind
> > ber_scanf fmt ({imt) ber:
> >
> > ber_scanf fmt (m}) ber:
> >>>> dnPrettyNormal: <>
> >
> > <<< dnPrettyNormal: <>, <>
> > do_bind: version=3 dn="" method=128
> > send_ldap_result: conn=0 op=0 p=3
> > send_ldap_response: msgid=1 tag=97 err=0
> > ber_flush: 14 bytes to sd 34
> > connection_get(34): got connid=0
> > connection_read(34): checking for input on id=0
> > ber_get_next
> >
> > Program received signal SIGSEGV, Segmentation fault.
> > [Switching to Thread 634641328 (LWP 14073)]
> > 0x0814c6bf in ?? ()
> > (gdb) where
> > #0  0x0814c6bf in ?? ()
> > #1  0x00000001 in ?? ()
> > #2  0x0000002c in ?? ()
> > #3  0x25f1cfe7 in ?? ()
> > #4  0x25f1cfdc in ?? ()
> > #5  0x25f18f4c in ?? ()
> > #6  0x25d410e0 in ?? ()
> > #7  0x25d3a268 in ?? ()
> > #8  0x00000000 in ?? ()
> > (gdb)
> >
> > Unfortunately, I am not getting any reference to the C source files.
> > What am I do wrong? Eventually, hopefully, I will know linux better.
> > Thanks for any help....
> >
> > -----Original Message-----
> > From: openldap-software-bounces+douglas=gpc.edu@OpenLDAP.org
> > [mailto:openldap-software-bounces+douglas=gpc.edu@OpenLDAP.org]On Behalf
> > Of Aaron Richton
> > Sent: Friday, March 09, 2007 12:37 PM
> > To: Douglas B. Jones
> > Cc: Howard Chu; openldap-software@openldap.org
> > Subject: RE: slapd stopping with no error message
> >
> >
> > Try "where". Make sure that you have line numbers from the -g compile
> > (e.g. "init.c:354"); otherwise the trace is near impossible to use.
> >
> > On Fri, 9 Mar 2007, Douglas B. Jones wrote:
> >> I apologize, but I am not familiar with gdb. I attached to the process,
> >> it crashed with a seg fault. What should I be typing in now? Thanks!
> >>
> >> -----Original Message-----
> >> From: openldap-software-bounces+douglas=gpc.edu@OpenLDAP.org
> >> [mailto:openldap-software-bounces+douglas=gpc.edu@OpenLDAP.org]On Behalf
> >> Of Douglas B. Jones
> >> Sent: Friday, March 09, 2007 8:34 AM
> >> To: Howard Chu
> >> Cc: Douglas B Jones; openldap-software@openldap.org
> >> Subject: RE: slapd stopping with no error message
> >>
> >>
> >>
> >> Thanks. I am compiling with efence library right now. I am not too
> >> worried about the speed, it is not prod. I would like to run it under
> >> gdb, and I will try that, the problem is that if I loose my connection
> >> the only thing I can think of is to kill the other gdb, then re-attach.
> >> I guess that would not cause a problem. I was hoping with the cd and
> >> ulimit that the core would be generated and then gdb on that. Thanks!
> >>
> >> -----Original Message-----
> >> From: openldap-software-bounces+douglas=gpc.edu@OpenLDAP.org
> >> [mailto:openldap-software-bounces+douglas=gpc.edu@OpenLDAP.org]On Behalf
> >> Of Howard Chu
> >> Sent: Friday, March 09, 2007 12:55 AM
> >> To: Douglas B. Jones
> >> Cc: openldap-software@openldap.org
> >> Subject: Re: slapd stopping with no error message
> >>
> >> Douglas B. Jones wrote:
> >>> The .34 failed. I set ulimit -c 100000 and found no core on the system.
> >>> I am running it again in case I made a mistake. It is compiled with
> >>> debug. More later. Thanks!
> >>
> >> Still better to run under gdb. But given the fact that you're getting
> >> heap corruption, you're really going to need something like efence in
> >> addition. Unfortunately that's going to make things run very slowly, but
> >> that's the only sure way to identify the root cause of the problem.
> >>
> >>> -----Original Message-----
> >>> From: openldap-software-bounces+douglas=gpc.edu@openldap.org
> >>> [mailto:openldap-software-bounces+douglas=gpc.edu@openldap.org] On
> >>> Behalf Of Douglas B. Jones
> >>> Sent: Friday, February 23, 2007 3:30 PM
> >>> To: openldap-software@openldap.org
> >>> Cc: Douglas B. Jones
> >>> Subject: RE: slapd stopping with no error message
> >>>
> >>>
> >>> I attached a gdb to the new process plus the -d 7 output (you know,
> >>> that neat thing that unix has called redirection). Hopefully, the
> >>> connection I have to the computer will not be lost and I will be
> >>> able to get the trace if it fails. Also, not the results below are
> >>> just the last part of the output. Let me know if I need more. For
> >>> what it is worth, I did the 'he' search shown at the bottom below
> >>> and it did not fail.... Thanks!
> >>>
> >>> -----Original Message-----
> >>> From: openldap-software-bounces+douglas=gpc.edu@openldap.org
> >>> [mailto:openldap-software-bounces+douglas=gpc.edu@openldap.org]On
> >>> Behalf Of Douglas B. Jones
> >>> Sent: Friday, February 23, 2007 2:22 PM
> >>> To: openldap-software@openldap.org
> >>> Cc: douglas@gpc.edu
> >>> Subject: RE: slapd stopping with no error message
> >>>
> >>>
> >>>
> >>> Finally failed, I did not have a gdb on it, but did have -d 7. Below
> >>> are the results. Thanks!
> >>>
> >>> do_search
> >>> ldap_read: want=8 error=Resource temporarily unavailable
> >>>
> >>> ber_scanf fmt ({miiiib) ber:
> >>>>>> dnPrettyNormal: <o=gpc,c=us>
> >>>
> >>> => ldap_bv2dn(o=gpc,c=us,0)
> >>> <= ldap_bv2dn(o=gpc,c=us)=0
> >>> => ldap_dn2bv(272)
> >>> <= ldap_dn2bv(o=gpc,c=us)=0
> >>> => ldap_dn2bv(272)
> >>> <= ldap_dn2bv(o=gpc,c=us)=0
> >>> <<< dnPrettyNormal: <o=gpc,c=us>, <o=gpc,c=us>
> >>> SRCH "o=gpc,c=us" 2 0    100 3600 0
> >>> ber_scanf fmt ({m) ber:
> >>> ber_scanf fmt (m) ber:
> >>> ber_scanf fmt ({m) ber:
> >>> ber_scanf fmt (m) ber:
> >>> ber_scanf fmt ({m) ber:
> >>> ber_scanf fmt (m) ber:
> >>>     filter: (|(cn=he*)(mail=he*)(sn=he*))
> >>> ber_scanf fmt ({M}}) ber:
> >>>     attrs: cn mail
> >>> *** glibc detected *** corrupted double-linked list: 0x29400048 ***
> >>> /etc/init.d/gpcldap-33: line 51: 11051 Aborted                 ${slapd}
> >>> -d 7 -u $user -h "ldap://:$P/"; $OPTIONS $SLAPD_OPTIONS
> >>
> >> --
> >>   -- Howard Chu
> >>   Chief Architect, Symas Corp.  http://www.symas.com
> >>   Director, Highland Sun        http://highlandsun.com/hyc
> >>   Chief Architect, OpenLDAP     http://www.openldap.org/project/

-- 
Ralf Haferkamp
SUSE LINUX Products GmbH, Maxfeldstrasse 5, D-90409 Nuernberg
T: +49-911-74053-0
F: +49-911-74053575 - Ralf.Haferkamp@suse.com
References:
- RE: slapd stopping with no error message
  - From: "Douglas B. Jones" <douglas@gpc.edu>
- Re: slapd stopping with no error message
  - From: Ralf Haferkamp <rhafer@suse.de>
Prev by Date: Re: Error: Objectclass: value #1 invalid per syntax
Next by Date: addressbook ACLs - cannot create contact in group address book.
Index(es):
- Chronological
- Thread