[Date Prev][Date Next] [Chronological] [Thread] [Top]
Re: (ITS#4308) poor performance under load

To: openldap-its@OpenLDAP.org
Subject: Re: (ITS#4308) poor performance under load
From: hyc@symas.com
Date: Fri, 6 Jan 2006 01:08:55 GMT
Spicer, Kevin wrote:
> -----Original Message-----
> From: Howard Chu [mailto:hyc@symas.com] 
>
>   
>> Attach to the slapd process with gdb and get a stack trace. We need to 
>> see what that CPU hogging thread is doing.
>>     
>
> I have to confess I don't know very much about using gdb, and have never
> done it with threads before - so I hope you'll excuse me and point me in
> the right direction if this isn't what you want...
>   

That's about right, but we need a slapd built without optimization, and 
with debug symbols intact. The stack trace you got here shows a sequence 
that doesn't occur in the code; this often happens on optimized code.
> ######
> First... output of prstat -L 
> ######
>
>    PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/LWPID
>  22870 ldap      258M   33M stop    30    0   0:00:26  17% slapd/2
>  22870 ldap      258M   33M stop    49    0   0:00:01 0.9% slapd/7
>  22870 ldap      258M   33M stop    49    0   0:00:01 0.7% slapd/5
>  22870 ldap      258M   33M stop    49    0   0:00:01 0.7% slapd/4
>  22870 ldap      258M   33M stop    49    0   0:00:00 0.7% slapd/8
>  22870 ldap      258M   33M stop    49    0   0:00:01 0.6% slapd/6
>  22870 ldap      258M   33M stop    49    0   0:00:01 0.6% slapd/3
>  22870 ldap      258M   33M stop    49    0   0:00:00 0.5% slapd/13
>  22870 ldap      258M   33M stop    49    0   0:00:00 0.4% slapd/10
>  22870 ldap      258M   33M stop    49    0   0:00:00 0.4% slapd/9
>  22870 ldap      258M   33M stop    49    0   0:00:00 0.4% slapd/17
>  22870 ldap      258M   33M stop    49    0   0:00:00 0.4% slapd/11
>  22870 ldap      258M   33M stop    49    0   0:00:00 0.4% slapd/12
>  22870 ldap      258M   33M stop    49    0   0:00:00 0.4% slapd/18
>  22870 ldap      258M   33M stop    49    0   0:00:00 0.4% slapd/15
>  22870 ldap      258M   33M stop    49    0   0:00:00 0.4% slapd/14
>  22870 ldap      258M   33M stop    49    0   0:00:00 0.3% slapd/16
>  22870 ldap      258M   33M stop    59    0   0:00:00 0.0% slapd/1
>
> ######
> now gdb (I've included the full output so you can see if I'm doing
> anything wrong!)
> ######
>
> GNU gdb 6.0
> Copyright 2003 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you
> are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for
> details.
> This GDB was configured as "sparc-sun-solaris2.9".
> (gdb) attach 22870
> Attaching to process 22870
> Reading symbols from /usr/local/libexec/slapd...(no debugging symbols
> found)...
> done.
> Reading symbols from /usr/local/ssl/lib/libssl.so.0.9.7...
> (no debugging symbols found)...done.
> Loaded symbols for /usr/local/ssl/lib/libssl.so.0.9.7
> Reading symbols from /usr/local/ssl/lib/libcrypto.so.0.9.7...
> (no debugging symbols found)...done.
> Loaded symbols for /usr/local/ssl/lib/libcrypto.so.0.9.7
> Reading symbols from /usr/lib/libresolv.so.2...(no debugging symbols
> found)...
> done.
> Loaded symbols for /usr/lib/libresolv.so.2
> Reading symbols from /usr/lib/libgen.so.1...(no debugging symbols
> found)...
> done.
> Loaded symbols for /usr/lib/libgen.so.1
> Reading symbols from /usr/lib/libnsl.so.1...(no debugging symbols
> found)...
> done.
> Loaded symbols for /usr/lib/libnsl.so.1
> Reading symbols from /usr/lib/libsocket.so.1...(no debugging symbols
> found)...
> done.
> Loaded symbols for /usr/lib/libsocket.so.1
> Reading symbols from /usr/lib/libdl.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib/libdl.so.1
> Reading symbols from /usr/sfw/lib/libwrap.so.1...
> (no debugging symbols found)...done.
> Loaded symbols for /usr/sfw/lib/libwrap.so.1
> Reading symbols from /usr/lib/libpthread.so.1...(no debugging symbols
> found)...
> done.
> Loaded symbols for /usr/lib/libpthread.so.1
> Reading symbols from /usr/lib/libc.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib/libc.so.1
> Reading symbols from /usr/local/lib/libgcc_s.so.1...done.
> Loaded symbols for /usr/local/lib/libgcc_s.so.1
> Reading symbols from /usr/lib/libmp.so.2...done.
> Loaded symbols for /usr/lib/libmp.so.2
> ---Type <return> to continue, or q <return> to quit---
> Reading symbols from
> /usr/platform/SUNW,Sun-Fire-V240/lib/libc_psr.so.1...done.
> Loaded symbols for /usr/platform/SUNW,Sun-Fire-V240/lib/libc_psr.so.1
> Reading symbols from /usr/lib/libthread.so.1...done.
> warning: sol_thread_new_objfile: td_ta_new: Debugger service failed
> Loaded symbols for /usr/lib/libthread.so.1
> Reading symbols from /usr/lib/libldap.so.5...done.
> warning: sol_thread_new_objfile: td_ta_new: Debugger service failed
> Loaded symbols for /usr/lib/libldap.so.5
> Reading symbols from /usr/lib/librt.so.1...done.
> warning: sol_thread_new_objfile: td_ta_new: Debugger service failed
> Loaded symbols for /usr/lib/librt.so.1
> Reading symbols from /usr/lib/libmd5.so.1...done.
> warning: sol_thread_new_objfile: td_ta_new: Debugger service failed
> Loaded symbols for /usr/lib/libmd5.so.1
> Reading symbols from /usr/lib/libaio.so.1...done.
> warning: sol_thread_new_objfile: td_ta_new: Debugger service failed
> Loaded symbols for /usr/lib/libaio.so.1
> Reading symbols from
> /usr/local/libexec/openldap/ppolicy-2.3.so.0...done.
> warning: sol_thread_new_objfile: td_ta_new: Debugger service failed
> Loaded symbols for /usr/local/libexec/openldap/ppolicy-2.3.so.0
> Reading symbols from /usr/local/lib/libltdl.so.3...done.
> warning: sol_thread_new_objfile: td_ta_new: Debugger service failed
> Loaded symbols for /usr/local/lib/libltdl.so.3
> Retry #1:
> Retry #2:
> Retry #3:
> Retry #4:
> [New LWP 1]
> Symbols already loaded for /usr/local/ssl/lib/libssl.so.0.9.7
> Symbols already loaded for /usr/local/ssl/lib/libcrypto.so.0.9.7
> Symbols already loaded for /usr/lib/libresolv.so.2
> Symbols already loaded for /usr/lib/libgen.so.1
> Symbols already loaded for /usr/lib/libnsl.so.1
> Symbols already loaded for /usr/lib/libsocket.so.1
> Symbols already loaded for /usr/lib/libdl.so.1
> Symbols already loaded for /usr/sfw/lib/libwrap.so.1
> Symbols already loaded for /usr/lib/libpthread.so.1
> Symbols already loaded for /usr/lib/libc.so.1
> Symbols already loaded for /usr/local/lib/libgcc_s.so.1
> Symbols already loaded for /usr/lib/libmp.so.2
> Symbols already loaded for
> /usr/platform/SUNW,Sun-Fire-V240/lib/libc_psr.so.1
> Symbols already loaded for /usr/lib/libthread.so.1
> Symbols already loaded for /usr/lib/libldap.so.5
> Symbols already loaded for /usr/lib/librt.so.1
> Symbols already loaded for /usr/lib/libmd5.so.1
> Symbols already loaded for /usr/lib/libaio.so.1
> Symbols already loaded for /usr/local/libexec/openldap/ppolicy-2.3.so.0
> Symbols already loaded for /usr/local/lib/libltdl.so.3
> 0xff01f870 in _lwp_wait () from /usr/lib/libc.so.1
> (gdb) info threads
>   18 LWP 2  0xff01fb74 in time () from /usr/lib/libc.so.1
>   17 LWP 3  0xfef158f4 in __lwp_park () from /usr/lib/libthread.so.1
>   16 LWP 4  0xfef158f4 in __lwp_park () from /usr/lib/libthread.so.1
>   15 LWP 5  0xfef158f4 in __lwp_park () from /usr/lib/libthread.so.1
>   14 LWP 6  0xfef158f4 in __lwp_park () from /usr/lib/libthread.so.1
>   13 LWP 7  0xfef158f4 in __lwp_park () from /usr/lib/libthread.so.1
>   12 LWP 8  0xfef158f4 in __lwp_park () from /usr/lib/libthread.so.1
>   11 LWP 9  0xfef158f4 in __lwp_park () from /usr/lib/libthread.so.1
>   10 LWP 10  0xfef158f4 in __lwp_park () from /usr/lib/libthread.so.1
>   9 LWP 11  0xfef158f4 in __lwp_park () from /usr/lib/libthread.so.1
>   8 LWP 12  0xfef158f4 in __lwp_park () from /usr/lib/libthread.so.1
>   7 LWP 13  0xfef158f4 in __lwp_park () from /usr/lib/libthread.so.1
>   6 LWP 14  0xfef158f4 in __lwp_park () from /usr/lib/libthread.so.1
>   5 LWP 15  0xfef158f4 in __lwp_park () from /usr/lib/libthread.so.1
>   4 LWP 16  0xfef158f4 in __lwp_park () from /usr/lib/libthread.so.1
>   3 LWP 17  0xfef158f4 in __lwp_park () from /usr/lib/libthread.so.1
>   2 LWP 18  0xfef158f4 in __lwp_park () from /usr/lib/libthread.so.1
> * 1 LWP 1  0xff01f870 in _lwp_wait () from /usr/lib/libc.so.1
> (gdb) thread 18
> [Switching to thread 18 (LWP 2)]#0  0xff01fb74 in time ()
>    from /usr/lib/libc.so.1
> (gdb) bt
> #0  0xff01fb74 in time () from /usr/lib/libc.so.1
> #1  0x000e6d14 in ldap_pvt_runqueue_resched ()
> #2  0x0003b660 in slapd_daemon_destroy ()
> #3  0xfef157bc in _lwp_start () from /usr/lib/libthread.so.1
> (gdb) detach
> Detaching from program:  LWP 2
> (gdb) quit
>
>   


-- 
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  OpenLDAP Core Team            http://www.openldap.org/project/
Prev by Date: Re: (ITS#4306) bad ACL syntax in modification to olcAccess attribute crashes slapd
Next by Date: RE: (ITS#4308) poor performance under load
Index(es):
- Chronological
- Thread