[Date Prev][Date Next] [Chronological] [Thread] [Top]

Segmentation Fault on ldap_chain_op

Hi everyone,

I am currently looking into an issue with an environment that consists of 3 producers and 6 consumers all on OpenLdap 2.4.48. All the masters share an identical master config and all of the consumers share an identical consumer config. One consumer server in particular is crashing on a random time interval with no obvious reason. Each ldap server has sasl pass through auth setup and there is an additional slapd process on each server being used as a ldap meta to AD.

One server in particular, is crashing at seemingly random intervals. Upon initial inspection, we noticed that around the time of a crash, saslauthd logs would have an increase in failed write messages:

Dec 11 07:23:20 {REDACTED} saslauthd[2720]: user ldap_search_st() failed: Timed out
Dec 11 07:23:30 {REDACTED} saslauthd[2718]: user ldap_search_st() failed: Timed out
Dec 11 07:23:40 {REDACTED} saslauthd[2718]: user ldap_search_st() failed: Timed out
Dec 11 07:23:56 {REDACTED} saslauthd[2717]: user ldap_search_st() failed: Timed out
Dec 11 07:23:57 {REDACTED} saslauthd[2720]: user ldap_search_st() failed: Timed out
Dec 11 07:23:57 {REDACTED} saslauthd[2719]: user ldap_search_st() failed: Timed out
Dec 11 07:24:05 {REDACTED} saslauthd[2718]: user ldap_search_st() failed: Timed out
Dec 11 07:24:06 {REDACTED} saslauthd[2717]: user ldap_search_st() failed: Timed out
Dec 11 07:24:54 {REDACTED} saslauthd[2719]:                : write failure
Dec 11 07:24:54 {REDACTED} saslauthd[2717]:                : write failure
Dec 11 07:24:54 {REDACTED} saslauthd[2719]:                : write: Broken pipe
Dec 11 07:24:54 {REDACTED} saslauthd[2717]:                : write: Broken pipe
Dec 11 07:24:54 {REDACTED} saslauthd[2718]:                : write failure
Dec 11 07:24:54 {REDACTED} saslauthd[2720]:                : write failure
Dec 11 07:24:54 {REDACTED} saslauthd[2718]:                : write: Broken pipe
Dec 11 07:24:54 {REDACTED} saslauthd[2720]:                : write: Broken pipe
Dec 11 07:25:04 {REDACTED} saslauthd[2721]:                : write failure
Dec 11 07:25:04 {REDACTED} saslauthd[2721]:                : write: Broken pipe

Initially we thought this meant that the crashes were caused by performance issues with AD, saslauthd or the ldap meta instance. We also ran slapd in full debugging mode to see if the logs have some kind of hints to the source of the issue.

The resulting slapd logs were normal. To obtain further information we obtained the following trace of the segfault using gdb:

starting
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".

frames-invalid
[New Thread 0x7ffbf19be700 (LWP 22956)]

new-thread
[New Thread 0x7ffbf11bd700 (LWP 22957)]

new-thread
[New Thread 0x7ffbf09bc700 (LWP 22958)]

new-thread
[New Thread 0x7ffbebfff700 (LWP 22959)]

new-thread
[New Thread 0x7ffbeb7fe700 (LWP 22963)]

new-thread
[New Thread 0x7ffbe9cfb700 (LWP 22965)]

new-thread
[New Thread 0x7ffbd3fff700 (LWP 22966)]

new-thread

signal

Program received signal 
signal-name
SIGSEGV
signal-name-end
, 
signal-string
Segmentation fault
signal-string-end
.
[Switching to Thread 0x7ffbf09bc700 (LWP 22958)]

thread-changed

frame-begin 0 0x4e52e3

frame-function-name
ldap_chain_op
frame-args
 (
arg-begin
op=op@entry
arg-name-end
=
arg-value *
0x7ffbe41f0870
arg-end
, 
arg-begin
rs=rs@entry
arg-name-end
=
arg-value *
0x7ffbf09bbb20
arg-end
, 
arg-begin
op_f
arg-name-end
=
arg-value *
0x4a845e <ldap_back_search>
arg-end
, 
arg-begin
ref=ref@entry
arg-name-end
=
arg-value *
0x0
arg-end
, 
arg-begin
depth=depth@entry
arg-name-end
=
arg-value -
0
arg-end
)
frame-source-begin
 at 
frame-source-file
chain.c
frame-source-file-end
:
frame-source-line
422
frame-source-end

source ${REDACTED DIR}/chain.c:422:11629:beg:0x4e52e3

frame-end

stopped

post-prompt

#0  ldap_chain_op (op=op@entry=0x7ffbe41f0870, rs=rs@entry=0x7ffbf09bbb20, op_f=0x4a845e <ldap_back_search>, ref=ref@entry=0x0, depth=depth@entry=0) at chain.c:422
#1  0x00000000004e61da in ldap_chain_response (op=0x7ffbe41f0870, rs=0x7ffbf09bbb20) at chain.c:1090
#2  0x000000000049d323 in over_back_response (op=0x7ffbe41f0870, rs=0x7ffbf09bbb20) at backover.c:237
#3  0x0000000000447c1a in slap_response_play (op=op@entry=0x7ffbe41f0870, rs=rs@entry=0x7ffbf09bbb20) at result.c:508
#4  0x00000000004480f5 in send_ldap_response (op=op@entry=0x7ffbe41f0870, rs=rs@entry=0x7ffbf09bbb20) at result.c:583
#5  0x0000000000448d1f in slap_send_ldap_result (op=0x7ffbe41f0870, rs=0x7ffbf09bbb20) at result.c:861
#6  0x00000000004bd8c3 in mdb_search (op=0x7ffbe41f0870, rs=0x7ffbf09bbb20) at search.c:558
#7  0x000000000049dd58 in overlay_op_walk (op=op@entry=0x7ffbe41f0870, rs=0x7ffbf09bbb20, which=op_search, oi=0xa3eeb0, on=0x0) at backover.c:677
#8  0x000000000049de86 in over_op_func (op=0x7ffbe41f0870, rs=<optimized out>, which=which@entry=op_search) at backover.c:730
#9  0x000000000049dfab in over_op_search (op=<optimized out>, rs=<optimized out>) at backover.c:757
#10 0x00000000004d1a02 in relay_back_op (op=0x7ffbe41f0870, rs=0x7ffbf09bbb20, which=which@entry=2) at op.c:211
#11 0x00000000004d1b07 in relay_back_op_search (op=<optimized out>, rs=<optimized out>) at op.c:251
#12 0x000000000049dd58 in overlay_op_walk (op=op@entry=0x7ffbe41f0870, rs=0x7ffbf09bbb20, which=op_search, oi=0xa3af00, on=0x0) at backover.c:677
#13 0x000000000049de86 in over_op_func (op=0x7ffbe41f0870, rs=<optimized out>, which=which@entry=op_search) at backover.c:730
#14 0x000000000049dfab in over_op_search (op=<optimized out>, rs=<optimized out>) at backover.c:757
#15 0x000000000043c33f in fe_op_search (op=0x7ffbe41f0870, rs=0x7ffbf09bbb20) at search.c:402
#16 0x000000000049dd58 in overlay_op_walk (op=op@entry=0x7ffbe41f0870, rs=0x7ffbf09bbb20, which=op_search, oi=0x980e00, on=0x0) at backover.c:677
#17 0x000000000049de86 in over_op_func (op=0x7ffbe41f0870, rs=<optimized out>, which=which@entry=op_search) at backover.c:730
#18 0x000000000049dfab in over_op_search (op=<optimized out>, rs=<optimized out>) at backover.c:757
#19 0x000000000043bef6 in do_search (op=0x7ffbe41f0870, rs=0x7ffbf09bbb20) at search.c:247
#20 0x0000000000439bc4 in connection_operation (ctx=0x7ffbf09bbbd0, arg_v=0x7ffbe41f0870) at connection.c:1158
#21 0x0000000000563cbf in ldap_int_thread_pool_wrapper (xpool=0x8e36a0) at tpool.c:696
#22 0x00007ffff754cdd5 in start_thread (arg=0x7ffbf09bc700) at pthread_create.c:307
#23 0x00007ffff674d02d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
#0  __lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
#1  0x00007ffff66bd6ac in _L_lock_39 () from /lib64/libc.so.6
#2  0x00007ffff66bd5b4 in __GI__IO_fflush (fp=0x937d00) at iofflush.c:39
#3  0x000000000058bbc6 in ber_error_print (data=0x7ffbd3ffdc10 "  aaf9:  04 3a 75 69 64 3d 53 49  41 53 47 52 50 2c 6f 75   .:uid=SIASGRP,ou  \n") at bprint.c:87
#4  0x000000000058be41 in ber_bprint (data=0x7ffbcd83df04 "0\203\n\036\273\004\funiqueMember1\203\n\036\250\004\071uid=wtso732,ou={REDACTED}“…, len=len@entry=663561) at bprint.c:202
#5  0x000000000058c053 in ber_dump (ber=ber@entry=0x7ffbcc122e20, inout=inout@entry=1) at bprint.c:276
#6  0x000000000058c0ab in ber_log_dump (errlvl=errlvl@entry=16, loglvl=<optimized out>, ber=ber@entry=0x7ffbcc122e20, inout=inout@entry=1) at bprint.c:247
#7  0x00000000005894c1 in ber_scanf (ber=0x7ffbcc122e20, fmt=fmt@entry=0x5b09b3 "{mW}") at decode.c:723
#8  0x0000000000492ba8 in syncrepl_message_to_entry (syncUUID=0x7ffbd3ffdf40, syncstate=<optimized out>, entry=<synthetic pointer>, modlist=0x7ffbd3ffe320, msg=<optimized out>, op=0x7ffbd3ffe510, si=0xa3e9e0) at syncrepl.c:2658
#9  do_syncrep2 (op=op@entry=0x7ffbd3ffe510, si=si@entry=0xa3e9e0) at syncrepl.c:1043
#10 0x00000000004976db in do_syncrepl (ctx=<optimized out>, arg=0xa3e670) at syncrepl.c:1571
#11 0x0000000000563cbf in ldap_int_thread_pool_wrapper (xpool=0x8e36a0) at tpool.c:696
#12 0x00007ffff754cdd5 in start_thread (arg=0x7ffbd3fff700) at pthread_create.c:307
#13 0x00007ffff674d02d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
#0  0x00007ffff673e16d in write () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007ffff66c91a3 in _IO_new_file_write (f=0x937c20, data=0x7ffff7ff8000, n=162) at fileops.c:1258
#2  0x00007ffff66ca9be in new_do_write (to_do=162, data=0x7ffff7ff8000 "  4530:  62 61 6c 2c 64 63 3d 61  65 70 2c 64 63 3d 63 6f   bal,dc={REDACTED} \n5e0a2340 >>> dnPretty: <uid={REDACTED}>\n..  \n5e0a2340 daemon: epoll: listen=19"..., fp=0x937c20) at fileops.c:519
#3  _IO_new_do_write (fp=0x937c20, data=0x7ffff7ff8000 "  4530:  62 61 6c 2c 64 63 3d 61  65 70 2c 64 63 3d 63 6f   bal,dc={REDACTED}  \n5e0a2340 >>> dnPretty: <uid={REDACTED}>\n..  \n5e0a2340 daemon: epoll: listen=19"..., to_do=162) at fileops.c:495
#4  0x00007ffff66c87f8 in _IO_new_file_sync (fp=0x937c20) at fileops.c:869
#5  0x00007ffff66bd5f3 in __GI__IO_fflush (fp=0x937c20) at iofflush.c:40
#6  0x000000000058c21a in lutil_debug (debug=<optimized out>, level=level@entry=1, fmt=fmt@entry=0x59c7a6 ">>> dnPretty: <%s>\n") at debug.c:81
#7  0x000000000044c792 in dnPretty (syntax=syntax@entry=0x8c4810, val=val@entry=0x7ffbe9cf9d00, out=out@entry=0x7ffbe9cf9de0, ctx=ctx@entry=0x0) at dn.c:539
#8  0x000000000046c614 in nameUIDPretty (syntax=0x8c4810, val=0x7ffbd9845600, out=0x7ffbe9cf9de0, ctx=0x0) at schema_init.c:1350
#9  0x000000000045383d in ordered_value_pretty (ad=ad@entry=0xa40470, val=0x7ffbd9845600, out=out@entry=0x7ffbe9cf9de0, ctx=ctx@entry=0x0) at value.c:511
#10 0x000000000044f676 in slap_mods_check (op=op@entry=0x7ffbe9cfa510, ml=0x7ffbd992c410, text=text@entry=0x7ffbe9cf9f60, textbuf=textbuf@entry=0x7ffbe9cfa180 "H\034\032\330\373\177", textlen=textlen@entry=256, ctx=ctx@entry=0x0) at modify.c:575
#11 0x0000000000492e10 in syncrepl_message_to_entry (syncUUID=0x7ffbe9cf9f40, syncstate=<optimized out>, entry=<synthetic pointer>, modlist=0x7ffbe9cfa320, msg=<optimized out>, op=0x7ffbe9cfa510, si=0xa3fb20) at syncrepl.c:2715
#12 do_syncrep2 (op=op@entry=0x7ffbe9cfa510, si=si@entry=0xa3fb20) at syncrepl.c:1043
#13 0x00000000004976db in do_syncrepl (ctx=<optimized out>, arg=0xa40060) at syncrepl.c:1571
#14 0x0000000000563cbf in ldap_int_thread_pool_wrapper (xpool=0x8e36a0) at tpool.c:696
#15 0x00007ffff754cdd5 in start_thread (arg=0x7ffbe9cfb700) at pthread_create.c:307
#16 0x00007ffff674d02d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
#0  __lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
#1  0x00007ffff66bd6ac in _L_lock_39 () from /lib64/libc.so.6
#2  0x00007ffff66bd5b4 in __GI__IO_fflush (fp=0x937d00) at iofflush.c:39
#3  0x000000000058bbc6 in ber_error_print (data=0x7ffbeb7fcc10 "  4530:  62 61 6c 2c 64 63 3d 61  65 70 2c 64 63 3d 63 6f   bal,dc={REDACTED}  \n") at bprint.c:87
#4  0x000000000058be41 in ber_bprint (data=0x7ffbd576fee7 "0\027\004\002cn1\021\004\017CSC User Access0-\004\025structuralObjectClass1\024\004\022groupOfUniqueNames03\004\tentryUUID1&\004$13257271-b706-49bc-a2a9-a5f3e7f4a0dd0L\004\fcreatorsName1<\004:cn={REDACTED}”…, len=len@entry=61665) at bprint.c:202
#5  0x000000000058c053 in ber_dump (ber=ber@entry=0x7ffbd4319130, inout=inout@entry=1) at bprint.c:276
#6  0x000000000058c0ab in ber_log_dump (errlvl=errlvl@entry=16, loglvl=<optimized out>, ber=ber@entry=0x7ffbd4319130, inout=inout@entry=1) at bprint.c:247
#7  0x00000000005894c1 in ber_scanf (ber=0x7ffbd4319130, fmt=fmt@entry=0x5b09b3 "{mW}") at decode.c:723
#8  0x0000000000492ba8 in syncrepl_message_to_entry (syncUUID=0x7ffbeb7fcf40, syncstate=<optimized out>, entry=<synthetic pointer>, modlist=0x7ffbeb7fd320, msg=<optimized out>, op=0x7ffbeb7fd510, si=0xa3f3b0) at syncrepl.c:2658
#9  do_syncrep2 (op=op@entry=0x7ffbeb7fd510, si=si@entry=0xa3f3b0) at syncrepl.c:1043
#10 0x00000000004976db in do_syncrepl (ctx=<optimized out>, arg=0xa3f8b0) at syncrepl.c:1571
#11 0x0000000000563cbf in ldap_int_thread_pool_wrapper (xpool=0x8e36a0) at tpool.c:696
#12 0x00007ffff754cdd5 in start_thread (arg=0x7ffbeb7fe700) at pthread_create.c:307
#13 0x00007ffff674d02d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
#0  __lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
#1  0x00007ffff66bd6ac in _L_lock_39 () from /lib64/libc.so.6
#2  0x00007ffff66bd5b4 in __GI__IO_fflush (fp=0x937d00) at iofflush.c:39
#3  0x000000000058bbc6 in ber_error_print (data=0x7ffbebe6cf20 "  b6c6:  03 0a 01 04 30 22 02 03  02 38 56 17 0d 31 38 30   ....0\"...8V..180  \n") at bprint.c:87
#4  0x000000000058be41 in ber_bprint (data=0x7ffbe04282a0 "0\203UK\275\002\001\001d\203UK\265\004\062cn={REDACTED}0\203UK|0\203UKw\004 certificateRevocationList;binary1\203UKP\004\203UKK0\203UKF0\203UJ-\002\001\001\060\r\006\t*\206H\206\367\r\001\001\v\005", len=len@entry=5589954) at bprint.c:202
#5  0x000000000058bfb3 in ber_log_bprint (errlvl=errlvl@entry=16, loglvl=<optimized out>, data=<optimized out>, len=len@entry=5589954) at bprint.c:175
#6  0x000000000058b179 in ber_flush2 (sb=0x7ffbe17f55b0, ber=ber@entry=0x7ffbebe6d0f0, freeit=freeit@entry=0) at io.c:240
#7  0x0000000000447ee5 in send_ldap_ber (op=op@entry=0x7ffbe41bd940, ber=ber@entry=0x7ffbebe6d0f0) at result.c:339
#8  0x000000000044b290 in slap_send_search_entry (op=0x7ffbe41bd940, rs=<optimized out>) at result.c:1430
#9  0x00000000004bf3b4 in mdb_search (op=<optimized out>, rs=0x7ffbebffeaa0) at search.c:1085
#10 0x000000000049dd58 in overlay_op_walk (op=op@entry=0x7ffbe41bd940, rs=0x7ffbebffeaa0, which=op_search, oi=0xa3eeb0, on=0x0) at backover.c:677
#11 0x000000000049de86 in over_op_func (op=0x7ffbe41bd940, rs=<optimized out>, which=which@entry=op_search) at backover.c:730
#12 0x000000000049dfab in over_op_search (op=<optimized out>, rs=<optimized out>) at backover.c:757
#13 0x000000000043c33f in fe_op_search (op=0x7ffbe41bd940, rs=0x7ffbebffeaa0) at search.c:402
#14 0x000000000049dd58 in overlay_op_walk (op=op@entry=0x7ffbe41bd940, rs=0x7ffbebffeaa0, which=op_search, oi=0x980e00, on=0x0) at backover.c:677
#15 0x000000000049de86 in over_op_func (op=0x7ffbe41bd940, rs=<optimized out>, which=which@entry=op_search) at backover.c:730
#16 0x000000000049dfab in over_op_search (op=<optimized out>, rs=<optimized out>) at backover.c:757
#17 0x000000000043bef6 in do_search (op=0x7ffbe41bd940, rs=0x7ffbebffeaa0) at search.c:247
#18 0x0000000000439bc4 in connection_operation (ctx=ctx@entry=0x7ffbebffebd0, arg_v=arg_v@entry=0x7ffbe41bd940) at connection.c:1158
#19 0x000000000043a9cb in connection_read_thread (ctx=0x7ffbebffebd0, argv=<optimized out>) at connection.c:1294
#20 0x0000000000563cbf in ldap_int_thread_pool_wrapper (xpool=0x8e36a0) at tpool.c:696
#21 0x00007ffff754cdd5 in start_thread (arg=0x7ffbebfff700) at pthread_create.c:307
#22 0x00007ffff674d02d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
#0  ldap_chain_op (op=op@entry=0x7ffbe41f0870, rs=rs@entry=0x7ffbf09bbb20, op_f=0x4a845e <ldap_back_search>, ref=ref@entry=0x0, depth=depth@entry=0) at chain.c:422
#1  0x00000000004e61da in ldap_chain_response (op=0x7ffbe41f0870, rs=0x7ffbf09bbb20) at chain.c:1090
#2  0x000000000049d323 in over_back_response (op=0x7ffbe41f0870, rs=0x7ffbf09bbb20) at backover.c:237
#3  0x0000000000447c1a in slap_response_play (op=op@entry=0x7ffbe41f0870, rs=rs@entry=0x7ffbf09bbb20) at result.c:508
#4  0x00000000004480f5 in send_ldap_response (op=op@entry=0x7ffbe41f0870, rs=rs@entry=0x7ffbf09bbb20) at result.c:583
#5  0x0000000000448d1f in slap_send_ldap_result (op=0x7ffbe41f0870, rs=0x7ffbf09bbb20) at result.c:861
#6  0x00000000004bd8c3 in mdb_search (op=0x7ffbe41f0870, rs=0x7ffbf09bbb20) at search.c:558
#7  0x000000000049dd58 in overlay_op_walk (op=op@entry=0x7ffbe41f0870, rs=0x7ffbf09bbb20, which=op_search, oi=0xa3eeb0, on=0x0) at backover.c:677
#8  0x000000000049de86 in over_op_func (op=0x7ffbe41f0870, rs=<optimized out>, which=which@entry=op_search) at backover.c:730
#9  0x000000000049dfab in over_op_search (op=<optimized out>, rs=<optimized out>) at backover.c:757
#10 0x00000000004d1a02 in relay_back_op (op=0x7ffbe41f0870, rs=0x7ffbf09bbb20, which=which@entry=2) at op.c:211
#11 0x00000000004d1b07 in relay_back_op_search (op=<optimized out>, rs=<optimized out>) at op.c:251
#12 0x000000000049dd58 in overlay_op_walk (op=op@entry=0x7ffbe41f0870, rs=0x7ffbf09bbb20, which=op_search, oi=0xa3af00, on=0x0) at backover.c:677
#13 0x000000000049de86 in over_op_func (op=0x7ffbe41f0870, rs=<optimized out>, which=which@entry=op_search) at backover.c:730
#14 0x000000000049dfab in over_op_search (op=<optimized out>, rs=<optimized out>) at backover.c:757
#15 0x000000000043c33f in fe_op_search (op=0x7ffbe41f0870, rs=0x7ffbf09bbb20) at search.c:402
#16 0x000000000049dd58 in overlay_op_walk (op=op@entry=0x7ffbe41f0870, rs=0x7ffbf09bbb20, which=op_search, oi=0x980e00, on=0x0) at backover.c:677
#17 0x000000000049de86 in over_op_func (op=0x7ffbe41f0870, rs=<optimized out>, which=which@entry=op_search) at backover.c:730
#18 0x000000000049dfab in over_op_search (op=<optimized out>, rs=<optimized out>) at backover.c:757
#19 0x000000000043bef6 in do_search (op=0x7ffbe41f0870, rs=0x7ffbf09bbb20) at search.c:247
#20 0x0000000000439bc4 in connection_operation (ctx=0x7ffbf09bbbd0, arg_v=0x7ffbe41f0870) at connection.c:1158
#21 0x0000000000563cbf in ldap_int_thread_pool_wrapper (xpool=0x8e36a0) at tpool.c:696
#22 0x00007ffff754cdd5 in start_thread (arg=0x7ffbf09bc700) at pthread_create.c:307
#23 0x00007ffff674d02d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
#0  __lll_lock_wait_private () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
#1  0x00007ffff66bd6ac in _L_lock_39 () from /lib64/libc.so.6
#2  0x00007ffff66bd5b4 in __GI__IO_fflush (fp=0x937d00) at iofflush.c:39
#3  0x000000000058bbc6 in ber_error_print (data=0x7ffbf102af20 "  e54c:  33 5a 30 0c 30 0a 06 03  55 1d 15 04 03 0a 01 04   3Z0.0...U.......  \n") at bprint.c:87
#4  0x000000000058be41 in ber_bprint (data=0x7ffbe42cc110 "0\203UK\275\002\001\021d\203UK\265\004\062cn={REDACTED}0\203UK|0\203UKw\004 certificateRevocationList;binary1\203UKP\004\203UKK0\203UKF0\203UJ-\002\001\001\060\r\006\t*\206H\206\367\r\001\001\v\005", len=len@entry=5589954) at bprint.c:202
#5  0x000000000058bfb3 in ber_log_bprint (errlvl=errlvl@entry=16, loglvl=<optimized out>, data=<optimized out>, len=len@entry=5589954) at bprint.c:175
#6  0x000000000058b179 in ber_flush2 (sb=0x7ffbe422daa0, ber=ber@entry=0x7ffbf102b0f0, freeit=freeit@entry=0) at io.c:240
#7  0x0000000000447ee5 in send_ldap_ber (op=op@entry=0x7ffbe42197b0, ber=ber@entry=0x7ffbf102b0f0) at result.c:339
#8  0x000000000044b290 in slap_send_search_entry (op=0x7ffbe42197b0, rs=<optimized out>) at result.c:1430
#9  0x00000000004bf3b4 in mdb_search (op=<optimized out>, rs=0x7ffbf11bcaa0) at search.c:1085
#10 0x000000000049dd58 in overlay_op_walk (op=op@entry=0x7ffbe42197b0, rs=0x7ffbf11bcaa0, which=op_search, oi=0xa3eeb0, on=0x0) at backover.c:677
#11 0x000000000049de86 in over_op_func (op=0x7ffbe42197b0, rs=<optimized out>, which=which@entry=op_search) at backover.c:730
#12 0x000000000049dfab in over_op_search (op=<optimized out>, rs=<optimized out>) at backover.c:757
#13 0x000000000043c33f in fe_op_search (op=0x7ffbe42197b0, rs=0x7ffbf11bcaa0) at search.c:402
#14 0x000000000049dd58 in overlay_op_walk (op=op@entry=0x7ffbe42197b0, rs=0x7ffbf11bcaa0, which=op_search, oi=0x980e00, on=0x0) at backover.c:677
#15 0x000000000049de86 in over_op_func (op=0x7ffbe42197b0, rs=<optimized out>, which=which@entry=op_search) at backover.c:730
#16 0x000000000049dfab in over_op_search (op=<optimized out>, rs=<optimized out>) at backover.c:757
#17 0x000000000043bef6 in do_search (op=0x7ffbe42197b0, rs=0x7ffbf11bcaa0) at search.c:247
#18 0x0000000000439bc4 in connection_operation (ctx=ctx@entry=0x7ffbf11bcbd0, arg_v=arg_v@entry=0x7ffbe42197b0) at connection.c:1158
#19 0x000000000043a9cb in connection_read_thread (ctx=0x7ffbf11bcbd0, argv=<optimized out>) at connection.c:1294
#20 0x0000000000563cbf in ldap_int_thread_pool_wrapper (xpool=0x8e36a0) at tpool.c:696
#21 0x00007ffff754cdd5 in start_thread (arg=0x7ffbf11bd700) at pthread_create.c:307
#22 0x00007ffff674d02d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
#0  0x00007ffff674d603 in epoll_wait () at ../sysdeps/unix/syscall-template.S:81
#1  0x000000000043618b in slapd_daemon_task (ptr=<optimized out>) at daemon.c:2539
#2  0x00007ffff754cdd5 in start_thread (arg=0x7ffbf19be700) at pthread_create.c:307
#3  0x00007ffff674d02d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
#0  0x00007ffff754df47 in pthread_join (threadid=140720067045120, thread_return=thread_return@entry=0x0) at pthread_join.c:90
#1  0x00000000005642b8 in ldap_pvt_thread_join (thread=<optimized out>, thread_return=thread_return@entry=0x0) at thr_posix.c:197
#2  0x0000000000437766 in slapd_daemon () at daemon.c:2932
#3  0x0000000000420b84 in main (argc=<optimized out>, argv=0x7fffffffe408) at main.c:1017

The above trace seems to suggest the source of the segfault is coming from the ldap_chain_op function at chain.c:422. We use chain forwarding for ppolicy operations on the consumers.

This is the line:

for ( ; !BER_BVISNULL( ref ); ref++ ) {

BER_BVISNULL is not a function but a macro which is defined (lber_pvt.h:216) as follows:

#define BER_BVISNULL(bv)	((bv)->bv_val == NULL)

The above seems suggests that ref is set to null at the time of the check. 

When I double checked my chain configs I noticed there was an entry that did not contain the olcDbURI attribute that I did not create. It seems that this particular entry was created by the cn=config conversion for the following slapd.conf configuration:

Overlay				chain

chain-proxy-whoami	yes

chain-uri				{REDACTED URI 1}
Chain-cache-uri		FALSE
Chain-max-depth		1
Chain-return-error		TRUE
Chain-idassert-bind	bindmethod=“simple”
					binddn={REDACTED}
					credentials={REDACTED}
Chain-protocol-version	3

chain-uri				{REDACTED URI 2}
Chain-cache-uri		FALSE
Chain-max-depth		1
Chain-return-error		TRUE
Chain-idassert-bind	bindmethod=“simple”
					binddn={REDACTED}
					credentials={REDACTED}
Chain-protocol-version	3

chain-uri				{REDACTED URI 3}
Chain-cache-uri		FALSE
Chain-max-depth		1
Chain-return-error		TRUE
Chain-idassert-bind	bindmethod=“simple”
					binddn={REDACTED}
					credentials={REDACTED}
Chain-protocol-version	3

After the conversion to cn=config:

dn: olcOverlay={1}chain,olcDatabase={-1}frontend,cn=config
objectClass: olcOverlayConfig
objectClass: olcChainConfig
olcOverlay: {1}chain
olcChainCacheURI: FALSE
olcChainMaxReferralDepth: 1
olcChainReturnError: TRUE

dn: olcDatabase={0}ldap,olcOverlay={1}chain,olcDatabase={-1}frontend,cn=config
objectClass: olcLDAPConfig
objectClass: olcChainDatabase
olcDatabase: {0}ldap
olcDbStartTLS: ldaps  starttls=no 
olcDbRebindAsUser: FALSE
olcDbChaseReferrals: TRUE
olcDbTFSupport: no
olcDbProxyWhoAmI: TRUE
olcDbProtocolVersion: 3
olcDbSingleConn: FALSE
olcDbCancel: abandon
olcDbUseTemporaryConn: FALSE
olcDbConnectionPoolMax: 16
olcDbSessionTrackingRequest: FALSE
olcDbNoRefs: FALSE
olcDbNoUndefFilter: FALSE
olcDbOnErr: continue
olcDbKeepalive: 0:0:0

dn: olcDatabase={1}ldap,olcOverlay={1}chain,olcDatabase={-1}frontend,cn=config
objectClass: olcLDAPConfig
objectClass: olcChainDatabase
olcDatabase: {1}ldap
olcDbURI: “{REDACTED URI 1}“
olcDbStartTLS: ldaps  starttls=no tls_cacert=“/{REDACTED}/openldap/tls/
 cacerts.cer" tls_reqcert=demand tls_crlcheck=none
olcDbIDAssertBind: mode=self flags=prescriptive,proxy-authz-non-critical bindm
 ethod=simple timeout=0 network-timeout=0 binddn={REDACTED} credentials={REDACTED} keepalive=0:0:0
olcDbRebindAsUser: FALSE
olcDbChaseReferrals: TRUE
olcDbTFSupport: no
olcDbProxyWhoAmI: TRUE
olcDbProtocolVersion: 3
olcDbSingleConn: FALSE
olcDbCancel: abandon
olcDbUseTemporaryConn: FALSE
olcDbConnectionPoolMax: 16
olcDbSessionTrackingRequest: FALSE
olcDbNoRefs: FALSE
olcDbNoUndefFilter: FALSE
olcDbOnErr: continue
olcDbKeepalive: 0:0:0

dn: olcDatabase={2}ldap,olcOverlay={1}chain,olcDatabase={-1}frontend,cn=config
objectClass: olcLDAPConfig
objectClass: olcChainDatabase
olcDatabase: {2}ldap
olcDbURI: “{REDACTED URI 2}“
olcDbStartTLS: ldaps  starttls=no tls_cacert=“/{REDACTED}/openldap/tls/
 cacerts.cer" tls_reqcert=demand tls_crlcheck=none
olcDbIDAssertBind: mode=self flags=prescriptive,proxy-authz-non-critical bindm
 ethod=simple timeout=0 network-timeout=0 binddn={REDACTED} credentials={REDACTED} keepalive=0:0:0
olcDbRebindAsUser: FALSE
olcDbChaseReferrals: TRUE
olcDbTFSupport: no
olcDbProxyWhoAmI: TRUE
olcDbProtocolVersion: 3
olcDbSingleConn: FALSE
olcDbCancel: abandon
olcDbUseTemporaryConn: FALSE
olcDbConnectionPoolMax: 16
olcDbSessionTrackingRequest: FALSE
olcDbNoRefs: FALSE
olcDbNoUndefFilter: FALSE
olcDbOnErr: continue
olcDbKeepalive: 0:0:0

dn: olcDatabase={3}ldap,olcOverlay={1}chain,olcDatabase={-1}frontend,cn=config
objectClass: olcLDAPConfig
objectClass: olcChainDatabase
olcDatabase: {3}ldap
olcDbURI: “{REDACTED URI 3}“
olcDbStartTLS: ldaps  starttls=no tls_cacert=“/{REDACTED}/openldap/tls/
 cacerts.cer" tls_reqcert=demand tls_crlcheck=none
olcDbIDAssertBind: mode=self flags=prescriptive,proxy-authz-non-critical bindm
 ethod=simple timeout=0 network-timeout=0 binddn={REDACTED} credentials={REDACTED} keepalive=0:0:0
olcDbRebindAsUser: FALSE
olcDbChaseReferrals: TRUE
olcDbTFSupport: no
olcDbProxyWhoAmI: TRUE
olcDbProtocolVersion: 3
olcDbSingleConn: FALSE
olcDbCancel: abandon
olcDbUseTemporaryConn: FALSE
olcDbConnectionPoolMax: 16
olcDbSessionTrackingRequest: FALSE
olcDbNoRefs: FALSE
olcDbNoUndefFilter: FALSE
olcDbOnErr: continue
olcDbKeepalive: 0:0:0

I am wondering if the following is causing the segfaults:

When setting up the configs during the slapd.d conversion, the slapd.conf -> cn=config converter added an extra unnecessary entry for the chain config (there are 4 ldap proxy entries instead of 3). This particular entry that was added does not have an olcDbURI value. Since the referral string is set by the olcDbURI attribute value the missing value results in NULL being set for ref. 

However, I have not yet been able to manually trigger the segfault by invoking ppolicy.

I also found a red hat ticket that seemd to detail the same issue (see https://bugzilla.redhat.com/show_bug.cgi?id=1644654).

Any ideas on this? I can also obtain a trace of all the threads if needed.

Jeremy


-- 
Jeremy Diaz 
Rex Consulting, Inc 
5652 Florence Terrace, Oakland, CA 94611 
email: jeremy.diaz@rexconsulting.net 
web: [ http://www.rexconsulting.net/ | http://www.rexconsulting.net ] 
phone, toll-free: +1 (888) 403-8996 EXT: 5 


The information transmitted is intended only for the person or entity to 
which it is addressed and may contain confidential and/or privileged 
material. Any review, retransmission, dissemination or other use of, 
or taking of any action in reliance upon, this information by persons 
or entities other than the intended recipient is prohibited. 
Rex Consulting, Inc. has been a California Corporation since 2001.