[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: slapd crashes with ch_realloc of X bytes failed



Hello,

 I could update both systems during my vacation to 2.4.33.
Both servers had 16GByte RAM.
The system crashed  again randomly (as expected).
So we increased the memory on one server to 24 GByte RAM. No effect,
this server crashes too, sometime till 5 times a day.

I installed a test machine (VM),  but only with 7GByte RAM and could
reproduce the crashes with one/two parallel searches.
So I tried tcmalloc, but with the same result.

I digged a little deeper and the results are, that not the physical
memory from the system is insufficient.

First - slapd tries to allocate memory more then available, but not
use it. It is enough physical memory available during slapd crashes.


I ran the commands continuously with while and sleep 1:
 free -m, and pmap $PID_OF_LDAP
and
vmstat 1

The last memory information before the crash was:

pmap:
--------------------------
21954: slapd
START               SIZE     RSS     PSS   DIRTY    SWAP PERM MAPPING
00007f63f4904000      4K      0K      0K      0K      0K rw-p [anon]
00007f63f4905000      4K      0K      0K      0K      0K ---p [anon]
00007f63f4906000  16384K   2080K   2080K   2080K      0K rw-p [anon]
00007f63f5906000      4K      0K      0K      0K      0K ---p [anon]
00007f63f5907000  16384K   4120K   4120K   4120K      0K rw-p [anon]
00007f63f6907000      4K      0K      0K      0K      0K ---p [anon]
00007f63f6908000  16384K   4120K   4120K   4120K      0K rw-p [anon]
00007f63f7908000      4K      0K      0K      0K      0K ---p [anon]
00007f63f7909000  16384K   4104K   4104K   4104K     16K rw-p [anon]
00007f63f8909000      4K      0K      0K      0K      0K ---p [anon]
00007f63f890a000  16384K      8K      8K      8K      4K rw-p [anon]
00007f63f990a000     24K      8K      8K      8K      0K rw-s /SYSV0000000c
00007f63f9910000    544K     44K     44K     44K      0K rw-s /SYSV0000000b
00007f63f9998000   2304K     12K     12K     12K      0K rw-s /SYSV0000000a
00007f63f9bd8000 2097152K 2097052K 2097052K 2097052K      0K rw-s /SYSV00000009
00007f6479bd8000  78008K  60508K  60508K  60508K      0K rw-s /SYSV00000008
00007f647e806000     16K      0K      0K      0K      0K r-xp
/usr/lib64/sasl2/libplain.so.2.0.22
00007f647e80a000   2044K      0K      0K      0K      0K ---p
/usr/lib64/sasl2/libplain.so.2.0.22
00007f647ea09000      4K      0K      0K      0K      4K r--p
/usr/lib64/sasl2/libplain.so.2.0.22
00007f647ea0a000      4K      0K      0K      0K      4K rw-p
/usr/lib64/sasl2/libplain.so.2.0.22
00007f647ea0b000     16K      0K      0K      0K      0K r-xp
/usr/lib64/sasl2/libanonymous.so.2.0.22
00007f647ea0f000   2044K      0K      0K      0K      0K ---p
/usr/lib64/sasl2/libanonymous.so.2.0.22
00007f647ec0e000      4K      0K      0K      0K      4K r--p
/usr/lib64/sasl2/libanonymous.so.2.0.22
00007f647ec0f000      4K      0K      0K      0K      4K rw-p
/usr/lib64/sasl2/libanonymous.so.2.0.22
00007f647ec10000     20K      0K      0K      0K      0K r-xp
/usr/lib64/sasl2/libsasldb.so.2.0.22
00007f647ec15000   2044K      0K      0K      0K      0K ---p
/usr/lib64/sasl2/libsasldb.so.2.0.22
00007f647ee14000      4K      0K      0K      0K      4K r--p
/usr/lib64/sasl2/libsasldb.so.2.0.22
00007f647ee15000      4K      0K      0K      0K      4K rw-p
/usr/lib64/sasl2/libsasldb.so.2.0.22
00007f647ee16000     16K      0K      0K      0K      0K r-xp
/usr/lib64/sasl2/liblogin.so.2.0.22
00007f647ee1a000   2044K      0K      0K      0K      0K ---p
/usr/lib64/sasl2/liblogin.so.2.0.22
00007f647f019000      4K      0K      0K      0K      4K r--p
/usr/lib64/sasl2/liblogin.so.2.0.22
00007f647f01a000      4K      0K      0K      0K      4K rw-p
/usr/lib64/sasl2/liblogin.so.2.0.22
00007f647f01b000      8K      0K      0K      0K      0K r-xp
/lib64/libkeyutils-1.2.so
00007f647f01d000   2044K      0K      0K      0K      0K ---p
/lib64/libkeyutils-1.2.so
00007f647f21c000      4K      0K      0K      0K      4K r--p
/lib64/libkeyutils-1.2.so
00007f647f21d000      4K      0K      0K      0K      4K rw-p
/lib64/libkeyutils-1.2.so
00007f647f21e000     28K      0K      0K      0K      0K r-xp
/usr/lib64/libkrb5support.so.0.1
00007f647f225000   2048K      0K      0K      0K      0K ---p
/usr/lib64/libkrb5support.so.0.1
00007f647f425000      4K      0K      0K      0K      4K r--p
/usr/lib64/libkrb5support.so.0.1
00007f647f426000      4K      0K      0K      0K      4K rw-p
/usr/lib64/libkrb5support.so.0.1
00007f647f427000     12K      0K      0K      0K      0K r-xp
/lib64/libcom_err.so.2.1
00007f647f42a000   2044K      0K      0K      0K      0K ---p
/lib64/libcom_err.so.2.1
00007f647f629000      4K      0K      0K      0K      4K r--p
/lib64/libcom_err.so.2.1
00007f647f62a000      4K      0K      0K      0K      4K rw-p
/lib64/libcom_err.so.2.1
00007f647f62b000    144K      0K      0K      0K      0K r-xp
/usr/lib64/libk5crypto.so.3.1
00007f647f64f000   2044K      0K      0K      0K      0K ---p
/usr/lib64/libk5crypto.so.3.1
00007f647f84e000      8K      0K      0K      0K      8K r--p
/usr/lib64/libk5crypto.so.3.1
00007f647f850000      4K      0K      0K      0K      4K rw-p
/usr/lib64/libk5crypto.so.3.1
00007f647f851000    632K      0K      0K      0K      0K r-xp
/usr/lib64/libkrb5.so.3.3
00007f647f8ef000   2044K      0K      0K      0K      0K ---p
/usr/lib64/libkrb5.so.3.3
00007f647faee000     12K      0K      0K      0K     12K r--p
/usr/lib64/libkrb5.so.3.3
00007f647faf1000      8K      0K      0K      0K      8K rw-p
/usr/lib64/libkrb5.so.3.3
00007f647faf3000    172K      0K      0K      0K      0K r-xp
/usr/lib64/libgssapi_krb5.so.2.2
00007f647fb1e000   2048K      0K      0K      0K      0K ---p
/usr/lib64/libgssapi_krb5.so.2.2
00007f647fd1e000      4K      0K      0K      0K      4K r--p
/usr/lib64/libgssapi_krb5.so.2.2
00007f647fd1f000      4K      0K      0K      0K      4K rw-p
/usr/lib64/libgssapi_krb5.so.2.2
00007f647fd20000     28K      0K      0K      0K      0K r-xp
/usr/lib64/sasl2/libgssapiv2.so.2.0.22
00007f647fd27000   2044K      0K      0K      0K      0K ---p
/usr/lib64/sasl2/libgssapiv2.so.2.0.22
00007f647ff26000      4K      0K      0K      0K      4K r--p
/usr/lib64/sasl2/libgssapiv2.so.2.0.22
00007f647ff27000      4K      0K      0K      0K      4K rw-p
/usr/lib64/sasl2/libgssapiv2.so.2.0.22
00007f647ff28000     48K      0K      0K      0K      0K r-xp
/usr/lib64/sasl2/libdigestmd5.so.2.0.22
00007f647ff34000   2044K      0K      0K      0K      0K ---p
/usr/lib64/sasl2/libdigestmd5.so.2.0.22
00007f6480133000      4K      0K      0K      0K      4K r--p
/usr/lib64/sasl2/libdigestmd5.so.2.0.22
00007f6480134000      4K      0K      0K      0K      4K rw-p
/usr/lib64/sasl2/libdigestmd5.so.2.0.22
00007f6480135000     84K      0K      0K      0K      0K r-xp
/lib64/libz.so.1.2.3
00007f648014a000   2044K      0K      0K      0K      0K ---p
/lib64/libz.so.1.2.3
00007f6480349000      4K      0K      0K      0K      4K r--p
/lib64/libz.so.1.2.3
00007f648034a000      4K      0K      0K      0K      4K rw-p
/lib64/libz.so.1.2.3
00007f648034b000     84K      0K      0K      0K      0K r-xp
/lib64/libgcc_s.so.1
00007f6480360000   2044K      0K      0K      0K      0K ---p
/lib64/libgcc_s.so.1
00007f648055f000      4K      0K      0K      0K      4K r--p
/lib64/libgcc_s.so.1
00007f6480560000      4K      0K      0K      0K      4K rw-p
/lib64/libgcc_s.so.1
00007f6480561000    940K      0K      0K      0K      0K r-xp
/usr/lib64/libstdc++.so.6.0.16
00007f648064c000   2048K      0K      0K      0K      0K ---p
/usr/lib64/libstdc++.so.6.0.16
00007f648084c000     32K      0K      0K      0K     32K r--p
/usr/lib64/libstdc++.so.6.0.16
00007f6480854000      8K      0K      0K      0K      8K rw-p
/usr/lib64/libstdc++.so.6.0.16
00007f6480856000     84K      0K      0K      0K      4K rw-p [anon]
00007f648086b000     36K     28K     28K      0K      0K r-xp
/lib64/libunwind.so.7.0.0
00007f6480874000   2044K      0K      0K      0K      0K ---p
/lib64/libunwind.so.7.0.0
00007f6480a73000      4K      4K      4K      4K      0K r--p
/lib64/libunwind.so.7.0.0
00007f6480a74000      4K      4K      4K      4K      0K rw-p
/lib64/libunwind.so.7.0.0
00007f6480a75000     16K      4K      4K      4K      0K rw-p [anon]
00007f6480a79000   1460K    444K     38K      0K      0K r-xp
/lib64/libc-2.11.3.so
00007f6480be6000   2048K      0K      0K      0K      0K ---p
/lib64/libc-2.11.3.so
00007f6480de6000     16K     16K     16K      8K      0K r--p
/lib64/libc-2.11.3.so
00007f6480dea000      4K      4K      4K      4K      0K rw-p
/lib64/libc-2.11.3.so
00007f6480deb000     20K     16K     16K     16K      4K rw-p [anon]
00007f6480df0000     36K     28K     10K      0K      0K r-xp
/lib64/libwrap.so.0.7.6
00007f6480df9000   2044K      0K      0K      0K      0K ---p
/lib64/libwrap.so.0.7.6
00007f6480ff8000      4K      4K      4K      0K      0K r--p
/lib64/libwrap.so.0.7.6
00007f6480ff9000      4K      4K      4K      4K      0K rw-p
/lib64/libwrap.so.0.7.6
00007f6480ffa000      4K      4K      4K      4K      0K rw-p [anon]
00007f6480ffb000     76K      4K      0K      0K      0K r-xp
/lib64/libresolv-2.11.3.so
00007f648100e000   2048K      0K      0K      0K      0K ---p
/lib64/libresolv-2.11.3.so
00007f648120e000      4K      0K      0K      0K      4K r--p
/lib64/libresolv-2.11.3.so
00007f648120f000      4K      0K      0K      0K      4K rw-p
/lib64/libresolv-2.11.3.so
00007f6481210000      8K      0K      0K      0K      0K rw-p [anon]
00007f6481212000     48K      4K      0K      0K      0K r-xp
/lib64/libcrypt-2.11.3.so
00007f648121e000   2044K      0K      0K      0K      0K ---p
/lib64/libcrypt-2.11.3.so
00007f648141d000      4K      0K      0K      0K      4K r--p
/lib64/libcrypt-2.11.3.so
00007f648141e000      4K      0K      0K      0K      4K rw-p
/lib64/libcrypt-2.11.3.so
00007f648141f000    184K      0K      0K      0K      0K rw-p [anon]
00007f648144d000   1484K      4K      4K      0K      0K r-xp
/usr/lib64/libcrypto.so.0.9.8
00007f64815c0000   2044K      0K      0K      0K      0K ---p
/usr/lib64/libcrypto.so.0.9.8
00007f64817bf000     64K      0K      0K      0K     64K r--p
/usr/lib64/libcrypto.so.0.9.8
00007f64817cf000    100K      0K      0K      0K    100K rw-p
/usr/lib64/libcrypto.so.0.9.8
00007f64817e8000     16K      0K      0K      0K     12K rw-p [anon]
00007f64817ec000    308K      4K      4K      0K      0K r-xp
/usr/lib64/libssl.so.0.9.8
00007f6481839000   2044K      0K      0K      0K      0K ---p
/usr/lib64/libssl.so.0.9.8
00007f6481a38000      8K      0K      0K      0K      8K r--p
/usr/lib64/libssl.so.0.9.8
00007f6481a3a000     24K      0K      0K      0K     24K rw-p
/usr/lib64/libssl.so.0.9.8
00007f6481a40000      8K      4K      0K      0K      0K r-xp
/lib64/libdl-2.11.3.so
00007f6481a42000   2048K      0K      0K      0K      0K ---p
/lib64/libdl-2.11.3.so
00007f6481c42000      4K      0K      0K      0K      4K r--p
/lib64/libdl-2.11.3.so
00007f6481c43000      4K      0K      0K      0K      4K rw-p
/lib64/libdl-2.11.3.so
00007f6481c44000    100K     44K     36K      0K      0K r-xp
/usr/lib64/libsasl2.so.2.0.22
00007f6481c5d000   2048K      0K      0K      0K      0K ---p
/usr/lib64/libsasl2.so.2.0.22
00007f6481e5d000      4K      4K      4K      4K      0K r--p
/usr/lib64/libsasl2.so.2.0.22
00007f6481e5e000      4K      4K      4K      4K      0K rw-p
/usr/lib64/libsasl2.so.2.0.22
00007f6481e5f000     84K      4K      0K      0K      0K r-xp
/lib64/libnsl-2.11.3.so
00007f6481e74000   2044K      0K      0K      0K      0K ---p
/lib64/libnsl-2.11.3.so
00007f6482073000      4K      0K      0K      0K      4K r--p
/lib64/libnsl-2.11.3.so
00007f6482074000      4K      0K      0K      0K      4K rw-p
/lib64/libnsl-2.11.3.so
00007f6482075000      8K      0K      0K      0K      0K rw-p [anon]
00007f6482077000    364K      4K      0K      0K      0K r-xp
/lib64/libm-2.11.3.so
00007f64820d2000   2044K      0K      0K      0K      0K ---p
/lib64/libm-2.11.3.so
00007f64822d1000      4K      0K      0K      0K      4K r--p
/lib64/libm-2.11.3.so
00007f64822d2000    120K      0K      0K      0K      8K rw-p
/lib64/libm-2.11.3.so
00007f64822f0000     96K      4K      4K      0K      0K r-xp
/usr/lib64/libslp.so.1.0.0
00007f6482308000   2044K      0K      0K      0K      0K ---p
/usr/lib64/libslp.so.1.0.0
00007f6482507000      4K      0K      0K      0K      4K r--p
/usr/lib64/libslp.so.1.0.0
00007f6482508000      4K      0K      0K      0K      4K rw-p
/usr/lib64/libslp.so.1.0.0
00007f6482509000     92K     76K     26K      0K      0K r-xp
/lib64/libpthread-2.11.3.so
00007f6482520000   2048K      0K      0K      0K      0K ---p
/lib64/libpthread-2.11.3.so
00007f6482720000      4K      4K      4K      4K      0K r--p
/lib64/libpthread-2.11.3.so
00007f6482721000      4K      4K      4K      4K      0K rw-p
/lib64/libpthread-2.11.3.so
00007f6482722000     16K      4K      4K      4K      0K rw-p [anon]
00007f6482726000    400K      4K      4K      0K      0K r-xp
/usr/lib64/libodbc.so.1.0.0
00007f648278a000   2044K      0K      0K      0K      0K ---p
/usr/lib64/libodbc.so.1.0.0
00007f6482989000      4K      0K      0K      0K      4K r--p
/usr/lib64/libodbc.so.1.0.0
00007f648298a000     28K      0K      0K      0K     28K rw-p
/usr/lib64/libodbc.so.1.0.0
00007f6482991000      4K      0K      0K      0K      0K rw-p [anon]
00007f6482992000   1224K    196K    187K      0K      0K r-xp
/usr/lib64/libdb-4.5.so
00007f6482ac4000   2048K      0K      0K      0K      0K ---p
/usr/lib64/libdb-4.5.so
00007f6482cc4000     12K      4K      4K      4K      8K r--p
/usr/lib64/libdb-4.5.so
00007f6482cc7000     12K     12K     12K     12K      0K rw-p
/usr/lib64/libdb-4.5.so
00007f6482cca000     36K      4K      0K      0K      0K r-xp
/usr/lib64/libltdl.so.7.2.0
00007f6482cd3000   2044K      0K      0K      0K      0K ---p
/usr/lib64/libltdl.so.7.2.0
00007f6482ed2000      4K      0K      0K      0K      4K r--p
/usr/lib64/libltdl.so.7.2.0
00007f6482ed3000      4K      0K      0K      0K      4K rw-p
/usr/lib64/libltdl.so.7.2.0
00007f6482ed4000     56K     44K     14K      0K      0K r-xp
/usr/lib64/liblber-2.4.so.2.8.5
00007f6482ee2000   2044K      0K      0K      0K      0K ---p
/usr/lib64/liblber-2.4.so.2.8.5
00007f64830e1000      4K      4K      4K      4K      0K r--p
/usr/lib64/liblber-2.4.so.2.8.5
00007f64830e2000      4K      4K      4K      4K      0K rw-p
/usr/lib64/liblber-2.4.so.2.8.5
00007f64830e3000    308K     64K     64K      0K      0K r-xp
/usr/lib64/libldap_r-2.4.so.2.8.5
00007f6483130000   2048K      0K      0K      0K      0K ---p
/usr/lib64/libldap_r-2.4.so.2.8.5
00007f6483330000      4K      4K      4K      4K      0K r--p
/usr/lib64/libldap_r-2.4.so.2.8.5
00007f6483331000      8K      8K      8K      8K      0K rw-p
/usr/lib64/libldap_r-2.4.so.2.8.5
00007f6483333000     12K      8K      8K      8K      4K rw-p [anon]
00007f6483336000    288K     96K     96K      0K      0K r-xp
/usr/local/lib/libtcmalloc.so.4.1.0
00007f648337e000   2044K      0K      0K      0K      0K ---p
/usr/local/lib/libtcmalloc.so.4.1.0
00007f648357d000      4K      4K      4K      4K      0K r--p
/usr/local/lib/libtcmalloc.so.4.1.0
00007f648357e000      4K      4K      4K      4K      0K rw-p
/usr/local/lib/libtcmalloc.so.4.1.0
00007f648357f000    156K     92K     92K     88K     36K rw-p [anon]
00007f64835a6000    124K     40K      0K      0K      0K r-xp
/lib64/ld-2.11.3.so
00007f6483708000    212K      0K      0K      0K      0K r--s
/var/run/nscd/group
00007f648373d000    212K      0K      0K      0K      0K r--s
/var/run/nscd/passwd
00007f6483772000    212K      0K      0K      0K      0K r--s
/var/run/nscd/dbvbS4oY
00007f64837a7000     48K     28K     28K     24K     20K rw-p [anon]
00007f64837bc000     24K      8K      8K      8K      0K rw-s /SYSV00000007
00007f64837c2000      8K      4K      4K      4K      4K rw-p [anon]
00007f64837c4000      4K      4K      4K      4K      0K r--p
/lib64/ld-2.11.3.so
00007f64837c5000      4K      4K      4K      4K      0K rw-p
/lib64/ld-2.11.3.so
00007f64837c6000      4K      4K      4K      4K      0K rw-p [anon]
00007f64837c7000   1944K    456K    456K      0K      0K r-xp
/usr/lib/openldap/slapd
00007f6483bac000     20K      8K      8K      8K     12K r--p
/usr/lib/openldap/slapd
00007f6483bb1000     48K     24K     24K     24K     24K rw-p
/usr/lib/openldap/slapd
00007f6483bbd000   3812K   1052K   1052K    984K   1788K rw-p [heap]
00007f6483f76000 5004408K 4574484K 4574484K 4574392K 257508K rw-p [heap]
00007fff9f532000    132K      4K      4K      4K     32K rw-p [stack]
00007fff9f5b6000      4K      4K      0K      0K      0K r-xp [vdso]
ffffffffff600000      4K      0K      0K      0K      0K r-xp [vsyscall]
Total:           7350688K 6749464K 6748875K 6747724K 259932K

5091328K writable-private, 80668K readonly-private, 2178692K shared,
and 4900700K referenced

================================================================================

last free -m before crash:
-------------------------------
             total       used       free     shared    buffers     cached
Mem:          6928       6813        115          0          0       2180
-/+ buffers/cache:       4632       2296
Swap:         2047        288       1759

last free after crash:
-------------------------------

             total       used       free     shared    buffers     cached
Mem:          6928       4978       1950          0          0       2166
-/+ buffers/cache:       2810       4117
Swap:         2047         42       2005
================================================================================

and vmstat:
-------------------------------
procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  4 295284 110148    540 2255804  108    0 13428     0  796 1373  3
5 27 65  0
 0  3 295260 118140    608 2242804   36    0  9308     0 2227 1309  2
5 23 70  0
 0  3 295224 116668    680 2237396   48    0  4008     0 1253 2002  3
6 52 39  0
 1  0 295164 116296    736 2232588  128    0  6028     0 1083  954  3
14 45 39  0
 0  0  42332 4742716    768 2219044  784    0  1104     0 1428 1178  3
23 63 11  0
 0  0  42332 4743020    768 2219168    0    0     0     0  253   97  0
 0 100  0  0
 0  0  42332 4743408    768 2219160    0    0     0     0  201   98  0
 0 100  0  0
 0  1  42300 4743408    768 2219140  216    0   216     0  295  150  0
 0 99  1  0



What I see, that slapd had reserved "Total: 7350688K"
(overcommitted?), but only referenced 4900700K.
Why does slapd reserve so much memory and use it not? Because of this,
 I changed the default values for memory overcomittment

from:
vm.overcommit_memory = 0 (guess, default)
vm.overcommit_ratio = 50

to:
vm.overcommit_memory=2 (strict overcommit)
vm.overcommit_ratio=100

but it crashed again (ch_malloc of 34545 bytes failed):
.
.
00007f2bf8e1a000 5019920K 4597460K 4597460K 4597116K 333472K rw-p [heap]
00007fffe86c5000    132K      0K      0K      0K     36K rw-p [stack]
00007fffe87af000      4K      4K      0K      0K      0K r-xp [vdso]
ffffffffff600000      4K      0K      0K      0K      0K r-xp [vsyscall]
Total:           7333420K 6766996K 6766484K 6765536K 336192K

5074068K writable-private, 80660K readonly-private, 2178692K shared,
and 4681876K referenced



So in spite of increase the memory from 16GByte to 24 GByte (one of
the production server - configuration described in the first mail)
same results.

Is there anywhere a memory leak - how can I find out this?

Thanks Meike

P.S., mdb is not the option now, we need time to evaluate this ...