Openldap 2.4.15, TLS, SASL EXTERNAL and authzregexp = segfaults

[Mathieu MILLET <ldap@htam.net>]

Hi everyone, 

We have a configuration with 2 Openldap in Multimaster Replication mode,
using TLS, client certificate and SASL EXTERNAL to secure the replication.
(Two sets of certificate are used to differentiate the replication of
cn=config and the data backend)

It is working in 2.4.13 (on Red Hat Entreprise Linux 4.5 and Debian 5),
compiled from sources, with openssl libs (not gnutls).

Being affected by ITS#5906 (slapo-rwm with back-config) and ITS#5843 (slapd
syncrepl MMR with deleted entries), I decided to try on a (test)
environment this new version. 

With 2.4.15 (and also reproduced in 2.4.14), our configuration segfaults on
one of the two nodes at a short period of time after the 1st replication.
When restarting the segfaulted node, the other segfaults and so on.

The segfault happens when just adding the syncrepl configuration for the
cn=config backend, but some times they are alive long enough to enable
syncrepl options for the databackend, but then again, segfaults always
happen.

During some segfaults, I got some backtraces that follow :
*** glibc detected *** /usr/local/libexec/slapd: realloc(): invalid
pointer: 0xb6db9260 ***
======= Backtrace: =========
/lib/i686/cmov/libc.so.6[0xb6ccf624]
/lib/i686/cmov/libc.so.6(realloc+0x242)[0xb6cd3c82]
/usr/lib/i686/cmov/libcrypto.so.0.9.8[0xb6e224c5]
/usr/lib/i686/cmov/libcrypto.so.0.9.8(CRYPTO_realloc+0xab)[0xb6e22c0b]
/usr/lib/i686/cmov/libcrypto.so.0.9.8(BUF_MEM_grow+0x75)[0xb6e83415]
/usr/lib/i686/cmov/libcrypto.so.0.9.8[0xb6ea95a4]
/usr/lib/i686/cmov/libcrypto.so.0.9.8(ASN1_item_ex_i2d+0x14d)Abandon

or
*** glibc detected *** /usr/local/libexec/slapd: realloc(): invalid
pointer: 0xb6de4260 ***
======= Backtrace: =========
/lib/i686/cmov/libc.so.6[0xb6cfa624]
/lib/i686/cmov/libc.so.6(realloc+0x242)[0xb6cfec82]
/usr/lib/i686/cmov/libcrypto.so.0.9.8[0xb6e4d4c5]
/usr/lib/i686/cmov/libcrypto.so.0.9.8(CRYPTO_realloc+0xab)[0xb6e4dc0b]
/usr/lib/i686/cmov/libcrypto.so.0.9.8(BUF_MEM_grow+0x75)[0xb6eae415]
/usr/lib/i686/cmov/libcrypto.so.0.9.8[0xb6ed45a4]
/usr/lib/i686/cmov/libcrypto.so.0.9.8(ASN1_item_ex_i2d+0x14d)[0xb6edbfbd]
/usr/lib/i686/cmov/libcrypto.so.0.9.8[0xb6edc5b5]
/usr/lib/i686/cmov/libcrypto.so.0.9.8(ASN1_item_ex_i2d+0x2f3)[0xb6edc163]
/usr/lib/i686/cmov/libcrypto.so.0.9.8[0xb6edc5b5]
/usr/lib/i686/cmov/libcrypto.so.0.9.8(ASN1_item_ex_i2d+0x2f3)[0xb6edc163]
/usr/lib/i686/cmov/libcrypto.so.0.9.8(ASN1_item_i2d+0x53)[0xb6edc923]
/usr/lib/i686/cmov/libcrypto.so.0.9.8(i2d_X509+0x2e)[0xb6ed506e]
/usr/lib/i686/cmov/libssl.so.0.9.8(ssl3_output_cert_chain+0x3d4)[0xb6f7b824]
/usr/lib/i686/cmov/libssl.so.0.9.8(ssl3_send_client_certificate+0x142)[0xb6f721b2]
/usr/lib/i686/cmov/libssl.so.0.9.8(ssl3_connect+0xb3)[0xb6f759d3]
/usr/lib/i686/cmov/libssl.so.0.9.8(SSL_connect+0x2a)[0xb6f89c1a]
/usAbandon

It definitely has something to do with TLS stuff.

After more testing, the ldap* clients also segfault when performing TLS and
SASL External with Client Certificate.

Has anybody encounter this behaviour ?

Thanks in advance for any help,
Sincerely yours, Mathieu MILLET.


******************* Startup config (of one node) **************
----------------
slapd.d/cn=config/olcDatabase={-1}frontend.ldif
----------------
dn: olcDatabase={-1}frontend
objectClass: olcDatabaseConfig
objectClass: olcFrontendConfig
olcDatabase: {-1}frontend
olcAccess: {0}to dn.base=""  by * read
olcAccess: {1}to dn.base="cn=subschema"  by * read
olcAccess: {2}to *  by self write  by users read  by anonymous auth
olcAddContentAcl: FALSE
olcLastMod: TRUE
olcMaxDerefDepth: 0
olcReadOnly: FALSE
olcSchemaDN: cn=Subschema
olcMonitoring: FALSE
structuralObjectClass: olcDatabaseConfig
entryUUID: 10002a99-3485-4805-a247-9e4ee777135d
creatorsName: cn=config
createTimestamp: 20090224192423Z
entryCSN: 20090224192423.202231Z#000000#000#000000
modifiersName: cn=config
modifyTimestamp: 20090224192423Z

----------------
slapd.d/cn=config/olcDatabase={0}config.ldif
----------------
dn: olcDatabase={0}config
objectClass: olcDatabaseConfig
olcDatabase: {0}config
olcAccess: {0}to *  by * none
olcAddContentAcl: TRUE
olcLastMod: TRUE
olcMaxDerefDepth: 15
olcReadOnly: FALSE
olcRootDN: cn=config
olcRootPW:: c2VjcmV0
olcMonitoring: FALSE
structuralObjectClass: olcDatabaseConfig
entryUUID: fc35a505-ba8f-4bbf-828e-b061bb3aabba
creatorsName: cn=config
createTimestamp: 20090224192423Z
entryCSN: 20090224192423.202231Z#000000#000#000000
modifiersName: cn=config
modifyTimestamp: 20090224192423Z

----------------
slapd.d/cn=config/olcDatabase={2}hdb/olcOverlay={0}ppolicy.ldif
----------------
dn: olcOverlay={0}ppolicy
objectClass: olcConfig
objectClass: olcOverlayConfig
objectClass: olcPPolicyConfig
olcOverlay: {0}ppolicy
olcPPolicyDefault: cn=default,ou=policies,dc=htam,dc=net
olcPPolicyHashCleartext: FALSE
olcPPolicyUseLockout: FALSE
structuralObjectClass: olcPPolicyConfig
entryUUID: 8078dd1d-369e-4c62-9fdc-1ce6820482d8
creatorsName: cn=config
createTimestamp: 20090302125140Z
entryCSN: 20090302125140.681319Z#000000#001#000000
modifiersName: cn=config
modifyTimestamp: 20090302125140Z

----------------
slapd.d/cn=config/olcDatabase={2}hdb/olcOverlay={1}memberof.ldif
----------------
dn: olcOverlay={1}memberof
objectClass: olcConfig
objectClass: olcOverlayConfig
objectClass: olcMemberOf
olcOverlay: {1}memberof
olcMemberOfDangling: ignore
olcMemberOfRefInt: TRUE
olcMemberOfGroupOC: groupOfUniqueNames
olcMemberOfMemberAD: uniqueMember
olcMemberOfMemberOfAD: memberOf
structuralObjectClass: olcMemberOf
entryUUID: b0a0abdd-77ef-47f6-a1e1-52637e30ebcc
creatorsName: cn=config
createTimestamp: 20090302125140Z
entryCSN: 20090302125140.683800Z#000000#001#000000
modifiersName: cn=config
modifyTimestamp: 20090302125140Z

----------------
slapd.d/cn=config/olcDatabase={2}hdb/olcOverlay={2}refint.ldif
----------------
dn: olcOverlay={2}refint
objectClass: olcConfig
objectClass: olcOverlayConfig
objectClass: olcRefintConfig
olcOverlay: {2}refint
olcRefintAttribute: uniqueMember
olcRefintNothing: cn=Manager,dc=htam,dc=net
structuralObjectClass: olcRefintConfig
entryUUID: 13d0a0a0-8284-447c-9d49-426e37692f57
creatorsName: cn=config
createTimestamp: 20090302125140Z
entryCSN: 20090302125140.685440Z#000000#001#000000
modifiersName: cn=config
modifyTimestamp: 20090302125140Z

----------------
slapd.d/cn=config/cn=module{0}.ldif
----------------
dn: cn=module{0}
objectClass: olcModuleList
cn: module{0}
olcModulePath: /usr/local/libexec/openldap
olcModuleLoad: {0}memberof.la
olcModuleLoad: {1}ppolicy.la
olcModuleLoad: {2}refint.la
olcModuleLoad: {3}retcode.la
olcModuleLoad: {4}rwm.la
olcModuleLoad: {5}syncprov.la
olcModuleLoad: {6}unique.la
olcModuleLoad: {7}back_monitor.la
olcModuleLoad: {8}back_hdb.la
olcModuleLoad: {9}back_relay.la
structuralObjectClass: olcModuleList
entryUUID: 353f4a38-3a12-446f-9176-570021c59341
creatorsName: cn=config
createTimestamp: 20090224192423Z
entryCSN: 20090224192423.202231Z#000000#001#000000
modifiersName: cn=config
modifyTimestamp: 20090224192423Z

----------------
slapd.d/cn=config/olcDatabase={2}hdb.ldif
----------------
dn: olcDatabase={2}hdb
objectClass: olcDatabaseConfig
objectClass: olcHdbConfig
olcDatabase: {2}hdb
olcDbDirectory: /usr/local/var/openldap-data/
olcSuffix: dc=htam,dc=net
olcAccess: {0}to attrs=userPassword by
group/groupOfUniqueNames/uniqueMember="
 cn=ldapadmins,ou=groups,dc=htam,dc=net" write by
dn.subtree="ou=replicators,d
 c=htam,dc=net" read by dn.subtree="ou=computers,dc=htam,dc=net" auth by
self 
 =xwd by anonymous auth
olcAccess: {1}to
attrs=entry,objectClass,uid,uidNumber,gidNumber,loginShell,cn
 ,gecos,description,homeDirectory by
group/groupOfUniqueNames/uniqueMember="cn
 =ldapadmins,ou=groups,dc=htam,dc=net" write by
dn.subtree="ou=replicators,dc=
 htam,dc=net" read by dn.subtree="ou=computers,dc=htam,dc=net" read by self
re
 ad
olcAccess::
ezJ9dG8gYXR0cnM9dW5pcXVlTWVtYmVyIGJ5IGdyb3VwL2dyb3VwT2ZVbmlxdWVOYW
 1lcy91bmlxdWVNZW1iZXI9ImNuPWxkYXBhZG1pbnMsb3U9Z3JvdXBzLGRjPWh0YW0sZGM9bmV0IiB
 3cml0ZSBieSBkbi5zdWJ0cmVlPSJvdT1yZXBsaWNhdG9ycyxkYz1odGFtLGRjPW5ldCIgcmVhZCBi
 eSBkbi5zdWJ0cmVlPSJvdT1jb21wdXRlcnMsZGM9aHRhbSxkYz1uZXQiIHJlYWQg
olcAccess: {3}to * by
group/groupOfUniqueNames/uniqueMember="cn=ldapadmins,ou=
 groups,dc=htam,dc=net" write by dn.subtree="ou=replicators,dc=htam,dc=net"
re
 ad by self read
olcLastMod: TRUE
olcMaxDerefDepth: 15
olcReadOnly: FALSE
olcRootDN: cn=Manager,dc=htam,dc=net
olcRootPW:: c2VjcmV0
olcMonitoring: TRUE
olcDbCacheSize: 1000
olcDbConfig: {0}set_cachesize 0 268435456 1
olcDbConfig: {1}set_lg_regionmax 262144
olcDbConfig: {2}set_lg_bsize 2097152
olcDbConfig: {3}set_flags DB_LOG_AUTOREMOVE
olcDbIndex: objectClass eq
olcDbIndex: uid pres,eq
olcDbIndex: cn pres,eq,sub
olcDbIndex: sn pres,eq,sub
olcDbIndex: givenname pres,eq,sub
olcDbIndex: uniqueMember pres,eq
olcDbIndex: memberUid pres,eq
olcDbIndex: entryCSN eq
olcDbIndex: entryUUID eq
olcDbIndex: ipServicePort eq
olcDbIndex: ipServiceProtocol eq
olcDbIndex: oncRpcNumber eq
olcDbIndex: ipProtocolNumber eq
structuralObjectClass: olcHdbConfig
entryUUID: 9f1eb1ca-a001-46db-aa58-4fc7897c64cc
creatorsName: cn=config
createTimestamp: 20090302125140Z
entryCSN: 20090302125140.183122Z#000000#001#000000
modifiersName: cn=config
modifyTimestamp: 20090302125140Z

----------------
slapd.d/cn=config/olcDatabase={1}monitor.ldif
----------------
dn: olcDatabase={1}monitor
objectClass: olcDatabaseConfig
olcDatabase: {1}monitor
olcAccess: {0}to *  by dn.base="cn=Manager,dc=htam,dc=net" read  by * none
olcLastMod: TRUE
olcMaxDerefDepth: 15
olcReadOnly: FALSE
olcMonitoring: FALSE
structuralObjectClass: olcDatabaseConfig
entryUUID: 6d366d19-e3ce-417b-a0b6-fd41bc690d83
creatorsName: cn=config
createTimestamp: 20090302125140Z
entryCSN: 20090302125140.118423Z#000000#001#000000
modifiersName: cn=config
modifyTimestamp: 20090302125140Z

----------------
slapd.d/cn=config.ldif
----------------
dn: cn=config
objectClass: olcGlobal
cn: config
olcConfigFile: slapd.conf.start
olcConfigDir: slapd.d.start
olcArgsFile: /usr/local/var/run/slapd.args
olcAttributeOptions: lang-
olcAuthzPolicy: none
olcConcurrency: 0
olcConnMaxPending: 100
olcConnMaxPendingAuth: 1000
olcGentleHUP: FALSE
olcIdleTimeout: 0
olcIndexSubstrIfMaxLen: 4
olcIndexSubstrIfMinLen: 2
olcIndexSubstrAnyLen: 4
olcIndexSubstrAnyStep: 2
olcIndexIntLen: 4
olcLocalSSF: 71
olcLogLevel: Packets
olcLogLevel: Config
olcLogLevel: Stats
olcLogLevel: Sync
olcPidFile: /usr/local/var/run/slapd.pid
olcReadOnly: FALSE
olcSaslSecProps: noplain,noanonymous
olcServerID: 1 ldap://vmlinux01/
olcSockbufMaxIncoming: 262143
olcSockbufMaxIncomingAuth: 16777215
olcThreads: 16
olcTLSCACertificateFile: /usr/local/etc/openldap/cacerts/cacert.pem
olcTLSCertificateFile: /usr/local/etc/openldap/slapd.cert.pem
olcTLSCertificateKeyFile: /usr/local/etc/openldap/slapd.key.pem
olcTLSCipherSuite: HIGH:MEDIUM
olcTLSCRLCheck: none
olcTLSVerifyClient: try
olcToolThreads: 1
structuralObjectClass: olcGlobal
entryUUID: 67b85bb6-58a2-4c6e-abd5-2bf7ce077d69
creatorsName: cn=config
createTimestamp: 20090224192423Z
entryCSN: 20090302142216.165509Z#000000#001#000000
modifiersName: cn=config
modifyTimestamp: 20090302142216Z


******************* LDIF for activating syncrepl on cn=config
**************
dn: cn=config
changetype: modify
replace: olcServerID
olcServerID: 1 "ldap://vmlinux01";
olcServerID: 2 "ldap://vmlinux02";
-
add: olcAuthzRegexp
olcAuthzRegexp: "cn=.*_repl_config,o=Htam.net Inc.,c=fr" "cn=config"

dn: olcOverlay=syncprov,olcDatabase={0}config,cn=config
changetype: add
objectClass: olcOverlayConfig
objectClass: olcSyncProvConfig
olcOverlay: syncprov

dn: olcDatabase={0}config,cn=config
changetype: modify
replace: olcSyncRepl
olcSyncRepl: rid=001 
 provider="ldap://vmlinux01"; 
 bindmethod=sasl 
 saslmech="EXTERNAL" 
 searchbase="cn=config" 
 type=refreshAndPersist 
 starttls=critical 
 retry="5 5 60 +" 
 timeout=1 
 tls_cacert=/usr/local/etc/openldap/cacerts/cacert.pem 
 tls_cert=/usr/local/etc/openldap/slapd_repl_config.cert.pem 
 tls_key=/usr/local/etc/openldap/slapd_repl_config.key.pem
olcSyncRepl: rid=002 
 provider="ldap://vmlinux02"; 
 bindmethod=sasl 
 saslmech="EXTERNAL" 
 searchbase="cn=config" 
 type=refreshAndPersist 
 starttls=critical 
 retry="5 5 60 +" 
 timeout=1 
 tls_cacert=/usr/local/etc/openldap/cacerts/cacert.pem 
 tls_cert=/usr/local/etc/openldap/slapd_repl_config.cert.pem 
 tls_key=/usr/local/etc/openldap/slapd_repl_config.key.pem
-
add: olcMirrorMode
olcMirrorMode: TRUE
-
add: olcLimits
olcLimits: dn="cn=config" size=unlimited time=unlimited

******************* LDIF for activating syncrepl on data backend
**************
dn: cn=config
changetype: modify
add: olcAuthzRegexp
olcAuthzRegexp: "cn=.*_replicator,o=Htam.net Inc.,c=FR"
cn=Replicator,ou=replicators,dc=htam,dc=net

dn: olcOverlay=syncprov,olcDatabase={2}hdb,cn=config
changetype: add
objectClass: olcOverlayConfig
objectClass: olcSyncProvConfig
olcOverlay: syncprov

dn: olcDatabase={2}hdb,cn=config
changetype: modify
add: olcLimits
olcLimits: dn.subtree="ou=replicators,dc=htam,dc=net" size=unlimited
time=unlimited
-
add: olcSyncRepl
olcSyncRepl: rid=201 
 provider="ldap://vmlinux01"; 
 bindmethod=sasl 
 saslmech="EXTERNAL" 
 searchbase="dc=htam,dc=net" 
 type=refreshOnly 
 interval=00:00:00:10 
 retry="5 5 300 +" 
 timeout=1 
 starttls=critical 
 tls_cacert=/usr/local/etc/openldap/cacerts/cacert.pem 
 tls_cert=/usr/local/etc/openldap/slapd_replicator.cert 
 tls_key=/usr/local/etc/openldap/slapd_replicator.key
olcSyncRepl: rid=202 
 provider="ldap://vmlinux02"; 
 bindmethod=sasl 
 saslmech="EXTERNAL" 
 searchbase="dc=htam,dc=net" 
 type=refreshOnly 
 interval=00:00:00:10 
 retry="5 5 300 +" 
 timeout=1 
 starttls=critical 
 tls_cacert=/usr/local/etc/openldap/cacerts/cacert.pem 
 tls_cert=/usr/local/etc/openldap/slapd_replicator.cert 
 tls_key=/usr/local/etc/openldap/slapd_replicator.key
-
add: olcMirrorMode
olcMirrorMode: TRUE


--
Mathieu MILLET
mailto:ldap@htam.net
----