ldap_sasl_interactive_bind_s failed (-2) using openldap v2.3.43 when trying to replicate via syncrepl.

[Kristian Kostecky <kris@theendless.org>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi guys,

I have a configuration that consists of 3 ldap servers.  One is the provider and there are 2 consumers.  I am using syncrepl to do the synchronization.  simple and anonymous binds are totally disabled and Kerberos must be used via SASL (GSSAPI) and TLS to connect to the LDAP server.  

distro: centos 5.4
openldap 2.3.43 
cyrus-sasl 2.1.22

Other things:
- - clocks are all in sync
- - hostnames all have forward and reverse mappings and all dns servers in /etc/resolv.conf respond with proper entries on the consumer and both providers.

Here's the catch, the two providers are configured the same (except for hostnames/ips) and the first one works perfect.  What is really frustrating is the lack of logging that is available to tell me what the problem is.  I've tried loglevel -1 and it gave me even less info in regards to the SASL authentication than leaving it off.  

The affected consumer is giving me:

Mar 31 22:41:00 ZZZ slapd[2442]: slapd starting 
Mar 31 22:41:02 ZZZ slapd[2442]: do_syncrep1: rid 010 ldap_sasl_interactive_bind_s failed (-2) 
Mar 31 22:41:02 ZZZ slapd[2442]: do_syncrepl: rid 010 quitting 

On the "Provider":

Mar 31 17:49:54 XXX slapd[3494]: conn=212 fd=21 ACCEPT from IP=10.130.1.230:60288 (IP=0.0.0.0:389) 
Mar 31 17:49:54 XXX slapd[3494]: conn=212 op=0 STARTTLS 
Mar 31 17:49:54 XXX slapd[3494]: conn=212 op=0 RESULT oid= err=0 text= 
Mar 31 17:49:54 XXX slapd[3494]: conn=212 fd=21 TLS established tls_ssf=256 ssf=256 
Mar 31 17:49:54 XXX slapd[3494]: conn=212 op=1 UNBIND 
Mar 31 17:49:54 XXX slapd[3494]: conn=212 fd=21 closed 

This is what's REALLY weird - from the affected/broken box, ZZZ, after I kinit, I can do an LDAP search or ldapwhoami, no problems!  So, kerberos and GSSAPI via SASL is working fine.  ie:

ldapsearch -H ldaps://XXX/ -Y GSSAPI -> will dump the entries.
or
ldapwhoami -H ldaps://XXX/ -Y GSSAPI -> shows me that proper creds

If I destroy the credentials, it doesn't work as would be expected.

ON the working consumer, the behaviour is that I can ldapsearch and ldapwhoami properly after I kinit and when I start ldap it will authenticate properly with the provider via SASL GSSAPI and replicates the DB.  If I kdestroy the credentials and start it, I get the same error that I'm struggling with on the box that doesn't work ->ldap_sasl_interactive_bind_s failed (-2)  This behaviour leads me to believe that for some reason the ldap server on the box that doesn't work is having problems transmitting the kerberos credentials to the provider, whereas the ldapsearch and ldapwhoami binaries are not having problems.

There are some suspicious differences between the consumer that works and the broken one.  The provider and consumer that works both have TLDs that match - '.com' and the consumer whose synrepl process won't authenticate is part of the .eu TLD.  However, as you can see below in the krb5.conf files, the .com and .eu TLDS are always mapped to the same authentication realm.  PLUS, again, ldapsearch and ldapwhoami WORK.  It's just the syncrepl process that isn't quite getting the auth right.

This is the provider's pertinent configs:

slapd.conf:
overlay syncprov
syncprov-checkpoint 100 10
syncprov-sessionlog 100

This is the consumer's pertinent configs (WORKS ON one, not on the other)
slapd.conf:
syncrepl rid=10
        provider=ldap://xxx.XXX.com
        starttls=yes
        type=refreshOnly
        interval=00:00:01:00
        searchbase="dc=XXX,dc=com"
        schemachecking=off
        bindmethod=sasl
        saslmech=GSSAPI

krb5.conf [same as provider and kerb server]:
[libdefaults]
 default_realm = BOUNCE.AAA.COM
 encrypt = true
 allow_weak_crypto = false
 clockskew = 600
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 8h
 forwardable = no
 proxiable = no

[realms]
 BOUNCE.AAA.COM = {
  kdc = XXX.com
  kdc = YYY.com
  kdc = ZZZ.eu
  admin_server = XXX.com
 }

[domain_realm]
 .com = BOUNCE.AAA.COM
 .eu = BOUNCE.AAA.COM


All help is greatly appreciated!  This has been going on for days and I've already yanked out most of my hair.  Thank you.

Kris.

PGP Key: 4CC63A18
PGP Server: pool.sks-keyservers.net

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.14 (Darwin)

iEYEARECAAYFAkuzyroACgkQ2C/J5/UUQWEuUACdH/BhiZgTXFWbNMXS7Q99k8Rg
VY8An3YWKcpnkxVYvZMlelkT0TIpYuAP
=O9KI
-----END PGP SIGNATURE-----