[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: ldap clients cant connect to replica after failure

To: Jonathan Clarke <jonathan@phillipoux.net>
Subject: Re: ldap clients cant connect to replica after failure
From: Scott Behrens <sbehrens@gmail.com>
Date: Mon, 9 Nov 2009 08:43:21 -0600
Cc: openldap-technical@openldap.org
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=i2r9i1Fs14Ysuq5xJM4hgGaKHWx8iXpi6kGIuQeKE8E=; b=AUdfrwwH5ZYw3YulZqt+CATamO11KD0EqJVJYKw4L3RfavQmqT2sAZujnsyDxTv2S3 PmXM0fcAyyJp72+qe+to/hp91r0ylNmLO/31yNx/TYZ0X6HEU4SBHv8RqAjQOemBh68e ur6AIN/FKLDo9G/MzQR6uUYrDiMqtYs892lSU=
Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=MeUhiPhpOupp1L6ShfDPobTSywRGQ/s3oLCRetD1noQ+Lc0OAeRSlkpi8s+xCnRcy7 uv+7pLkxpZfDHOUEIBGMwehCYh8WuJIjv+bHR/ON7nAbRwHXAFkKNLAMiLm7m/9epVg9 H7Dj8q2C4e4Gje0RJ6e7+MlZxG+v850uxzgRs=
In-reply-to: <4AF82014.40001@phillipoux.net>
References: <46fca6db0911060631x134bbcd7hffdd8c8cf96d6a7a@mail.gmail.com> <4AF7DDE5.6030908@phillipoux.net> <46fca6db0911090533t75226108ge371e64a8707cdf5@mail.gmail.com> <4AF82014.40001@phillipoux.net>

I restarted slapd after changing the log setting in slapd.conf.  I
also noticed that when doing a ldapsearch from the client host, it
always searched the primary.  Nothing seems to be happening on the
replica:

ldapsearch -xLLL -b "dc=domain,dc=com"

Nothing in replica ldap.log.  In primary log:

ov  9 08:42:23 mnetmgt1 slapd[14019]: >>> slap_listener(ldap:///)
Nov  9 08:42:23 mnetmgt1 slapd[14019]: connection_get(21): got connid=69
Nov  9 08:42:23 mnetmgt1 slapd[14019]: connection_read(21): checking
for input on id=69
Nov  9 08:42:23 mnetmgt1 slapd[14019]: do_bind
Nov  9 08:42:23 mnetmgt1 slapd[14019]: >>> dnPrettyNormal: <>
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <<< dnPrettyNormal: <>, <>
Nov  9 08:42:23 mnetmgt1 slapd[14019]: do_bind: version=3 dn="" method=128
Nov  9 08:42:23 mnetmgt1 slapd[14019]: send_ldap_result: conn=69 op=0 p=3
Nov  9 08:42:23 mnetmgt1 slapd[14019]: send_ldap_response: msgid=1 tag=97 err=0
Nov  9 08:42:23 mnetmgt1 slapd[14019]: do_bind: v3 anonymous bind
Nov  9 08:42:23 mnetmgt1 slapd[14019]: connection_get(21): got connid=69
Nov  9 08:42:23 mnetmgt1 slapd[14019]: connection_read(21): checking
for input on id=69
Nov  9 08:42:23 mnetmgt1 slapd[14019]: do_search
Nov  9 08:42:23 mnetmgt1 slapd[14019]: >>> dnPrettyNormal: <dc=domain,dc=com>
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <<< dnPrettyNormal:
<dc=domain,dc=com>, <dc=domain,dc=com>
Nov  9 08:42:23 mnetmgt1 slapd[14019]: ==> limits_get: conn=69 op=1
dn="[anonymous]"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => bdb_search
Nov  9 08:42:23 mnetmgt1 slapd[14019]: bdb_dn2entry("dc=domain,dc=com")
Nov  9 08:42:23 mnetmgt1 slapd[14019]: search_candidates:
base="dc=domain,dc=com" (0x00000001) scope=2
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => bdb_dn2idl("dc=domain,dc=com")
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => bdb_presence_candidates (objectClass)
Nov  9 08:42:23 mnetmgt1 slapd[14019]: bdb_search_candidates: id=-1
first=1 last=27
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="cn=Manager,dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="ou=profile,dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="ou=People,dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="ou=Group,dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="cn=Users,ou=Group,dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="cn=proxyagent,ou=profile,dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="cn=default,ou=profile,dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="cn=tls_profile,ou=profile,dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="uid=bobbyt,ou=People,dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="ou=Groups,dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="cn=bobbyt,ou=Groups,dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="uid=jenkins,ou=People,dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="uid=scott,ou=People,dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="cn=developers,ou=Group,dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="cn=developers,ou=Groups,dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="uid=izaydlin,ou=People,dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="cn=izaydlin,ou=Groups,dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="uid=egutman,ou=People,dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="cn=egutman,ou=Groups,dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="uid=trodgers,ou=People,dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: => send_search_entry: conn 69
dn="cn=trodgers,ou=Groups,dc=domain,dc=com"
Nov  9 08:42:23 mnetmgt1 slapd[14019]: <= send_search_entry: conn 69 exit.
Nov  9 08:42:23 mnetmgt1 slapd[14019]: send_ldap_result: conn=69 op=1 p=3
Nov  9 08:42:23 mnetmgt1 slapd[14019]: send_ldap_response: msgid=2 tag=101 err=0
Nov  9 08:42:23 mnetmgt1 slapd[14019]: connection_get(21): got connid=69
Nov  9 08:42:23 mnetmgt1 slapd[14019]: connection_read(21): checking
for input on id=69
Nov  9 08:42:23 mnetmgt1 slapd[14019]: ber_get_next on fd 21 failed
errno=0 (Success)
Nov  9 08:42:23 mnetmgt1 slapd[14019]: connection_closing: readying
conn=69 sd=21 for close
Nov  9 08:42:23 mnetmgt1 slapd[14019]: do_unbind
Nov  9 08:42:23 mnetmgt1 slapd[14019]: connection_close: deferring conn=69 sd=-1
Nov  9 08:42:23 mnetmgt1 slapd[14019]: connection_resched: attempting
closing conn=69 sd=21
Nov  9 08:42:23 mnetmgt1 slapd[14019]: connection_close: conn=69 sd=-1



On Mon, Nov 9, 2009 at 7:58 AM, Jonathan Clarke <jonathan@phillipoux.net> wrote:
> On 09/11/2009 14:33, Scott Behrens wrote:
>>
>> The client pointing to the replica is a 3rd host also running centOS.
>> OpenLDAP 2.3.43.  I can pretty much narrow down the issue to
>> configuration, our network/server config side is pretty
>> straightforward.
>>
>> I turned the logging to 256 on the replica, I pointed my client using
>> authconfig to the replica.
>>
>>> su - ldapuser
>>
>> Just hangs, in the logs on replica:
>>
>> Nov  9 02:25:48 mnetmgt2 slapd[30963]: slapd shutdown: initiated
>> Nov  9 02:25:48 mnetmgt2 slapd[30963]: ====>  bdb_cache_release_all
>> Nov  9 02:25:48 mnetmgt2 slapd[30963]: slapd destroy: freeing system
>> resources.
>> Nov  9 02:25:48 mnetmgt2 slapd[30963]: slapd stopped.
>> Nov  9 02:25:48 mnetmgt2 slapd[15086]: @(#) $OpenLDAP: slapd 2.3.43
>> (Jan 21 2009 03:59:37) $
>>
>> mockbuild@builder10.centos.org:/builddir/build/BUILD/openldap-2.3.43/openldap-2.3.43/build-servers/servers/slapd
>> Nov  9 02:25:48 mnetmgt2 slapd[15087]: slapd starting
>
> This log just shows slapd being shut down and restarted. Nothing connection
> related - did you restart slapd after changing the loglevel in slapd.conf?
> Without more logs, it's hard to tell what's going on.
>
> A good test would be to use ldapsearch directly to interrogate your two
> servers. This will help determine whether the problem is coming from your
> LDAP server(s) or your system authentication settings.
>
> Hope this helps,
> Jonathan
>
>
>> If I point it back to the primary, it works fine.  Here are the slapd
>> files, they look correct based on what I read in the documentation.
>>
>>
>> Primary Slapd.conf:
>>
>> include /etc/openldap/schema/core.schema
>> include /etc/openldap/schema/cosine.schema
>> include /etc/openldap/schema/nis.schema
>> include /etc/openldap/schema/inetorgperson.schema
>> include /etc/openldap/schema/solaris.schema
>> include /etc/openldap/schema/sudo.schema
>> include /etc/openldap/schema/dyngroup.schema
>> include /etc/openldap/schema/misc.schema
>> include /etc/openldap/schema/openldap.schema
>>
>>
>> # DATABASE CONFIG
>> database bdb
>> suffix "dc=domain,dc=com"
>> rootdn "cn=Manager,dc=domain,dc=com"
>> rootpw {SSHA}somepassword
>>
>> directory       /var/lib/ldap
>>
>> index entryCSN,entryUUID eq,pres
>> index objectClass,uid,uidNumber,gidNumber,ou eq
>> index cn,mail,surname,givenname eq,subinitial
>> index memberUid eq
>> index nisDomain eq
>> index uniqueMember pres
>> #index sudoUser eq,sub
>>
>> overlay syncprov
>> syncprov-checkpoint 100 10
>> syncprov-sessionlog 100
>> loglevel 1
>>
>> # Performance tuning directives
>> sizelimit 5000
>> threads 16
>> idletimeout 14400
>> cachesize 10000
>> checkpoint 256 15
>> password-hash {SSHA}
>>
>>
>> Replica slapd.conf file:
>>
>> nclude /etc/openldap/schema/cosine.schema
>> include /etc/openldap/schema/nis.schema
>> include /etc/openldap/schema/inetorgperson.schema
>> include /etc/openldap/schema/solaris.schema
>> include /etc/openldap/schema/sudo.schema
>> include /etc/openldap/schema/dyngroup.schema
>> include /etc/openldap/schema/misc.schema
>> include /etc/openldap/schema/openldap.schema
>> include /etc/openldap/schema/duaconf.schema
>>
>> pidfile         /var/run/openldap/slapd.pid
>> argsfile        /var/run/openldap/slapd.args
>>
>> # DATABASE CONFIG
>> database bdb
>> suffix "dc=domain,dc=com"
>> rootdn "cn=Manager,dc=domain,dc=com"
>> rootpw {SSHAsomehash
>>
>> index entryCSN,entryUUID eq,pres
>> directory       /var/lib/ldap
>> loglevel 256
>>
>> syncrepl      rid=001
>>                      provider=ldap://10.10.1.91
>>                      interval=00:05:00:00
>>                      bindmethod=simple
>>                      scope=sub
>>                      filter="(objectClass=organizationalPerson)"
>>                      attrs="cn,sn,ou,telephoneNumber,title,l"
>>                      searchbase="dc=matlock,dc=com"
>>                      binddn="cn=proxyagent,ou=profile,dc=matlock,dc=com"
>>                      credentials="abc123"
>>                      schemachecking=off
>>                      type=refreshAndPersist
>>                      retry="60 +"
>>
>>
>>
>> # Indices to maintain
>>
>> index objectClass,uid,uidNumber,gidNumber,ou eq
>> index cn,mail,surname,givenname eq,subinitial
>> index memberUid eq
>> index nisDomain eq
>> index uniqueMember pres
>>
>>
>> # Performance tuning directives
>> sizelimit 5000
>> threads 16
>> idletimeout 14400
>> cachesize 10000
>> checkpoint 256 15
>> password-hash {SSHA}
>>
>>
>> On Mon, Nov 9, 2009 at 3:16 AM, Jonathan Clarke<jonathan@phillipoux.net>
>>  wrote:
>>>
>>> On 06/11/2009 15:31, Scott Behrens wrote:
>>>>
>>>> Hello,
>>>>
>>>> I am running 2 hosts CentOS release 5.3 (Final) with kernel
>>>> 2.6.18-164.el5l.
>>>>
>>>> One box is the provider and the other is a consumer using syncrepl.
>>>> When the primary goes down, I can perform searches against the domain
>>>> from the replica so I know the replica has replicated the data (when I
>>>> stop both hosts I can't preform the search obviously).  When I point a
>>>> client to the replica, I get a timeout when trying to perform searches
>>>> or pull data.  ldap_bind: Can't contact LDAP server (-1).
>>>>
>>>> Any ideas?  I will provide more information if needed.
>>>
>>> Hi,
>>>
>>> Which client are you pointing to the replica? ldapsearch? And what
>>> version
>>> of OpenLDAP are you running?
>>>
>>> This is a pretty vague error, many things could be the cause: bad client
>>> setup, network problems, server problems...
>>>
>>> I suggest you check out the logs from the server you're trying to connect
>>> to, having previously set them to include "stats" (level 256).
>>>
>>> Hope this helps,
>>> Jonathan
>>>
>
>

Follow-Ups:
- Re: ldap clients cant connect to replica after failure
  - From: Buchan Milne <bgmilne@staff.telkomsa.net>

References:
- ldap clients cant connect to replica after failure
  - From: Scott Behrens <sbehrens@gmail.com>
- Re: ldap clients cant connect to replica after failure
  - From: Jonathan Clarke <jonathan@phillipoux.net>
- Re: ldap clients cant connect to replica after failure
  - From: Scott Behrens <sbehrens@gmail.com>
- Re: ldap clients cant connect to replica after failure
  - From: Jonathan Clarke <jonathan@phillipoux.net>

Prev by Date: Re: unable to user evolutionperson.schema
Next by Date: Re: unable to user evolutionperson.schema
Index(es):
- Chronological
- Thread