[Date Prev][Date Next] [Chronological] [Thread] [Top]

Syncrepl unregularly stops on slaves, leaving DB in inconsistent state



Hello OpenLDAP users,

I have a Syncrepl setup with one master server and seven slaves. 
The slaves are mail servers running Postfix, SpamAssassin and Amavis as LDAP clients and have a relatively heavy load.

Every two weeks or so (not regularly) the Syncrepl stops on some of the slaves are stopping; there are no Syncrepl requests from the slaves any more.

Restarting the Slapd on the slaves fixes the problem in most cases, but sometimes some entries are not replicated until I modify them manually on the master. After that, it works fine again.

My OpenLDAP version is 2.4.23 running on SunOS 5.10 Generic_139555-08 sun4v sparc SUNW,Sun-Fire-T1000 Solaris. The servers that are affected more often are running in non-global zone.

Any ideas would be helpful.

Thanks in advance,

Karsten Kroesch
____________________________
Internet Application Engineer
Applications Operations

karsten.kroesch@swisscom.com
____________________________
Swisscom (Schweiz) AG
Corporate Business Unit
Müllerstrasse 16
8004 Zürich
____________________________


-------8<---------------------------------------

Affected entries, log files and configuration see below:

#
# On the master:

# ldapsearch mail=mthudianplackal@[domain-deleted].ch
# extended LDIF
#
# LDAPv3
# base <dc=ip-plus, dc=net> (default) with scope subtree
# filter: mail=mthudianplackal@[domain-deleted].ch
# requesting: ALL
#

# mthudianplackal@[domain-deleted].ch, [domain-deleted].ch, vsf, ip-plus.net
dn: mail=mthudianplackal@[domain-deleted].ch,dc=[domain-deleted].ch,ou=vsf,dc=ip-plus,
 dc=net
objectClass: top
objectClass: mailObject
objectClass: amavisAccount
mail: mthudianplackal@[domain-deleted].ch

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1


# On some of the slaves:

$ ldapsearch mail=mthudianplackal@[domain-deleted].ch
# extended LDIF
#
# LDAPv3
# base <dc=ip-plus, dc=net> (default) with scope subtree
# filter: mail=mthudianplackal@[domain-deleted].ch
# requesting: ALL
#

# search result
search: 2
result: 0 Success

# numResponses: 1


Log files at the time, the entries were made:

May 16 11:56:20 v-vsf4 slapd[14302]: [ID 538834 local4.debug] daemon: select: listen=7 active_threads=0 tvp=zero
May 16 11:56:20 v-vsf4 slapd[14302]: [ID 538834 local4.debug] daemon: select: listen=8 active_threads=0 tvp=zero
May 16 11:56:31 v-vsf4 slapd[14302]: [ID 538834 local4.debug] daemon: select: listen=7 active_threads=0 tvp=zero
May 16 11:56:31 v-vsf4 slapd[14302]: [ID 538834 local4.debug] daemon: select: listen=8 active_threads=0 tvp=zero
May 16 11:56:31 v-vsf4 slapd[14302]: [ID 365351 local4.debug] do_syncrep2: rid=000 LDAP_RES_SEARCH_RESULT

# 15 Seconds no action -- unusual on a server with heavy load.


May 16 11:56:46 v-vsf4 slapd[14302]: [ID 538834 local4.debug] daemon: select: listen=7 active_threads=0 tvp=zero
May 16 11:56:46 v-vsf4 slapd[14302]: [ID 538834 local4.debug] daemon: select: listen=8 active_threads=0 tvp=zero
May 16 11:56:46 v-vsf4 slapd[14302]: [ID 977386 local4.debug] syncrepl_entry: rid=000 LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_ADD)
May 16 11:56:46 v-vsf4 slapd[14302]: [ID 580501 local4.debug] syncrepl_entry: rid=000 inserted UUID a36b3802-525a-1032-9442-17888436c71f
May 16 11:56:46 v-vsf4 slapd[14302]: [ID 565591 local4.debug] syncrepl_entry: rid=000 be_search (0)
May 16 11:56:46 v-vsf4 slapd[14302]: [ID 709484 local4.debug] syncrepl_entry: rid=000 mail=mthudianplackal@[domain-deleted].ch,dc=[domain-deleted].ch,ou=vsf,dc=ip-plus,dc=net
May 16 11:56:48 v-vsf4 slapd[14302]: [ID 601841 local4.debug] daemon: activity on 1 descriptor
May 16 11:56:48 v-vsf4 slapd[14302]: [ID 538834 local4.debug] daemon: select: listen=7 active_threads=0 tvp=zero
May 16 11:56:48 v-vsf4 slapd[14302]: [ID 300852 local4.debug] daemon: listen=8, new connection on 91
May 16 11:56:48 v-vsf4 slapd[14302]: [ID 538834 local4.debug] daemon: select: listen=8 active_threads=0 tvp=zero
May 16 11:56:48 v-vsf4 slapd[14302]: [ID 368480 local4.debug] daemon: added 91r (active) listener=0
May 16 11:56:48 v-vsf4 slapd[14302]: [ID 848112 local4.debug] conn=35253 fd=91 ACCEPT from IP=192.168.1.4:45922 (IP=0.0.0.0:389)
May 16 11:56:48 v-vsf4 slapd[14302]: [ID 601841 local4.debug] daemon: activity on 1 descriptor
May 16 11:56:48 v-vsf4 slapd[14302]: [ID 609413 local4.debug] daemon: waked
May 16 11:56:48 v-vsf4 slapd[14302]: [ID 538834 local4.debug] daemon: select: listen=7 active_threads=0 tvp=zero
May 16 11:56:48 v-vsf4 slapd[14302]: [ID 538834 local4.debug] daemon: select: listen=8 active_threads=0 tvp=zero
May 16 11:56:48 v-vsf4 slapd[14302]: [ID 601841 local4.debug] daemon: activity on 1 descriptor
May 16 11:56:48 v-vsf4 slapd[14302]: [ID 802679 local4.debug] daemon: activity on:
May 16 11:56:48 v-vsf4 slapd[14302]: [ID 522297 local4.debug]  91r
May 16 11:56:48 v-vsf4 slapd[14302]: [ID 100000 local4.debug] 
May 16 11:56:48 v-vsf4 slapd[14302]: [ID 694296 local4.debug] daemon: read activity on 91
May 16 11:56:48 v-vsf4 slapd[14302]: [ID 538834 local4.debug] daemon: select: listen=7 active_threads=0 tvp=zero
May 16 11:56:48 v-vsf4 slapd[14302]: [ID 215403 local4.debug] conn=35253 op=0 BIND dn="" method=128
May 16 11:56:48 v-vsf4 slapd[14302]: [ID 538834 local4.debug] daemon: select: listen=8 active_threads=0 tvp=zero

May 17 08:43:18 v-vsf4 slapd[14302]: [ID 515743 local4.debug] syncrepl_entry: rid=000 be_add mail=mthudianplackal@[domain-deleted].ch,dc=[domain-deleted].ch,ou=vsf,dc=ip-plus,dc=net (0)
May 17 08:43:34 v-vsf4 slapd[3312]: [ID 709484 local4.debug] syncrepl_entry: rid=000 mail=mthudianplackal@[domain-deleted].ch,dc=[domain-deleted].ch,ou=vsf,dc=ip-plus,dc=net
May 17 08:43:34 v-vsf4 slapd[3312]: [ID 515743 local4.debug] syncrepl_entry: rid=000 be_add mail=mthudianplackal@[domain-deleted].ch,dc=[domain-deleted].ch,ou=vsf,dc=ip-plus,dc=net (68)
May 17 08:43:34 v-vsf4 slapd[3312]: [ID 933660 local4.debug] syncrepl_entry: rid=000 be_modify mail=mthudianplackal@[domain-deleted].ch,dc=[domain-deleted].ch,ou=vsf,dc=ip-plus,dc=net (0)
May 17 08:43:47 v-vsf4 slapd[3312]: [ID 338579 local4.debug] nonpresent_callback: rid=000 nonpresent UUID a36b3802-525a-1032-9442-17888436c71f, dn mail=mthudianplackal@[domain-deleted].ch,dc=[domain-deleted].ch,ou=vsf,dc=ip-plus,dc=net
May 17 08:43:48 v-vsf4 slapd[3312]: [ID 905397 local4.debug] syncrepl_del_nonpresent: rid=000 be_delete mail=mthudianplackal@[domain-deleted].ch,dc=[domain-deleted].ch,ou=vsf,dc=ip-plus,dc=net (0)
May 17 10:11:05 v-vsf4 slapd[3312]: [ID 469902 local4.debug] conn=1480 op=1 SRCH base="dc=ip-plus,dc=net" scope=2 deref=0 filter="(mail=mthudianplackal@[domain-deleted].ch)"
May 17 10:39:39 v-vsf4 slapd[3312]: [ID 469902 local4.debug] conn=1595 op=1 SRCH base="dc=ip-plus,dc=net" scope=2 deref=0 filter="(mail=mthudianplackal@[domain-deleted].ch)"
May 17 10:41:15 v-vsf4 slapd[3312]: [ID 469902 local4.debug] conn=1599 op=1 SRCH base="dc=ip-plus,dc=net" scope=2 deref=0 filter="(mail=mthudianplackal@[domain-deleted].ch)"
May 17 10:41:22 v-vsf4 slapd[3312]: [ID 709484 local4.debug] syncrepl_entry: rid=000 mail=mthudianplackal@[domain-deleted].ch,dc=[domain-deleted].ch,ou=vsf,dc=ip-plus,dc=net
May 17 10:41:22 v-vsf4 slapd[3312]: [ID 515743 local4.debug] syncrepl_entry: rid=000 be_add mail=mthudianplackal@[domain-deleted].ch,dc=[domain-deleted].ch,ou=vsf,dc=ip-plus,dc=net (0)
May 17 10:41:37 v-vsf4 slapd[3312]: [ID 469902 local4.debug] conn=1601 op=1 SRCH base="dc=ip-plus,dc=net" scope=2 deref=0 filter="(mail=mthudianplackal@[domain-deleted].ch)"
May 17 10:41:37 v-vsf4 slapd[3312]: [ID 580335 local4.debug] conn=1601 op=1 ENTRY dn="mail=mthudianplackal@[domain-deleted].ch,dc=[domain-deleted].ch,ou=vsf,dc=ip-plus,dc=net"



Master configuration:

# See slapd.conf(5) for details on configuration options.
# This file should NOT be world readable.
#
include         /etc/openldap/schema/core.schema
include         /etc/openldap/schema/cosine.schema
include         /etc/openldap/schema/inetorgperson.schema
include         /etc/openldap/schema/nis.schema
include         /etc/openldap/schema/openldap.schema
include         /etc/openldap/schema/amavisd-new.schema
include         /etc/openldap/schema/ipplus.schema



pidfile         /var/run/slapd.pid
argsfile        /var/run/slapd.args

# allow ldap protocol v2
allow bind_v2

# debug level
loglevel        256

#######################################################################
# ldbm database definitions
#######################################################################

database        bdb
suffix          "dc=ip-plus,dc=net"
rootdn          "cn=root,dc=ip-plus,dc=net"
# Cleartext passwords, especially for the rootdn, should
# be avoid.  See slappasswd(8) and slapd.conf(5) for details.
# Use of strong authentication encouraged.
rootpw          swisscom
# The database directory MUST exist prior to running slapd AND 
# should only be accessible by the slapd/tools. Mode 700 recommended.
directory       /var/openldap-data
# Indices to maintain
index objectclass,entryCSN,entryUUID eq
index   dc,cn,mail      eq

#######################################################################
# replication
#######################################################################

overlay syncprov
syncprov-checkpoint 100 10
syncprov-sessionlog 100


On the slaves, the config looks like:

[ ... same as above, execpt replication: ]

#######################################################################
# replication
#######################################################################

syncrepl rid=0
        provider=ldap://v-ldapmaster-lan:389
        type=refreshOnly
        interval=00:00:00:15
        searchbase="dc=ip-plus,dc=net"
        filter="(objectClass=*)"
        scope=sub
        attrs="*"
        bindmethod=simple
        binddn="cn=root,dc=ip-plus,dc=net"
        credentials=swisscom
        schemachecking=off
        retry="5 +"







Attachment: smime.p7s
Description: S/MIME cryptographic signature