[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: syncrepl and mass deletes

To: openldap-technical@openldap.org
Subject: Re: syncrepl and mass deletes
From: LALOT Dominique <dom.lalot@gmail.com>
Date: Wed, 23 Mar 2011 15:47:47 +0100
Dkim-signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=4BI68cds7wJ/YZMXMmTQ8hiu22sMwLIp7g1uXuUULOo=; b=Q6Zf5VqdU07xr6ANLsB2sS/HrcX7kK0s/K2vkxZHWAYCGi9oiGetU/2Uq77FZvMiR2 djuTTRNH2x++OsOmuLyVtvKoQ5irN0M1DqUcmpvl1INJ6KjdPkyqiHpP6fZfFm76qdTy fdY2gq0c+9qM8GCce5fDcNcQrVKWRV98h3Emo=
Domainkey-signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=bn3qEz5b/jdZ3tPClrWIqAlktVim4KLAeV2CIPfDZtgli+TuhsQpCGevFAw3fglN+l zas/CR9kGQf19WXo5BBvGA3Zi29gmRMEhhtClsNs/KpuG72RgP2UoReNcuvO24fP58ub BRlST4w2AqtrrE5b9+1l/+f8lQxele1lQs75o=
In-reply-to: <AANLkTi=XkYa9gY-sgEpn2nX-sOAnwLzeXz5ETv8y7iNs@mail.gmail.com>
References: <AANLkTi=XkYa9gY-sgEpn2nX-sOAnwLzeXz5ETv8y7iNs@mail.gmail.com>

I have to mention that openldap 2.4.23 is an ubuntu package and compiled with db4.8.30

I did some other tries:
using just a standard replication (one provider) it works. I stopped the consumer during the mass delete and start again, and it's synced

I noticed some messages:
Mar 23 15:36:14 ldaprelay slapd[24616]: syncrepl_del_nonpresent: rid=020 be_delete uid=t606435,ou=people,dc=univ-yy,dc=fr (0)

I then restarted again with two providers, but modify two things. To prevent a warning I added a memberof overlay, modify the url to add a trailing /, the stopped that server during mass deletes and it stays like this:

Mar 23 15:41:39 ldaprelay slapd[24632]: @(#) $OpenLDAP: slapd 2.4.23 (Nov 19 2010 17:41:28) $#012#011buildd@allspice:/build/buildd/openldap-2.4.23/debian/build/servers/slapd
Mar 23 15:41:39 ldaprelay slapd[24634]: slapd starting
Mar 23 15:41:39 ldaprelay slapd[24634]: do_syncrep2: rid=040 LDAP_RES_INTERMEDIATE - REFRESH_DELETE
Mar 23 15:41:39 ldaprelay slapd[24634]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - SYNC_ID_SET
Mar 23 15:41:42 ldaprelay slapd[24634]: last message repeated 175 times
Mar 23 15:41:42 ldaprelay slapd[24634]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - REFRESH_PRESENT
Mar 23 15:41:42 ldaprelay slapd[24634]: do_syncrep2: rid=020 cookie=rid=020,sid=020,csn=20110323144119.644587Z#000000#020#000000
Mar 23 15:41:42 ldaprelay slapd[24634]: slap_queue_csn: queing 0x7f674545b4b0 20110323144119.644587Z#000000#020#000000
Mar 23 15:41:42 ldaprelay slapd[24634]: slap_graduate_commit_csn: removing 0x7f674545b570 20110323144119.644587Z#000000#020#000000

In this case, still 14000 entries not deleted. I don't understand what to do. One provider, it's working, two providers no. The second one has just 10 entries and no traffic.
So is me setup OK, should we add something on the provider side?

Thanks

Dom

2011/3/23 LALOT Dominique <dom.lalot@gmail.com>

Hello,

I am testing the replication feature in a multimaster environment replicating into a single database. As stated before, I added serverid to my providers. I just have two providers for test purpose.
I tested mass updates on a provider, stopped my replica during updates, then start again and it's OK, it updates the entries
If I do the same for mass deletes. I deleted 40000 entries while stopping the consumer. My consumer is still with 30000 undeleted entries. I left the consumer for hours, restarting it twice.
It seems there is no regular compare between consumer or provider in such situation. I'll simplify to test in a single provider setup, to see if it works.

All servers are 2.4.23

consumer side. A script is populating the DIT via slapadd first to create the dc entries.

database        bdb
suffix      "dc=fr"

serverid 0x40 ldap://ldapmaitre.univ-yy.fr
syncrepl rid=040
    provider=ldap://ldapmaitre.univ-yy.fr/
    type=refreshAndPersist
    searchbase="dc=univ-yy,dc=fr"
    retry="60 10 300 +"
    scope=sub
    filter="(objectClass=*)"
    attrs="*,+"
    schemachecking=off
    bindmethod=simple

serverid 0x20 ldap://anutest.univ-xx.fr
syncrepl rid=020
    provider=ldap://anutest.univ-xx.fr/
    type=refreshAndPersist
    searchbase="dc=univ-xx,dc=fr"
    retry="60 10 300 +"
    scope=sub
    filter="(objectClass=*)"
    attrs="*,+"
    schemachecking=off
    bindmethod=simple

Mar 23 13:29:11 ldaprelay slapd[24319]: @(#) $OpenLDAP: slapd 2.4.23 (Nov 19 2010 17:41:28) $#012#011buildd@allspice:/build/buildd/openldap-2.4.23/debian/build/servers/slapd
Mar 23 13:29:11 ldaprelay slapd[24320]: slapd starting
Mar 23 13:29:11 ldaprelay slapd[24320]: do_syncrep2: rid=040 LDAP_RES_INTERMEDIATE - REFRESH_DELETE
Mar 23 13:29:11 ldaprelay slapd[24320]: do_syncrep2: rid=020 LDAP_RES_INTERMEDIATE - REFRESH_DELETE

root@ldaprelay:/etc/ldap# ldapsearch -x -LLL -s base contextcsn
dn: dc=fr
contextCSN: 20110323101636.221613Z#000000#040#000000
contextCSN: 20110323100346.844382Z#000000#020#000000

Thanks for any hints

Dom

--
Dominique LALOT
Ingénieur Systèmes et Réseaux
http://annuaire.univmed.fr/showuser.php?uid=lalot

--
Dominique LALOT
Ingénieur Systèmes et Réseaux
http://annuaire.univmed.fr/showuser.php?uid=lalot

References:
- syncrepl and mass deletes
  - From: LALOT Dominique <dom.lalot@gmail.com>

Prev by Date: syncrepl and mass deletes
Next by Date: Re: syncrepl and mass deletes
Index(es):
- Chronological
- Thread