OpenLDAP 2.4.15 : CSN too old when using 4-way multimaster

[Adrien Futschik <adrien.futschik@atosorigin.com>]

Hello everyone,

I have been testing n-way multimaster replication with OpenLDAP for a while 
now (from 2.4.11, to 2.4.15) and just when I though that everything was 
working perfectly, I dicided to test N-way multimaster not only with 2 masters 
on different servers, but with 4 !

2 OpenLDAP instances per server.

I have been configuring syncprov and syncrepl accordingly :
olcServerID: 1 ldap://163.106.38.90:9011/
olcServerID: 2 ldap://163.106.38.92:9012/
olcServerID: 3 ldap://163.106.38.90:9013/
olcServerID: 4 ldap://163.106.38.92:9014/

olcSyncrepl: {0}rid=011 provider=ldap://163.106.38.90:9011/ binddn="cn=admin,c
 =fr" bindmethod=simple credentials=secret searchbase="c=fr" type=refreshAndPe
 rsist retry="5 5 300 12 3600 +" timeout=3
olcSyncrepl: {1}rid=012 provider=ldap://163.106.38.92:9012/ binddn="cn=admin,c
 =fr" bindmethod=simple credentials=secret searchbase="c=fr" type=refreshAndPe
 rsist retry="5 5 300 12 3600 +" timeout=3
olcSyncrepl: {2}rid=013 provider=ldap://163.106.38.90:9013/ binddn="cn=admin,c
 =fr" bindmethod=simple credentials=secret searchbase="c=fr" type=refreshAndPe
 rsist retry="5 5 300 12 3600 +" timeout=3
olcSyncrepl: {3}rid=014 provider=ldap://163.106.38.92:9014/ binddn="cn=admin,c
 =fr" bindmethod=simple credentials=secret searchbase="c=fr" type=refreshAndPe
 rsist retry="5 5 300 12 3600 +" timeout=3

I am starting with all instances synced and I am trying to add entries en all 
four instances (in //). If I do so, I have a few entries that are not 
replicated on the others. I am getting this kind of messages :

do_syncrep2: 
cookie=rid=011,sid=002,csn=20090227130003.849482Z#000000#004#000000
do_syncrep2: rid=011 CSN too old, ignoring 
20090227130003.849482Z#000000#004#000000
do_syncrep2: 
cookie=rid=013,sid=002,csn=20090227130003.849482Z#000000#004#000000
do_syncrep2: rid=013 CSN too old, ignoring 
20090227130003.849482Z#000000#004#000000
do_syncrep2: 
cookie=rid=014,sid=002,csn=20090227130003.946474Z#000000#004#000000

Did someone face the same issue ? 

Here is my configuration : (I am using refreshAndPersist mode for both cn=config 
and olcDatabase={1}bdb)

M1 on IP1 / PORT1 :
dn: cn=config
objectClass: olcGlobal
cn: config
structuralObjectClass: olcGlobal
creatorsName: cn=config
olcServerID: 1 ldap://163.106.38.90:9011/
olcServerID: 2 ldap://163.106.38.92:9012/
olcServerID: 3 ldap://163.106.38.90:9013/
olcServerID: 4 ldap://163.106.38.92:9014/
entryUUID: ef89c876-adb3-4dc7-aa7d-024bbc359c98
createTimestamp: 20090227085748Z
entryCSN: 20090227085749.920499Z#000000#004#000000
modifiersName: cn=config
modifyTimestamp: 20090227085749Z
contextCSN: 20090227085752.833630Z#000000#001#000000

dn: olcDatabase={1}bdb
objectClass: olcDatabaseConfig
objectClass: olcBdbConfig
olcDatabase: {1}bdb
olcDbDirectory: ./openldap-data
olcSuffix: c=fr
olcRootDN: cn=admin,c=fr
olcRootPW:: e1NTSEF9WVZNSHJtYTRvUGd4KzFoak9kYWhBcm5NVHJxU1Zmdno=
olcSizeLimit: 100
olcSyncrepl: {0}rid=011 provider=ldap://163.106.38.90:9011/ binddn="cn=admin,c
 =fr" bindmethod=simple credentials=secret searchbase="c=fr" type=refreshAndPe
 rsist retry="5 5 300 12 3600 +" timeout=3
olcSyncrepl: {1}rid=012 provider=ldap://163.106.38.92:9012/ binddn="cn=admin,c
 =fr" bindmethod=simple credentials=secret searchbase="c=fr" type=refreshAndPe
 rsist retry="5 5 300 12 3600 +" timeout=3
olcSyncrepl: {2}rid=013 provider=ldap://163.106.38.90:9013/ binddn="cn=admin,c
 =fr" bindmethod=simple credentials=secret searchbase="c=fr" type=refreshAndPe
 rsist retry="5 5 300 12 3600 +" timeout=3
olcSyncrepl: {3}rid=014 provider=ldap://163.106.38.92:9014/ binddn="cn=admin,c
 =fr" bindmethod=simple credentials=secret searchbase="c=fr" type=refreshAndPe
 rsist retry="5 5 300 12 3600 +" timeout=3
olcTimeLimit: 600
olcMirrorMode: TRUE
olcDbCacheSize: 2000
olcDbCheckpoint: 2000 10
olcDbIndex: default pres,eq
olcDbIndex: cn,sn pres,eq,sub
olcDbIndex: objectClass,entryCSN,entryUUID eq
structuralObjectClass: olcBdbConfig
entryUUID: 00c01e5d-69ee-4baa-8e5a-4ef609dfd958
creatorsName: cn=config
createTimestamp: 20090227085752Z
entryCSN: 20090227085752.729899Z#000000#001#000000
modifiersName: cn=config
modifyTimestamp: 20090227085752Z

M2 on IP2 / PORT2 :
dn: cn=config
objectClass: olcGlobal
cn: config
structuralObjectClass: olcGlobal
entryUUID: 8da75037-65e6-4375-8c21-7e5c0194a60b
creatorsName: cn=config
createTimestamp: 20090227085723Z
olcServerID: 1 ldap://163.106.38.90:9011/
olcServerID: 2 ldap://163.106.38.92:9012/
olcServerID: 3 ldap://163.106.38.90:9013/
olcServerID: 4 ldap://163.106.38.92:9014/
entryCSN: 20090227085725.003182Z#000000#002#000000
modifiersName: cn=config
modifyTimestamp: 20090227085725Z
contextCSN: 20090227085752.833630Z#000000#001#000000

dn: olcDatabase={1}bdb
objectClass: olcDatabaseConfig
objectClass: olcBdbConfig
olcDatabase: {1}bdb
olcDbDirectory: ./openldap-data
olcSuffix: c=fr
olcRootDN: cn=admin,c=fr
olcRootPW:: e1NTSEF9WVZNSHJtYTRvUGd4KzFoak9kYWhBcm5NVHJxU1Zmdno=
olcSizeLimit: 100
olcSyncrepl: {0}rid=011 provider=ldap://163.106.38.90:9011/ binddn="cn=admin,c
 =fr" bindmethod=simple credentials=secret searchbase="c=fr" type=refreshAndPe
 rsist retry="5 5 300 12 3600 +" timeout=3
olcSyncrepl: {1}rid=012 provider=ldap://163.106.38.92:9012/ binddn="cn=admin,c
 =fr" bindmethod=simple credentials=secret searchbase="c=fr" type=refreshAndPe
 rsist retry="5 5 300 12 3600 +" timeout=3
olcSyncrepl: {2}rid=013 provider=ldap://163.106.38.90:9013/ binddn="cn=admin,c
 =fr" bindmethod=simple credentials=secret searchbase="c=fr" type=refreshAndPe
 rsist retry="5 5 300 12 3600 +" timeout=3
olcSyncrepl: {3}rid=014 provider=ldap://163.106.38.92:9014/ binddn="cn=admin,c
 =fr" bindmethod=simple credentials=secret searchbase="c=fr" type=refreshAndPe
 rsist retry="5 5 300 12 3600 +" timeout=3
olcTimeLimit: 600
olcMirrorMode: TRUE
olcDbCacheSize: 2000
olcDbCheckpoint: 2000 10
olcDbIndex: default pres,eq
olcDbIndex: cn,sn pres,eq,sub
olcDbIndex: objectClass,entryCSN,entryUUID eq
structuralObjectClass: olcBdbConfig
entryUUID: 00c01e5d-69ee-4baa-8e5a-4ef609dfd958
creatorsName: cn=config
createTimestamp: 20090227085752Z
entryCSN: 20090227085752.729899Z#000000#001#000000
modifiersName: cn=config
modifyTimestamp: 20090227085752Z

M3 on IP1 / PORT3 :
dn: cn=config
objectClass: olcGlobal
cn: config
structuralObjectClass: olcGlobal
entryUUID: cf068647-318f-4848-9c72-9c7745a8a4b3
creatorsName: cn=config
createTimestamp: 20090227085742Z
olcServerID: 1 ldap://163.106.38.90:9011/
olcServerID: 2 ldap://163.106.38.92:9012/
olcServerID: 3 ldap://163.106.38.90:9013/
olcServerID: 4 ldap://163.106.38.92:9014/
entryCSN: 20090227085743.825685Z#000000#003#000000
modifiersName: cn=config
modifyTimestamp: 20090227085743Z
contextCSN: 20090227085752.833630Z#000000#001#000000

dn: olcDatabase={1}bdb
objectClass: olcDatabaseConfig
objectClass: olcBdbConfig
olcDatabase: {1}bdb
olcDbDirectory: ./openldap-data
olcSuffix: c=fr
olcRootDN: cn=admin,c=fr
olcRootPW:: e1NTSEF9WVZNSHJtYTRvUGd4KzFoak9kYWhBcm5NVHJxU1Zmdno=
olcSizeLimit: 100
olcSyncrepl: {0}rid=011 provider=ldap://163.106.38.90:9011/ binddn="cn=admin,c
 =fr" bindmethod=simple credentials=secret searchbase="c=fr" type=refreshAndPe
 rsist retry="5 5 300 12 3600 +" timeout=3
olcSyncrepl: {1}rid=012 provider=ldap://163.106.38.92:9012/ binddn="cn=admin,c
 =fr" bindmethod=simple credentials=secret searchbase="c=fr" type=refreshAndPe
 rsist retry="5 5 300 12 3600 +" timeout=3
olcSyncrepl: {2}rid=013 provider=ldap://163.106.38.90:9013/ binddn="cn=admin,c
 =fr" bindmethod=simple credentials=secret searchbase="c=fr" type=refreshAndPe
 rsist retry="5 5 300 12 3600 +" timeout=3
olcSyncrepl: {3}rid=014 provider=ldap://163.106.38.92:9014/ binddn="cn=admin,c
 =fr" bindmethod=simple credentials=secret searchbase="c=fr" type=refreshAndPe
 rsist retry="5 5 300 12 3600 +" timeout=3
olcTimeLimit: 600
olcMirrorMode: TRUE
olcDbCacheSize: 2000
olcDbCheckpoint: 2000 10
olcDbIndex: default pres,eq
olcDbIndex: cn,sn pres,eq,sub
olcDbIndex: objectClass,entryCSN,entryUUID eq
structuralObjectClass: olcBdbConfig
entryUUID: 00c01e5d-69ee-4baa-8e5a-4ef609dfd958
creatorsName: cn=config
createTimestamp: 20090227085752Z
entryCSN: 20090227085752.729899Z#000000#001#000000
modifiersName: cn=config
modifyTimestamp: 20090227085752Z

M4 on IP2 / PORT4 :
dn: cn=config
objectClass: olcGlobal
cn: config
structuralObjectClass: olcGlobal
entryUUID: ef89c876-adb3-4dc7-aa7d-024bbc359c98
creatorsName: cn=config
createTimestamp: 20090227085748Z
olcServerID: 1 ldap://163.106.38.90:9011/
olcServerID: 2 ldap://163.106.38.92:9012/
olcServerID: 3 ldap://163.106.38.90:9013/
olcServerID: 4 ldap://163.106.38.92:9014/
entryCSN: 20090227085749.920499Z#000000#004#000000
modifiersName: cn=config
modifyTimestamp: 20090227085749Z
contextCSN: 20090227085752.833630Z#000000#001#000000

dn: olcDatabase={1}bdb
objectClass: olcDatabaseConfig
objectClass: olcBdbConfig
olcDatabase: {1}bdb
olcDbDirectory: ./openldap-data
olcSuffix: c=fr
olcRootDN: cn=admin,c=fr
olcRootPW:: e1NTSEF9WVZNSHJtYTRvUGd4KzFoak9kYWhBcm5NVHJxU1Zmdno=
olcSizeLimit: 100
olcSyncrepl: {0}rid=011 provider=ldap://163.106.38.90:9011/ binddn="cn=admin,c
 =fr" bindmethod=simple credentials=secret searchbase="c=fr" type=refreshAndPe
 rsist retry="5 5 300 12 3600 +" timeout=3
olcSyncrepl: {1}rid=012 provider=ldap://163.106.38.92:9012/ binddn="cn=admin,c
 =fr" bindmethod=simple credentials=secret searchbase="c=fr" type=refreshAndPe
 rsist retry="5 5 300 12 3600 +" timeout=3
olcSyncrepl: {2}rid=013 provider=ldap://163.106.38.90:9013/ binddn="cn=admin,c
 =fr" bindmethod=simple credentials=secret searchbase="c=fr" type=refreshAndPe
 rsist retry="5 5 300 12 3600 +" timeout=3
olcSyncrepl: {3}rid=014 provider=ldap://163.106.38.92:9014/ binddn="cn=admin,c
 =fr" bindmethod=simple credentials=secret searchbase="c=fr" type=refreshAndPe
 rsist retry="5 5 300 12 3600 +" timeout=3
olcTimeLimit: 600
olcMirrorMode: TRUE
olcDbCacheSize: 2000
olcDbCheckpoint: 2000 10
olcDbIndex: default pres,eq
olcDbIndex: cn,sn pres,eq,sub
olcDbIndex: objectClass,entryCSN,entryUUID eq
structuralObjectClass: olcBdbConfig
entryUUID: 00c01e5d-69ee-4baa-8e5a-4ef609dfd958
creatorsName: cn=config
createTimestamp: 20090227085752Z
entryCSN: 20090227085752.729899Z#000000#001#000000
modifiersName: cn=config
modifyTimestamp: 20090227085752Z


I also have to say that if I stop M3 & M4 and try to add entries to M1 & M2, I 
don't get the "CSN too old" messages and all entries a replicated correctly 
!!!

Is this because Multimaster is limited to 2-3 instances ? The reason why I am 
trying to build a 4-way master architecture is because I would like to be able 
to stop one server and perform a slapcat, even if one "physical" server is 
down...

Thanks very much for your consideration & time..

Adrien Futschik