New consumer replication issues

[Christopher Strider Cook <ccook@pandora.com>]

I've got a mirrormode cluster of 2.4.25 running and using test059 as an 
example I had setup a script that would create new consumers of this 
cluster. It worked fine.. until now. Now when I run it I seem to run 
into, what I can only describe as, an ordering problem with the initial 
replication. The server is replicating parts of cn=config over but seems 
to complain because of an syntax check. Am I missing some flag in the 
sync or do entyCSNs need to be in a particular order...or am I missing 
something else altogether?

Hear are some logs and configs

In this case it looks like a needed schema wasn't synced first:

May 26 15:02:49 ramones slapd[10377]: syncrepl_message_to_entry: rid=001 
mods check (olcSuffix: value #0 invalid per syntax)
May 26 15:02:49 ramones slapd[10377]: do_syncrepl: rid=001 rc 21 retrying

And here I tried to sync the schema specifically and it errors on not 
understanding the creatorsName, because the database that contains it 
hasn't been synced yet:

May 26 14:58:10 ramones slapd[7260]: syncrepl_message_to_entry: rid=001 
mods check (creatorsName: value #0 invalid per syntax)
May 26 14:58:10 ramones slapd[7260]: do_syncrepl: rid=001 rc 21 retrying


Here is a debug:

==> rewrite_context_apply [depth=1] 
string='olcDatabase={1}hdb,cn=config,cn=slave'
==> rewrite_rule_apply rule='(.*)cn=config,cn=slave$' 
string='olcDatabase={1}hdb,cn=config,cn=slave' [1 pass(es)]
==> rewrite_context_apply [depth=1] res={0,'olcDatabase={1}hdb,cn=config'}
ber_scanf fmt ({mW}) ber:
ber_dump: buf=0xb155b0 ptr=0xb16005 end=0xb16046 len=65
  0000:  30 23 04 11 73 75 62 73  63 68 65 6d 61 53 75 62   
0#..subschemaSub
  0010:  65 6e 74 72 79 31 0e 04  0c 63 6e 3d 53 75 62 73   
entry1...cn=Subs
  0020:  63 68 65 6d 61 30 1a 04  0f 68 61 73 53 75 62 6f   
chema0...hasSubo
  0030:  72 64 69 6e 61 74 65 73  31 07 04 05 46 41 4c 53   
rdinates1...FALS
  0040:  45                                                 E
==> rewrite_context_apply [depth=1] string='cn=Subschema'
==> rewrite_rule_apply rule='(.*)cn=config,cn=slave$' 
string='cn=Subschema' [1 pass(es)]
==> rewrite_context_apply [depth=1] res={0,'cn=Subschema'}
ber_scanf fmt ({mW}) ber:
ber_dump: buf=0xb155b0 ptr=0xb1602a end=0xb16046 len=28
  0000:  30 1a 04 0f 68 61 73 53  75 62 6f 72 64 69 6e 61   
0...hasSubordina
  0010:  74 65 73 31 07 04 05 46  41 4c 53 45               tes1...FALSE
>>> dnPretty: <dc=savagebeast,dc=com>
=> ldap_bv2dn(dc=savagebeast,dc=com,0)
<= ldap_bv2dn(dc=savagebeast,dc=com)=0
syncrepl_message_to_entry: rid=001 mods check (olcSuffix: value #0 
invalid per syntax)
ldap_msgfree


And here is the consumer's olcDatabase={0}config.ldif before the sync 
starts--

dn: olcDatabase={0}config,cn=config
objectClass: olcDatabaseConfig
olcDatabase: {0}config
structuralObjectClass: olcDatabaseConfig
entryUUID: 3ff5aef4-ff02-102f-8945-2f8fc203546c
creatorsName: cn=admin,cn=config
createTimestamp: 20110419185450Z
olcRootDN: cn=admin,cn=config
olcRootPW:: xxx
olcSyncrepl: {0}rid=001 provider=ldaps://ldap.savagebeast.com binddn="cn=
 admin,cn=config,cn=slave" bindmethod=simple credentials=xxx 
searchbase="cn=
 config,cn=slave" type=refreshAndPersist retry="60 +" timeout=3 
suffixmassage=
 "cn=config" schemachecking=off
entryCSN: 20110420231921.726316Z#000000#001#000000
modifiersName: cn=admin,cn=config
modifyTimestamp: 20110420231921Z


And here is the consumer's olcDatabase={0}config.ldif at the point it fails.

dn: olcDatabase={0}config
objectClass: olcDatabaseConfig
olcDatabase: {0}config
structuralObjectClass: olcDatabaseConfig
entryUUID: 3ff5aef4-ff02-102f-8945-2f8fc203546c
creatorsName: cn=admin,cn=config
createTimestamp: 20110419185450Z
olcRootDN: cn=admin,cn=config
olcRootPW:: xxx
olcUpdateRef: ldaps://ldap.savagebeast.com
olcSyncrepl: {0}rid=001 provider=ldaps://guess-who.savagebeast.com 
binddn="cn=
 admin,cn=config,cn=slave" bindmethod=simple credentials=xxx 
searchbase="cn=
 config,cn=slave" schemachecking=off type=refreshAndPersist retry="60 
+" timeo
 ut=3 suffixmassage="cn=config"
entryCSN: 20110526212623.826014Z#000000#001#000000
modifiersName: cn=admin,cn=config
modifyTimestamp: 20110526212623Z

--

and here it is on the provider

dn: olcDatabase={0}config,cn=config,cn=slave
objectClass: olcDatabaseConfig
olcDatabase: {0}config
olcRootDN: cn=admin,cn=config
olcRootPW: {SHA}/xxx
olcSyncrepl: {0}rid=001 provider=ldaps://guess-who.savagebeast.com binddn="c
 n=admin,cn=config,cn=slave" bindmethod=simple credentials=xxx searchbase=
 "cn=config,cn=slave" schemachecking=off type=refreshAndPersist 
retry="60 +"
  timeout=3 suffixmassage="cn=config"
olcUpdateRef: ldaps://ldap.savagebeast.com
createTimestamp: 20110419185450Z
creatorsName: cn=admin,cn=config
entryCSN: 20110526212623.826014Z#000000#001#000000
entryDN: olcDatabase={0}config,cn=config,cn=slave
entryUUID:: M2ZmNWFlZjQtZmYwMi0xMDJmLTg5NDUtMmY4ZmMyMDM1NDZj
hasSubordinates: FALSE
modifiersName: cn=admin,cn=config
modifyTimestamp: 20110526212623Z
structuralObjectClass: olcDatabaseConfig
subschemaSubentry: cn=Subschema

Thanks for reading this far down!