Re: multiway replication

[Venkata Pingali <pingali@ISI.EDU>]

Thanks, Howard! I will surely check with 2.3. Meanwhile  I got another 
configuration "sort-of" working (see diagram).

What I had to do was the following:

1. Have the sites write into a "local" (ou=local,ou=xbone) subtree
2. The local subtree mods are propogated to a "proxy" slapd listening on 
port 10000 at theHQ machine. This "proxy" runs a slapd-meta 
configuration to rewrite "ou=local,ou=xbone" to "ou=global,ou=xbone" and 
forward the updates to the main slapd server (port 636) with the 
rewritten DNs.
3. The sites then synrepl the global shared tree (ou=global,ou=xbone)

In effect there is no loop. However, I had two problems:

1. meta backend at the proxy was filtering timestamp attributes when 
forwarding mods to the main slapd
2. the main slapd's contextCSN was not getting updated.

For (1) I had temporarily comment the following lines in back-meta/add.c:

          if ( a->a_desc->ad_type->sat_no_user_mod  ) {
                       continue;
               }

I am not sure about the appropriateness of the action and I was going to 
ask you all.

For (2) I had a background job that just fiddled (add and delete an 
entree) with the main server to force an update. Hopefully I wont have 
to do this with 2.3.

Do you have an idea about (1) above?

thanks!
-Venkata

Howard Chu wrote:

> Sounds like an interesting solution. It should work using the 2.3 
> syncprov overlay, as the method for recording the contextCSN has been 
> changed (among other things).
>
> Venkata Pingali wrote:
>
> > What:
> >
> > A tree that is (1) shared at multiple sites (arbitrary number) (2) each
> > site responsible for its part of the subtree.  However when a site makes
> > changes the changes have to propagate to all site so that everyone sees
> > the same tree. We dont know ahead of time how many sites there will be.
> > The data  volume is expected to be low i.e., tree is quite small.
> >
> > One solution:
> >
> > I have looked at the multimaster solution in which each site updates
> > N other sites through N replica commands.
> >
> > The problem is that as new sites get added, the configuration at every
> > site will have to changed.
> >
> > Alternate solution:
> >
> > An alternative design  is shown in the attached text file. Basically 
> > there is a
> > central repository Each site pushes changes into the tree at the central
> > repository using slurpd and sucks in any updates using the syncrepl
> > functionality. (All slapd/slurpd compiled with multimaster flag enabled)
> > A nice property of  this setup is that the central server need not
> > track how many sites exist and any site that is added get
> > reflected automatically in the tree.
> >
> > Problem:
> >
> > Each site is able to push the changes to the central slapd server but 
> > not
> > pull changes using syncrepl. If the changes are done at the central site
> > using ldapadd the sites are able to pull the updates.
> >
> > After some digging I found that the ldapsync object's contextCSN
> > (a time stamp) attribute at the central tree was not getting updated
> > after site slurpd modifies the tree at the central server. Therefore
> > when the site slapd server tries to look for updates using syncrepl,
> > it doesnt find any even when the tree has been updated
> > (since the timestamp - contextCSN - has not been updated).
> >
> > Is there a compile time/runtime option to force the contextCSN
> > updates? Alternatively is it possible to update using some offline
> > mechanism that I can put into cron?
> >
> > More fundamentally, does this sound like a reasonable approach
> > for N-way replication? Are you aware of other solutions?
> >
> > thanks!
> > Venkata Pingali
> > USC/ISI
> >
> > ------------------------------------------------------------------------
> >
> >
> > Shows slapd at HQ and slapd/slurpd combination at one site.
> >                      HQ             +--------------------+
> >          +->|        slapd       |
> >         /   +--------------------+
> >        /     |                 /      |
> > +-------------------+      +----
> > |     /       V     |      |
> > |   +-+      +-+    |      |
> > |   | |<-----| |    | .... |
> > |   +-+      +-+    |      |
> > |  slurpd   slapd   |      |
> > +-------------------+      +-----
> >
> >        SITE 1                SITE 2 ...
> >  


    	              HQ 
        +-----+     +-----+
     +->|meta |---->|slapd| 
     |  +-----+     +-----+ 
     |                 |
     |        +--------+
+-------------+-----+      +----
|    |        V     |      |
|   +-+      +-+    |      |
|   | |<-----| |    | .... |
|   +-+      +-+    |      |
|  slurpd   slapd   |      |
+-------------------+      +-----

        SITE 1                SITE 2 ...