Re: Some interesting thoughts

[Howard Chu <hyc@symas.com>]

David Boreham wrote:

> Quanah Gibson-Mount wrote:
>
> > 2) Replica initialization/re-initialization from the master - If you 
> > create an attribute "ndsReplicaRefresh" with a hostname as the value 
> > in the job class, it will reload that replica's DB, by sending binary 
> > indexes over the wire, and then removes the attributes from the job 
> > scheduler.  I can only see this working if the master & replica agree 
> > on their indexing, however. It also doesn't replicate the replica 
> > specific configuration options.
>
> This is not how it works. It sends the entries over the wire, but 
> feeds them into the
> fast concurrent import code on the replica.

That seems simple enough to do (except for the "fast" part. We 
definitely have work to do on speeding up indexing, and Jong has a 
number of items addressing that). Of course, unambiguously identifying a 
replica to update requires more than just a hostname, but that's a minor 
detail. Also, I have to wonder why such a feature is needed in a 
properly operating replication environment. For syncrepl, a freshly 
installed consumer will naturally pull down a complete copy of the 
database as its first action, so "initialization" should not be an 
issue. And if the replication mechanism is doing its job correctly, then 
"re-initialization" is not an issue.

> There _is_ an option to initialize a replica using binary indices, but 
> you have to
> move the data yourself (with ftp or whatever).

I suppose that could save some time. This is something I've been 
wondering about (in the context of "centipede") as well, coming up with 
a portable representation of an index that can be easily distributed. 
This would allow sparse replicas to fully evaluate search filters 
locally, then request specific matching entries from the remote masters. 
(The issue is most apparent in our translucent overlay, but probably has 
wider application.) If you represent an index as a set of key/value 
pairs, it's pretty obvious that the values must be entryUUIDs, but what 
would the keys be? Just AVAs? I guess the problem is that a portable 
representation would be just as voluminous as the original entry data, 
thus defeating some of the point of a "sparse" replica.

--
 -- Howard Chu
 Chief Architect, Symas Corp.       Director, Highland Sun
 http://www.symas.com               http://highlandsun.com/hyc
 Symas: Premier OpenSource Development and Support