[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: Getting around the single-threaded syncrepl model?

I'm on vacation the rest of the week, but the quick answer is no.  All write ops (add/mod/rename/delete etc) get stored in the accesslog db. Fallback only occurs when the change has been expired out of the accesslog db. 

--Quanah

On Nov 24, 2015, at 1:30 AM, Bannister, Mark <Mark.Bannister@morganstanley.com> wrote:

>>>> And here's some quick stats:
>>>> $ grep syncrepl CHANGES  | grep -v delta | wc -l
>>>> 76
>>>> 
>>>> $ grep delta-syncrepl CHANGES  | wc -l
>>>> 7
>>>> 
>>>> Or syncrepl has had 10x times the issues.
>>> 
>>> Or syncrepl has 10x the users, more eyes spot more bugs, and is now 
>>> more stable because it has had more fixes? I'm just speculating here, 
>>> but I wouldn't be more confident in product Y because it is mentioned 
>>> less in the change log than product X.
>> 
>> Nope.  I've been using syncrepl since it was first released in 2.2.  I switched over the production
>> ldap servers I was running @ Stanford to syncrepl when 2.3 came out, and found a really vexing
>> problem when mass updates occurred (primarily at quarter end when we'd receive 10s of
>> thousands of updates due to class enrollment changes).  While 1-2 of our 6 replicas would stay up
>> to date, the other 4 would fall hours or days behind.  I.e., I'm quite familiar with the issue you are
>> discussing, because I've been experiencing it for over a decade.  Out of this, delta-syncrepl was
>> born.  I've been using it steadily now for over a decade.  I switched Zimbra to it back in 2007, and
>> it is deployed at customers world wide, from insallations with 10-20 users to installations with many
>> millions of users.  Has there been the occassional issue? Yes.  But again, the vast majority of
>> problems I encounter when investigating problems around replication come from syncrepl fallback.
>> There is a very significant fix coming out in 2.4.43 that was found by Zimbra with syncrepl refresh
>> when it is interrupted during the refresh phase (ITS#8281).
>> 
>> So, when I talk about replication and reliability in relation to OpenLDAP, this is something I've been
>> working with since the early days of OpenLDAP 2.1.  It's something that as a part of my job, I have
>> to take with the utmost seriousness.
>> 
>> Now, you can ignore my advice if you wish, it doesn't matter to me one way or the other.  But to
>> me, mission critical systems must be as reliable as possible, and I've only found that to be the case
>> with openldap replication when I deploy delta-syncrepl as the primary replication mechanism.
> 
> Thanks for your insight, Quanah, and this is a much more substantial argument now than doing
> 'grep -c' on a changelog, and I appreciate the time you spent to make that case.
> 
> If we're primarily adding or deleting entire objects, however, not modifying single attributes,
> won't delta-syncrepl just fall back to syncrepl anyway?
> 
> And this still doesn't change the fact that I'm now going to have to spin up 10-20 LDAP replicas on
> a single machine to get around the single-threaded replication engine problem.
> 
> Mark.
> 
> 
> --------------------------------------------------------------------------------
> 
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.