Quanah,
Can you try the refreshOnly configuration with the current HEAD ?
HEAD was updated to make it efficient. Major updates are :
1) do not cache entries on the internal search for present entries
2) fixed (most) memory leaks
- Jong
------------------------
Jong Hyuk Choi
IBM Thomas J. Watson Research Center - Enterprise Linux Group
P. O. Box 218, Yorktown Heights, NY 10598
email: jongchoi@us.ibm.com
(phone) 914-945-3979 (fax) 914-945-4425 TL: 862-3979
Quanah Gibson-Mount <quanah@stanford.edu> Sent by: owner-openldap-devel@OpenLDAP.org
2003-09-24 01:22 AM
To: Jonghyuk Choi/Watson/IBM@IBMUS
cc: openldap-devel@OpenLDAP.org
Subject: Re: syncrepl questions
--On Wednesday, September 24, 2003 12:16 AM -0400 Jonghyuk Choi
<jongchoi@us.ibm.com> wrote:
>> So, if I have a database with 5 million entries, it is going to send
>> along 1 change + 4,999,999 ID's of unchanged entries?
>
> Yes.
> The revised (soon too be published) LDAP Sync protocol has two modes.
> One is state-based (sending add+present) and the other is log-based
> (sending add+delete).
> If OpenLDAP becomes capable of maintaining history store
> then the Syncrepl engine will be enabled to switch between the two modes
> depending on the consumer's state. If the consumer state is recent enough
> to be covered by the history store, then only changes would be sent if it
> is more efficient.
> If the consumer state cannot be covered by the history store, then the
> additive changes
> and IDs of present entries will be sent, which is far more efficient than
> a full reload.
> On the other hand, the LCUP proposal only supports the log-based mode.
> So, without the history store, full reload would be a frequent event.
Okay, then I understand what is happening with my replica/servers -- It is
sending all the blank pieces, which eats up a huge amount of resources and
causes my systems to start swapping horribly. I'll start poking into what
in syncrepl is causing the resource issues. The fact that it always has to
send everything in the current methodology, makes, IMHO, syncrepl useless
for anyone with a medium or lager database, since the servers are going to
be spending all their time sending the changes + empty entries. For our
systems, the dump is happening less quickly than changes are occuring, with
means changes would be stacking up, at least if it is left in persistent
mode. Probably the same thing if it is put in periodic updates, unless I
have it only do those once every couple of ours, which doesn't work for us,
because we need updates to be as close to real time as possible.
--Quanah
--
Quanah Gibson-Mount
Principal Software Developer
ITSS/TSS/Computing Systems
Stanford University
GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html