Re: syncrepl: large datasets and expediting consumer's initialization

[masarati@aero.polimi.it]

> My DB_CONFIG:
>> set_cachesize 0 268435456 1
>> set_lg_regionmax 262144
>> set_lg_bsize 2097152
>> set_lg_dir logs
>
>
> The filesystem is ext3 on RHEL5.
>
>>    -q   enable quick (fewer integrity checks) mode.  Does fewer
>> consis-
>>         tency  checks  on the input data, and no consistency checks
>> when
>>         writing the database.  Improves the load time but if any
>> errors
>>         or  interruptions occur the resulting database will be
>> unusable.
>
>
> That last bit was enough for me not to use the -q, but it did reduce
> load time to 17 minutes.
>
> The performance of slapadd is significant, but what about syncrepl?
> Why is the consumer reviewing every object? Reviewing "-q", I discovered
>
>>    -w   write  syncrepl  context  information.   After  all  entries
>> are
>>         added, the contextCSN will be updated with the greatest
>> CSN  in
>>         the database.
>
> And that looks like an option that would prime my syncrepl info. So
>
> 	slapadd -q -w -l SLAPCAT.LDIF
>
> took 14 minutes to build and then 3 minutes to close the databases.
> This consumer has the same hardware as the provider that took 35
> minutes to rebuild the database.
>
> That "slapadd -w" looks like the fix. Would someone confirm or reject
> that?
>
> The provider's log file still shows it's reviewing many records. I
> guess it's not returning them. Will the log file show the DNs of
> results (as opposed to visited)?
>
> I restarted the provider with less logging; logs of a full syncrepl
> scans are sucking up disk space. Only 5 or 6 records would have changed.
>
> Is it normal for the provider to visit many (all?) objects even when
> the consumer would have a very current CSN?

If you slapadd to the consumer the output of slapcat from the producer,
the CSNs will be consistent, and no refresh will occur.  Did you by chance
slapadd to the consumer a fresh LDIF, with no UUID/CSN information?  What
-w does is simply to set the contextCSN to the latest entryCSN found in
the database.  If you slapcat from the producer, the suffix entry will
have a valid contextCSN and -w is not needed.

p.