Buchan,
I made exactly what you said even using the -q flag in the slapadd
command. So in summary I did :
1) Load the master DB using LDIF file through slapdd(-q flag and
DB_CACHEZIZE to 1GB);
2) Load the slave DB using LDIF(same) file through slapdd(-q flag and
DB_CACHEZIZE to 1GB);
3)Then have the slapd.conf files appropriately configured
4)Start master and then after some minutes start slave.
So the slave is not starting blank but exactly loaded and in terms of
data there isn't any difference between provider(master) and
consumer(slave). I just see when the consumer slapd process start at
slave machine(slapd started with -d 256 flag) the connection from the
consumer to the provider slapd process.
Then all behavior explained start to happen. Please see the provider and
consumer configuration file attached.
Something appears not being following the expect behavior. Also the
memory consumed in the consumer is growing too fast and doesn't appear
to really follow the caches directives.
Regards,
Rodrigo.
Buchan Milne wrote:
> On Sunday 03 May 2009 04:15:47 Rodrigo Costa wrote:
>
>> openldap software,
>>
>> Sometime ago I open the ITS#5860 about some memory cache limitations not
>> being respected by config files. Even this issue was solved when I tried
>> to configured openldap to use replication(syncrepl) the system never
>> enter into sync and the behavior appears similar to the ITS#5860 bug.
>>
>> The system start to sync and in the provider(master) I see the query for
>> the DB sync. But the consumer(slave) memory consumption start to grow
>> very fast making me to constrain much more the dncachesize to a 1/10 of
>> the size of the provider(master) where at least system doesn't crash at
>> consumer.
>>
>> Since changes were done in the openldap 2.4.16 I download and made tests
>> with this version. I get into the same behavior with consumer(slave)
>> never getting in sync with provider(master).
>>
>> The behaviors are :
>>
>> 1) Consumer(slave) start query to the provider(master) DB;
>> 2) Memory allocation and number of threads in the provider(master) start
>> to increase as expected;
>> 3) dncachesize directive into provider(master) controls as expected the
>> maximum memory to be allocated by slapd process in provider(master);
>> 4) Consumer(slave) consumer memory in a much faster pace. dncachesize
>> configured to 1/10 of provider(master) to avoid memory allocation problems;
>> 5) After sometime the consumer(slave) CPU usage maintains in 200%.
>> Provider(master) stays with low CPU usage, around 1 to 3 %;
>> 6) A new provisioning in provider(master) isn't propagated to
>> consumer(slave);
>> 7) Bases never get in sync and CPU usage in consumer still high. Queries
>> to provider(master) are answer very fast and even multiple individual
>> queries to consumer(slave) are also answer in reasonable time.
>>
>> It looks like could exist certain issue in the replication logic where
>> some processing dead loop could be found by the replication
>> consumer(slave) logic.
>>
>> The newest openldap version and Berkeley DB 4.7 with all patches were
>> compiled in the platform running the code.
>>
>> Any idea about this behavior?
>>
>
> I have seen behaviour like this when there was something preventing
> synchronisation, and the comsumer would spawn more consumer threads, until the
> box ran out of memory. I fixed the real issue, and haven't seen it since (and
> haven't had time to try and reproduce it).
>
> However, on a large database, you may have better success by initialising the
> consumer via 'slapadd' with an export from a provider, instead of using
> syncrepl to do it. Since slapadd can run multiple threads (syncrepl only runs
> one thread), and doesn't need to bother serving client requests, and can run
> without transactions (see -q flag), it is much more efficient.
>
> Note that you could consider different tuning for import vs run-time, e.g. I
> usually increase the BDB cache_size for imports with slapadd, and decrease it
> for runtime.
>
> Regards,
> Buchan
>
>
>
>
>
Attachment:
slapd_slave.conf
Description: Binary data
Attachment:
slapd_master.conf
Description: Binary data