RE: (ITS#4235) Master crashes upon replica's serach request

[ali.pouya@free.fr]

Quanah;
Thanks for the useful and detailed information.
I finally could have a stable configuration with one master and two
replicas.
Now I am injecting in the master 15 million entries (this is a bench).
The replication delay is around one second.
I will use your recommendation in the final projet.
Best regards
Ali


-----Message d'origine-----
De : Quanah Gibson-Mount [mailto:quanah@stanford.edu]
Envoye : mercredi 30 novembre 2005 22:06
A : ali.pouya@free.fr; openldap-its@OpenLDAP.org
Objet : Re: (ITS#4235) Master crashes upon replica's serach request




--On Wednesday, November 30, 2005 7:39 PM +0000 ali.pouya@free.fr wrote:

> Hi Quanah,
> Below you find the answers to your questions.
> Best Regards
> Ali
>
>
>> I'm not sure what you mean by "slapd continues to overview all of the
>> entries".  slapd should always be able to see all of the entries in its
>> database.  I'm also not entirely clear what is taking a long time in your
>> message.
>
> I started with a stand alone directory loaded by an LDAP client (not
> through slapadd).

Okay.  I would generally advise loading a large directory via slapadd,
it'll save you many hours.

> I think that in this case when you promote your server to master (by
> adding syncprov verlay), after start up slapd revises every entry (see
> the trace in follow up 2). Is this to add or check an information related
> to the fact that now my server is a master ?

If you intend to have the server be a master, you should have syncprov in
there to start with.  For the syncrepl protocol to function correctly,
information has to be stored in the root entry.  Since you didn't have
syncprov in there previously, it then has to go and get that information
from the data you've loaded, and then set the information in the root
entry.  That is why it takes a very long time to start up.

> Also if I transfer my disk data to a replica, after start up the replica
> checks all of the entries with the server through syncrepl protocol.

How did you transfer the data to the replica?  Before transferring the data
to the replica, you will want the syncprov overlay to finish its search
described above, otherwise the replica will just try to refresh from the
master.  After then necessary information is present in the master, I would
suggest one of two ways to get the data to the replica:

1) If the replica has the identical database and identical indexing, simply
stop slapd on the master, run db_recover in the database directory, and
then copy the *.bdb files to the database directory on the replica.  Also
copy the BDB log files from wherever they are stored on the master to the
replica as well.  Then start up slapd on both servers.

2) Run slapcat on the master to store the file in LDIF format.  slapadd -q
-l file on the replica.

> As I have 10 million entries each operation take many hours.

Mainly, it appears, because you aren't using the correct tools or setting
things up correctly to start with.

>> Here are some basic questions:
>>
>> 1) Did you use "-w" with slapadd while loading the master?
> I did not use slapdadd.

Unfortunate.  ldapadd is not nearly as efficient or speedy as slapadd.


>> 2) Did you properly configure DB_CONFIG for the size of your database?
> I followed the exemple (250 MB)

I suggest you read the following documentation:
<http://www.stanford.edu/services/directory/openldap/configuration/bdb-confi
g.html>

You likely are causing serious problems for your database by having too
small of a BDB cache.

>> 3) I suggest using 2.3.13, there are some improvements to the syncprov
> When will it be available ?

It is available from the ftp site now.

--Quanah

--
Quanah Gibson-Mount
Principal Software Developer
ITSS/Shared Services
Stanford University
GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html