2011/3/25 Cannady, Mike
<mike.cannady@htcinc.net>
> -----Original Message-----
> From: Buchan Milne [mailto:
bgmilne@staff.telkomsa.net]
> Sent: Friday, March 25, 2011 6:17 AM
> To: Cannady, Mike
> Cc:
openldap-technical@openldap.org
> Subject: Re: Multimaster ldap related questions
>
>
> ----- "Mike Cannady" <
mike.cannady@htcinc.net> wrote:
>
> > I have implemented a multi-master two node ldap with openldap 2.4.22
> > and Berkely DB 4.8.26 on Redhat enterprise 5.4 with several readonly
> > replicas off of the masters.
> >
> > I have a need to add several optional attributes to a schema and
> > probably should upgrade to 2.4.24 as well. If this was a
> > single-master
> > server, it would be easy to do; just slapcat the ldap store, update
> > software, change schema, slapadd the ldap store back, and resume
> > slapd.
>
> Why would you need to slapcat/slapadd to "add several optional
attributes"
While testing the 2 node multi-master nodes, I did identical changes
(added an optional attribute to a schema) by stopping the 2 slapd
daemonds, changed schema, started the daemonds. After a lot of adding
and deleting of info out of the stores, it was apparent that something
was wrong with the data and the syncs. I stopped both daemons, did the
dump and restore on one and purge the database on the other and started
them back up. I didn't have any problems after that. I know I didn't
have issues prior to the changes, so I assumed the S.O.P. for schema
changes is dump, change, restore.
Are you indicating that isn't the case? What about newly defined
required attributes in a schema or one that was optional and now
required where all the instances have the optional already specified.
>
> > I'm not sure how to do that with multi-master. One reason for using
> > multi-master was if one master was down, the other would keep
> > running.
> > One should be able to upgrade one server, have it catch up with the
> > changes that the other master had done while the first master is
down
> > and then repeat for the 2nd master.
>
> Well, it would apply if you weren't modifying data offline on the 1st
> master.
>
> > Is this possible? Has anyone
> > done
> > this and how was it done?
> >
> > I know in the near future, a high-level branch on my DIT will be
> > purged
> > and bulk reloaded.
>
> I can't think of a strategy where a bulk load will have neither:
> -write downtime
> -inconsistency (changes made in the window between the bulk generation
and
> the startup of the server after import will be lost)
>
> You aren't clear which of these you want/prefer/require.
>
I want to minimize the time that the data in the store is unavailable.
If the delete and bulk-load take 10 hours to do online, then the service
is effectively down for 10 hours. If the offline version of dump,
delete what is to be deleted, slapadd the results, and then slapadd the
new info takes only 30-60 minutes. The offline would be the method I
choose based on time. The amount of time was my next question:
> > I have tested the load with a test setup of
> > multi-master ldap. If I do it via ldapadd, it takes over 6 hours to
> > load. With slapadd (and slapd down) it only takes 25 minutes plus
> > the
> > time for the other master to get up-to-date.
>
> What is tool-threads set to? Which interesting slapadd options (e.g.
-q)
> did you use?
Tool-threads is not specified, so I guess its one. The test box is a
single hyperthreaded cpu.
Slapadd command (for master # 1)
slapadd -c -f ldapfile -n 1 -S 001 -w
>
> > Is there any way that I
> > can speed-up the update with ldapadd?
>
> ldapadd will never be as fast as slapadd.
>
Granted. What I'm asking is there a way to speed up slapd for the
duration of the mass deletes and bulk loads. In normal circumstances,
the syncs and such would be OK for normal processing; but, with bulk
changes, I would like have the daemon run in such a way that syncs are
not done and everything is done in memory. My tests seem to indicate
that disk I/O is the bottleneck. I know if there is a failure during
this time, the database may be corrupted, but this OK for the bulk
change duration.
Hello,
I have a machine that was time consuming to restore a base
ldif with "slapadd.
My problem was the I / O (HD very slow).
Resolved with a small shell:
#!/bin/bash
/etc/init.d/slapd stop
slapcat /tmp/backup.ldif
dir_ldap="/var/lib/ldap"
rm -f /var/lib/ldap/{alock,__db.*,*bdb,log.*}
mv $dir_ldap $dir_ldap.real
mkdir $dir_ldap
mount -t tmpfs tmpfs $dir_ldap -o nr_inodes=3000000,size=4G
cp $dir_ldap.real/DB_CONFIG $dir_ldap
slapadd -c -q -w -b "dc=example,dc=com" -l /tmp/backup.ldif
rsync -a --delete $dir_ldap/ $dir_ldap.real/
umount $dir_ldap
rmdir $dir_ldap
mv $dir_ldap.real $dir_ldap
chown openldap:openldap $dir_ldap -R
/etc/init.d/slapd start
Thus, the I / O during the slapadd no longer exists, because all files are in memory. The time difference with "tmpfs" is striking.
> > I have pieces of my slapd.conf
> > for the 1st master at the end of this email.
> >
> > Slapadd has two options that appear to be needed when dealing with
> > multi-master or replicate nodes. The first is the "-S sid" option,
> > the
> > second is "-w". I'm a little confused what is used where. If you
> > are
> > doing a dump and restore operation (slapcat, delete database,
> > slapadd)
> > the only option you need is the "-w" option? If you are adding new
> > entries offline then do both options need to be specified?
>
> Adding, *or* modifying.
I don't understand this answer.
>
> > Is there a multi-master best practice quide somewhere?
>
> A good start is to never lie to slapd. If you have changed the
contents of
> an entry, the entryCSN should not be retained.
>
> I also prefer to avoid non-restore bulk-loading.
Me too, but if it is a decision of 10 hours vs. 1 hour, the 1 hour will
win.
>
> Regards,
> Buchan
Thanks,
Mike
**********************************************************************
HTC Disclaimer: The information contained in this message may be privileged and confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting it from your computer. Thank you.
**********************************************************************