13. Replication with slurpd
In certain configurations, a single slapd(8) instance may be insufficient to handle the number of clients requiring directory service via LDAP. It may become necessary to run more than one slapd instance. At many sites, for instance, there are multiple slapd servers: one master and one or more slaves.
slurpd(8) provides the capability for a master slapd to propagate changes to slave slapd instances, implementing the master/slave replication scheme described above. slurpd runs on the same host as the master slapd instance.
13.1. Overview
slurpd(8) provides replication services "in band". That is, it uses the LDAP protocol to update a slave database from the master. Perhaps the easiest way to illustrate this is with an example. In this example, we trace the propagation of an LDAP modify operation from its initiation by the LDAP client to its distribution to the slave slapd instance.
Sample replication scenario:
- The LDAP client submits an LDAP modify operation to the slave slapd.
- The slave slapd returns a referral to the LDAP client referring the client to the master slapd.
- The LDAP client submits the LDAP modify operation to the master slapd.
- The master slapd performs the modify operation, writes out the change to its replication log file and returns a success code to the client.
- The slurpd process notices that a new entry has been appended to the replication log file, reads the replication log entry, and sends the change to the slave slapd via LDAP.
- The slave slapd performs the modify operation and returns a success code to the slurpd process.
Note: ldapmodify(1) and other tools distributed as part of OpenLDAP Software do not support automatic referral chasing.
13.2. Replication Logs
When slapd is configured to generate a replication logfile, it writes out a file containing
replica: slave.example.com:389
time: 809618633
dn: uid=bjensen,dc=example,dc=com
changetype: modify
replace: multiLineDescription
description: A dreamer...
-
replace: modifiersName
modifiersName: uid=bjensen,dc=example,dc=com
-
replace: modifyTimestamp
modifyTimestamp: 20000805073308Z
-
The modifications to modifiersName and modifyTimestamp operational attributes were added by the master slapd.
13.3. Command-Line Options
This section details commonly used slurpd(8) command-line options.
-d <level> | ?
This option sets the slurpd debug level to <level>. When level is a `?' character, the various debugging levels are printed and slurpd exits, regardless of any other options you give it. Current debugging levels (a subset of slapd's debugging levels) are
| Level | Description |
| 4 | heavy trace debugging |
| 64 | configuration file processing |
| 65535 | enable all debugging |
Debugging levels are additive. That is, if you want heavy trace debugging and want to watch the config file being processed, you would set level to the sum of those two levels (in this case, 68).
-f <filename>
This option specifies an alternate slapd configuration file. Slurpd does not have its own configuration file. Instead, all configuration information is read from the slapd configuration file.
-r <filename>
This option specifies an alternate slapd replication log file. Under normal circumstances, slurpd reads the name of the slapd replication log file from the slapd configuration file. However, you can override this with the -r flag, to cause slurpd to process a different replication log file. See the Advanced slurpd Operation section for a discussion of how you might use this option.
-o
Operate in "one-shot" mode. Under normal circumstances, when slurpd finishes processing a replication log, it remains active and periodically checks to see if new entries have been added to the replication log. In one-shot mode, by comparison, slurpd processes a replication log and exits immediately. If the -o option is given, the replication log file must be explicitly specified with the -r option. See the One-shot mode and reject files section for a discussion of this mode.
-t <directory>
Specify an alternate directory for slurpd's temporary copies of replication logs. The default location is /usr/tmp.
13.4. Configuring slurpd and a slave slapd instance
To bring up a replica slapd instance, you must configure the master and slave slapd instances for replication, then shut down the master slapd so you can copy the database. Finally, you bring up the master slapd instance, the slave slapd instance, and the slurpd instance. These steps are detailed in the following sections. You can set up as many slave slapd instances as you wish.
13.4.1. Set up the master slapd
The following section assumes you have a properly working slapd(8) instance. To configure your working slapd(8) server as a replication master, you need to make the following changes to your slapd.conf(5).
- Add a replica directive for each replica. The binddn= parameter should match the updatedn option in the corresponding slave slapd configuration file, and should name an entry with write permission to the slave database (e.g., an entry allowed access via access directives in the slave slapd configuration file). This DN generally should not be the same as the master's rootdn.
- Add a replogfile directive, which tells slapd where to log changes. This file will be read by slurpd.
13.4.2. Set up the slave slapd
Install the slapd software on the host which is to be the slave slapd server. The configuration of the slave server should be identical to that of the master, with the following exceptions:
- Do not include a replica directive. While it is possible to create "chains" of replicas, in most cases this is inappropriate.
- Do not include a replogfile directive.
- Do include an updatedn line. The DN given should match the DN given in the binddn= parameter of the corresponding replica= directive in the master slapd config file. The updatedn generally should not be the same as the rootdn of the master database.
- Make sure the DN given in the updatedn directive has permission to write the database (e.g., it is is allowed access by one or more access directives).
- Use the updateref directive to define the URL the slave should return if an update request is received.
13.4.3. Shut down the master server
In order to ensure that the slave starts with an exact copy of the master's data, you must shut down the master slapd. Do this by sending the master slapd process an interrupt signal with kill -INT <pid>, where <pid> is the process-id of the master slapd process.
If you like, you may restart the master slapd in read-only mode while you are replicating the database. During this time, the master slapd will return an "unwilling to perform" error to clients that attempt to modify data.
13.4.4. Copy the master slapd's database to the slave
Copy the master's database(s) to the slave. For an
Note: This copy process assumes homogeneous servers with identically configured OpenLDAP installations. Alternatively, you may use slapcat to output the master's database in LDIF format and use the LDIF with slapadd to populate the slave. Using LDIF avoids any potential incompatibilities due to differing server architectures or software configurations. See the Database Creation and Maintenance Tools chapter for details on these tools.
13.4.5. Configure the master slapd for replication
To configure slapd to generate a replication logfile, you add a " replica" configuration option to the master slapd's config file. For example, if we wish to propagate changes to the slapd instance running on host slave.example.com:
replica uri=ldap://slave.example.com:389
binddn="cn=Replicator,dc=example,dc=com"
bindmethod=simple credentials=secret
In this example, changes will be sent to port 389 (the standard LDAP port) on host slave.example.com. The slurpd process will bind to the slave slapd as "cn=Replicator,dc=example,dc=com" using simple authentication with password "secret".
If we wish to perform the same replication using ldaps on port 636:
replica uri=ldaps://slave.example.com:636
binddn="cn=Replicator,dc=example,dc=com"
bindmethod=simple credentials=secret
The host option is deprecated in favor of uri, but the following replica configuration is still supported:
replica host=slave.example.com:389
binddn="cn=Replicator,dc=example,dc=com"
bindmethod=simple credentials=secret
Note that the DN given by the binddn= directive must exist in the slave slapd's database (or be the rootdn specified in the slapd config file) in order for the bind operation to succeed. The DN should also be listed as the updatedn for the database in the slave's slapd.conf(5). It is generally recommended that this DN be different than the rootdn of the master database.
Note: The use of strong authentication and transport security is highly recommended.
13.4.6. Restart the master slapd and start the slave slapd
Restart the master slapd process. To check that it is generating replication logs, perform a modification of any entry in the database, and check that data has been written to the log file.
13.4.7. Start slurpd
Start the slurpd process. Slurpd should immediately send the test modification you made to the slave slapd. Watch the slave slapd's logfile to be sure that the modification was sent.
slurpd -f <masterslapdconfigfile>
13.5. Advanced slurpd Operation
13.5.1. Replication errors
When slurpd propagates a change to a slave slapd and receives an error return code, it writes the reason for the error and the replication record to a reject file. The reject file is located in the same directory as the per-replica replication logfile, and has the same name, but with the string ".rej" appended. For example, for a replica running on host slave.example.com, port 389, the reject file, if it exists, will be named
/usr/local/var/openldap/replog.slave.example.com:389.rej
A sample rejection log entry follows:
ERROR: No such attribute
replica: slave.example.com:389
time: 809618633
dn: uid=bjensen,dc=example,dc=com
changetype: modify
replace: description
description: A dreamer...
-
replace: modifiersName
modifiersName: uid=bjensen,dc=example,dc=com
-
replace: modifyTimestamp
modifyTimestamp: 20000805073308Z
-
Note that this is precisely the same format as the original replication log entry, but with an ERROR line prepended to the entry.
13.5.2. One-shot mode and reject files
It is possible to use slurpd to process a rejection log with its "one-shot mode." In normal operation, slurpd watches for more replication records to be appended to the replication log file. In one-shot mode, by contrast, slurpd processes a single log file and exits. Slurpd ignores ERROR lines at the beginning of replication log entries, so it's not necessary to edit them out before feeding it the rejection log.
To use one-shot mode, specify the name of the rejection log on the command line as the argument to the -r flag, and specify one-shot mode with the -o flag. For example, to process the rejection log file /usr/local/var/openldap/replog.slave.example.com:389 and exit, use the command
slurpd -r /usr/tmp/replog.slave.example.com:389 -o