21. Load Balancing with lloadd

As covered in the Replication chapter, replication is a fundamental requirement for delivering a resilient enterprise deployment. As such there's a need for an LDAPv3 capable load balancer to spread the load between the various directory instances.

lloadd(8) provides the capability to distribute LDAP v3 requests between a set of running slapd instances. It can run as a standalone daemon lloadd, or as an embedded module running inside of slapd.


21.1. Overview

lloadd(8) was designed to handle LDAP loads. It is protocol-aware and can balance LDAP loads on a per-operation basis rather than on a per-connection basis.

lloadd(8) distributes the load across a set of slapd instances. The client connects to the load balancer instance which forwards the request to one of the servers and returns the response back to the client.


21.2. When to use the OpenLDAP load balancer

In general, the OpenLDAP load balancer spreads the load across configured backend servers. It does not perform so-called intelligent routing. It does not understand semantics behind the operations being performed by the clients.

More considerations:


21.3. Directing operations to backends

21.3.1. Default behaviour

In the simplest configuration several backends would be configured within a single roundrobin tier:

       feature proxyauthz

       bindconf bindmethod=simple
                binddn="cn=Manager,dc=example,dc=com"
                credentials=secret

       tier roundrobin
       backend-server uri=ldap://server1.example.com
                      numconns=5 bindconns=5
                      max-pending-ops=10 conn-max-pending=3
                      retry=5000
       backend-server uri=ldap://server2.example.com
                      numconns=5 bindconns=5
                      max-pending-ops=10 conn-max-pending=3
                      retry=5000

After startup lloadd will open 10 connections to each ldap://server1.example.com and ldap://server2.example.com, 5 for regular requests, where it will bind as cn=Manager,dc=example,dc=com, and 5 dedicated to serving client Bind requests. If connection set up fails, it will wait 5000ms (5 seconds) before making another attempt to that server.

When a new Bind request comes from a client, it will be allocated to one of the available bind connections, each of which can only carry one request at a time. For other requests that need to be passed on to the backends, backends are considered in order:

When a connection is chosen, the operation is forwarded and response(s) returned to the client. Should that connection go away before the final response is received, the client is notified with a LDAP_OTHER failure code.

So long as feature proxyauthz is configured, every operation forwarded over a regular connection has the PROXYAUTHZ control (RFC4370) prepended indicating the client's bound identity, unless that identity matches the binddn configured in bindconf.

If another tier is configured:

       tier roundrobin
       backend-server uri=ldap://fallback.example.com
                      numconns=5 bindconns=5
                      max-pending-ops=10 conn-max-pending=3
                      retry=5000

Backends in this tier will only be considered when lloadd would have returned LDAP_UNAVAILABLE in the above case.

21.3.2. Alternate selection strategies

For various reasons, the roundrobin tier is appropriate in the majority of use cases as it is both very scalable in terms of its implementation and how its self-limiting interacts with backends when multiple lloadd instances are being used at the same time.

Two alternative selection strategies have been implemented:

The weighted tier might be appropriate when servers have differing load capacity. Due to its reinforced self-limiting feedback, the bestof tier might be appropriate in large scale environments where each backend's capacity/latency fluctuates widely and rapidly.

21.3.3. Coherence

21.3.3.1. Write coherence

In default configurations, every operation submitted by the client is either processed internally (e.g. StartTLS, Abandon, Unbind, ...) or is forwarded to a connection of lloadd's choosing, independent of any other other operation submitted by the same client.

There are certain traffic patterns where such such freedom is undesirable and some kind of coherency is required. This applies to write traffic, controls like Paged Results or many extended operations.

Client's operations can be pinned to the same backend as the last write operation:

       write_coherence 5

In this case, client's requests will be passed over to the same backend (not necessarily over the same upstream connection) from the moment a write request is passed on till at least 5 seconds have elapsed since last write operation has finished.

       write_coherence -1

Here, there is no timeout and the moment a write request is passed on to a backend, the client's operations will forever be passed on to this backend.

In both cases above, this limitation is lifted the moment a Bind request is received from the client connection.

21.3.3.2. Extended operations/controls

Many controls and Extended operations establish shared state on the session. While lloadd implements some of these (StartTLS being one example), it supports the administrator in defining how to deal with those it does not implement special handling for.

       restrict_exop 1.1 reject
       # TXN Exop
       restrict_exop 1.3.6.1.1.21.1 connection
       # Password Modify Exop
       restrict_exop 1.3.6.1.4.1.4203.1.11.1 write

       # Paged Results Control
       restrict_control 1.2.840.113556.1.4.319 connection
       # Syncrepl
       restrict_control 1.3.6.1.4.1.4203.1.9.1 reject

The above configuration uses the special invalid OID of 1.1 to instruct lloadd to reject any Extended operation it does not recognize, except for Password Modify operation which is treated according to write_coherence above and the LDAP transactions, where it forwards all subsequent requests over to the same upstream connection. Similarly, once a Paged results control is seen on an operation, subsequent request will stick to the same upstream connection while LDAP Syncrepl requests will be rejected outright.

With both restrict_exop and restrict_control, any such limitation is lifted when a new Bind request comes in as any client state is assumed to be reset.

When configuring these to anything else than reject, keep in mind that many extensions have not been designed or implemented with a multiplexing proxy like lloadd in mind and might open considerable operational and/or security concerns when allowed.


21.4. Runtime configurations

It deploys in one of two ways:

  1. Standalone daemon: lloadd
  2. Loaded into the slapd daemon as a module: lloadd.la

It is recommended to run with the balancer module embedded in slapd because dynamic configuration (cn=config) and the monitor backend are then available.

Sample load balancer scenario:

Figure: Load balancer sample scenario

  1. The LDAP client submits an LDAP operation to the load balancer daemon.
  2. The load balancer forwards the request to one of the backend instances in its pool of servers.
  3. The backend slapd server processes the request and returns the response to the load balancer instance.
  4. The load balancer returns the response to the client. The client's unaware that it's connecting to a load balancer instead of slapd.

21.5. Build Notes

To build the load balancer from source, follow the instructions in the A Quick-Start Guide substituting the following commands:

  1. To configure as standalone daemon:
      ./configure --enable-balancer=yes
  2. To configure as embedded module to slapd:
      ./configure --enable-modules --enable-balancer=mod

21.6. Sample Runtime

  1. To run embedded as lloadd module:
      slapd [-h URLs] [-f lloadd-config-file] [-u user] [-g group]
    • the startup is the same as starting the slapd daemon.
    • URLs is for slapd management. The load balancer's listener URLs set in the configuration file or node. (more later)
  2. To run as standalone daemon:
      lloadd [-h URLs] [-f lloadd-config-file] [-u user] [-g group]
    • Other than a different daemon name, running standalone has the same options as starting slapd .
    • -h URLs specify the lloadd's interface directly, there is no management interface.

For a complete list of options, checkout the man page lloadd.8


21.7. Configuring load balancer

21.7.1. Common configuration options

Many of the same configuration options as slapd. For complete list, check the lloadd(5) man page.

To configure your working lloadd(8) you need to make the following changes to your configuration file:

  1. include core.schema (embedded only)
  2. TLSShareSlapdCTX { on | off }
  3. Other common TLS slapd options
  4. Setup argsfile/pidfile
  5. Setup moduleload path (embedded mode only)
  6. moduleload lloadd.la
  7. loglevel, threads, ACL's
  8. backend lload begin lloadd specific backend configurations
  9. listen ldap://:PORT Specify listen port for load balancer
  10. feature proxyauthz Use the proxy authZ control to forward client's identity
  11. io-threads INT specify the number of threads to use for the connection manager. The default is 1 and this is typically adequate for up to 16 CPU cores

21.7.2. Sample backend config

Sample setup config for load balancer running in front of four slapd instances.

backend lload

# The Load Balancer manages its own sockets, so they have to be separate
# from the ones slapd manages (as specified with the -h "URLS" option at
# startup).
listen ldap://:1389

# Enable authorization tracking
feature proxyauthz

# Specify the number of threads to use for the connection manager.  The default is 1 and this is typically adequate for up to 16 CPU cores.
# The value should be set to a power of 2:
io-threads  2

# If TLS is configured above, use the same context for the Load Balancer
# If using cn=config, this can be set to false and different settings
# can be used for the Load Balancer
TLSShareSlapdCTX true

# Authentication and other options (timeouts) shared between backends.
bindconf bindmethod=simple
         binddn=dc=example,dc=com credentials=secret
         network-timeout=5
         tls_cacert="/usr/local/etc/openldap/ca.crt"
         tls_cert="/usr/local/etc/openldap/host.crt"
         tls_key="/usr/local/etc/openldap/host.pem"


# List the backends we should relay operations to, they all have to be
# practically indistinguishable. Only TLS settings can be specified on
# a per-backend basis.

tier roundrobin
backend-server uri=ldap://ldaphost01 starttls=critical retry=5000
               max-pending-ops=50 conn-max-pending=10
               numconns=10 bindconns=5
backend-server uri=ldap://ldaphost02 starttls=critical retry=5000
               max-pending-ops=50 conn-max-pending=10
               numconns=10 bindconns=5
backend-server uri=ldap://ldaphost03 starttls=critical retry=5000
               max-pending-ops=50 conn-max-pending=10
               numconns=10 bindconns=5
backend-server uri=ldap://ldaphost04 starttls=critical retry=5000
               max-pending-ops=50 conn-max-pending=10
               numconns=10 bindconns=5

#######################################################################
# Monitor database
#######################################################################
database        monitor