Re: High load times with mdb

[Bill MacAllister <whm@stanford.edu>]

--On Tuesday, June 25, 2013 11:06:50 AM -0700 Quanah Gibson-Mount <quanah@zimbra.com> wrote:

> --On Tuesday, June 25, 2013 10:29 AM -0700 Bill MacAllister <whm@stanford.edu> wrote:
>
> > With the release of Debian 7 (wheezy) I was rebuilding a couple test
> > systems and was surprised to find that the load times I am seeing for
> > populating the mdb database with slapd have gone up dramatically.  The
> > load for a master server that was taking about 10 minutes just took
> > 35 minutes.  The slave is worse.  A normal load time is 20 minutes
> > and it is at 31 minutes now with an eta of about 2.5 hours.  These
> > systems are using OpenLDAP 2.4.35.
> >
> > Here are some relevent bits from the configuration.
> >
> > dn: cn=config
> > olcToolThreads: 2
> >
> > dn: olcDatabase={2}mdb,cn=config
> > olcDbCheckpoint: 1024 5
> > olcDbEnvFlags: writemap
> > olcDbEnvFlags: nometasync
> > olcDbNoSync: FALSE
> > olcDbMaxSize: 85899345920
> >
> > The systems are Dell r610s with 16 gbyte of memory.  Our database
> > is currently 3.2G on the master server.
> >
> > I have been loading wheezy/2.4.35 databases for weeks now in
> > preparation upgrading the OS and installing the new version of OpenLDAP
> > on our production servers.  This is the first time I have seen this.
> >
> > I fiddled with the hardware enough to the point I don't think it is
> > a hardware problem.  There is not really much tuning to do with mdb
> > and I would appreciate some suggestions for what to look at next.
>
> If you've been doing multiple tests, you likely filled RAM.
>
> I like to:
>
> echo 3 > /proc/sys/vm/drop_caches
>
> before doing a MDB load to empty RAM out.

Thanks for the suggestion.

Tried that and still got a slow load.  Then I thought "I know how to
clear memory cache" and rebooted the system.  And the slowness
persists.  Current load of a replica is at 46m04s with an eta of
02h10m.

I have been setting swappiness to 0 on the ldap servers for years now.
I tried setting that back to the default of 60 with no discernible
change.

The load starts out at a rate of about 2 M/s.  In the past I remember
that dropping to something like 900 k/s and staying there.  Now the
load starts in the same place, but after 30 seconds it alternates
between stalling out right, and a rate under 100 k/s.  Dips as low as
under 10 k/s and sometimes as high at 700 k/s.  (My undergraduate
degree was in watching water boil.)

Bill

--

Bill MacAllister
Infrastructure Delivery Group, Stanford University