Re: Slow to add 1 million items

[Dieter KlÃnter <dieter@dkluenter.de>]

Am Tue, 4 Feb 2014 10:26:40 -0200
schrieb Andrew Eross <eross@locatrix.com>:

> Hi Dieter,
> 
> I definitely read the man entries - for hdb it's more complicated, of
> course, but with mdb, my simple config came directly from the online
> docs.
> 
> My understanding is that there's really nothing needed/suggested for
> tuning parameters for mdb.
> 
> I've seen there's a 'checkpoint' option along with 'dbnosync' which
> can perhaps increase performance at the expense of reliability, but I
> wouldn't want to use that in production.
> 
> Using single inserts with auto-commit after each insert on Postgres
> on the same machine completed the same task in just under 2 hours.
> 
> I'd very much like to get my LDAP setup to add the entries in ~30 min,
> that'd be much more in-line with what I was shooting for on
> performance.

configure pool-threads, dbconfig, cachesize, idlcachsize, dncachsize,
checkpoint and dbnosync for back-bdb and for back-mdb pool-threads,
envflags, dbnosync and checkpoint.

-Dieter

> 
>  
> Andrew Eross
> CTO
> Locatrix Communications
> Office: +61 7 3123 1469
> Mobile: +55 37 9858 9815
> eross@locatrix.com
> 
> 
> On Tue, Feb 4, 2014 at 9:54 AM, Dieter KlÃnter <dieter@dkluenter.de>
> wrote:
> 
> > Am Tue, 4 Feb 2014 10:25:15 +0000
> > schrieb Chris Card <ctcard@hotmail.com>:
> >
> > > ________________________________
> > >
> > > > Hello all,
> > > >
> > > > I've been Google'ing around and searching the archives, but I
> > > > haven't quite been able to find an answer, so I wanted to ask
> > > > the list.
> > > >
> > > > I've been experimenting with OpenLDAP adds to see how quickly we
> > > > can get data inserted into the DB.
> > > >
> > > > I'm using Ubuntu 10.04, and I've tried both the packaged
> > > > OpenLDAP 2.4.21 using hdb, and just recently the latest
> > > > OpenLDAP 2.4.39 using lmdb, both with relatively similar
> > > > results.
> > > >
> > > > The short version is: to insert 1 million records, it's taking
> > > > about 8 hours on a machine with 2GB RAM / 3Ghz / SSD, which
> > > > seems like a long time to me.
> > > >
> > > > The insert method is to use a single big ldiff file like this:
> >
> > have you read slapd-hdb(5) and slapd-mdb(5) on proper database
> > configuration?
> > a slapadd of 1 mio entries takes about 30 min, depending on hardware
> > and filesystem, I haven't used ldapadd for some time, but it
> > shouldn't take much longer.
> >
> > -Dieter
> >
> > --
> > Dieter KlÃnter | Systemberatung
> > http://dkluenter.de
> > GPG Key ID: E9ED159B
> > 53Â37'09,95"N
> > 10Â08'02,42"E
> >
> >



-- 
Dieter KlÃnter | Systemberatung
http://dkluenter.de
GPG Key ID: E9ED159B
53Â37'09,95"N
10Â08'02,42"E