[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: Slow to add 1 million items



Hi Dieter,

I definitely read the man entries - for hdb it's more complicated, of course, but with mdb, my simple config came directly from the online docs. 

My understanding is that there's really nothing needed/suggested for tuning parameters for mdb.

I've seen there's a 'checkpoint' option along with 'dbnosync' which can perhaps increase performance at the expense of reliability, but I wouldn't want to use that in production.

Using single inserts with auto-commit after each insert on Postgres on the same machine completed the same task in just under 2 hours.

I'd very much like to get my LDAP setup to add the entries in ~30 min, that'd be much more in-line with what I was shooting for on performance.

Any ideas to try very welcome.

Thank you,
Andrew




Andrew Eross
CTO
Locatrix Communications
Office: +61 7 3123 1469
Mobile: +55 37 9858 9815
eross@locatrix.com


On Tue, Feb 4, 2014 at 9:54 AM, Dieter Klünter <dieter@dkluenter.de> wrote:
Am Tue, 4 Feb 2014 10:25:15 +0000
schrieb Chris Card <ctcard@hotmail.com>:

> ________________________________
>
> > Hello all,
> >
> > I've been Google'ing around and searching the archives, but I
> > haven't quite been able to find an answer, so I wanted to ask the
> > list.
> >
> > I've been experimenting with OpenLDAP adds to see how quickly we
> > can get data inserted into the DB.
> >
> > I'm using Ubuntu 10.04, and I've tried both the packaged OpenLDAP
> > 2.4.21 using hdb, and just recently the latest OpenLDAP 2.4.39
> > using lmdb, both with relatively similar results.
> >
> > The short version is: to insert 1 million records, it's taking
> > about 8 hours on a machine with 2GB RAM / 3Ghz / SSD, which seems
> > like a long time to me.
> >
> > The insert method is to use a single big ldiff file like this:

have you read slapd-hdb(5) and slapd-mdb(5) on proper database
configuration?
a slapadd of 1 mio entries takes about 30 min, depending on hardware
and filesystem, I haven't used ldapadd for some time, but it shouldn't
take much longer.

-Dieter

--
Dieter Klünter | Systemberatung
http://dkluenter.de
GPG Key ID: E9ED159B
53°37'09,95"N
10°08'02,42"E