Re: Slow to add 1 million items

Thanks, Dieter, Quanah.

I've been doing some experimenting with those mdb options.

I ran a few tests with inserting 10,000 records, wiping the DB in between, and changing just the one option at a time:

Base-line, no extra options: 4m8s

With "writemap" enabled: 8m55s

With "writemap+mapasync" enabled: 5m12s

With "dbnosync+checkpoint 0kb/1min": 0m14s

Oddly I didn't have much luck with writemap, which by default actually slowed things down for me.

The magic bullet of the bunch is dbnosync, which completed so quickly because it didn't even hit the 1 minute mark where it would've written to disk.

That seems a bit risky to me, though. You'd have to be willing to have 1 minute worth of data be theoretically expendable with that option.

Cheers,

Andrew

Andrew Eross
CTO
Locatrix Communications
Office: +61 7 3123 1469

Mobile: +55 37 9858 9815
eross@locatrix.com

On Tue, Feb 4, 2014 at 2:04 PM, Dieter Klünter <dieter@dkluenter.de> wrote:

Am Tue, 4 Feb 2014 08:45:47 -0200
schrieb Andrew Eross <eross@locatrix.com>:

> Thanks, Chris.
>
> Yeah, I hear you on that, but sorry, to be more specific, I was
> running this test to get an idea of performance for regular LDAP use,
> and slapadd is a purely offline solution.
>
> It would be helpful for a restore, of course, but not equivalent to
> when our application will be adding/modifying records (which is what
> I'm really trying to simulate).
>
> Running the same test of inserting 1M rows into postgres with the
> same type of data record on the same machine goes about 3x faster,
> which just doesn't sound right, since LDAP should be way faster than
> Postgres, right?
>
> Cheers,
> Andrew
>
>
>
>
> Andrew Eross
> CTO
> Locatrix Communications
> Office: +61 7 3123 1469
> Mobile: +55 37 9858 9815
> eross@locatrix.com
>
>
> On Tue, Feb 4, 2014 at 8:25 AM, Chris Card <ctcard@hotmail.com> wrote:
>
> > ________________________________
> >
> > > Hello all,
> > >
> > > I've been Google'ing around and searching the archives, but I
> > > haven't quite been able to find an answer, so I wanted to ask the
> > > list.
> > >
> > > I've been experimenting with OpenLDAP adds to see how quickly we
> > > can get data inserted into the DB.
> > >
> > > I'm using Ubuntu 10.04, and I've tried both the packaged OpenLDAP
> > > 2.4.21 using hdb, and just recently the latest OpenLDAP 2.4.39
> > > using lmdb, both with relatively similar results.
> > >
> > > The short version is: to insert 1 million records, it's taking
> > > about 8 hours on a machine with 2GB RAM / 3Ghz / SSD, which seems
> > > like a long time to me.
> > >
> > > The insert method is to use a single big ldiff file like this:

You may have a look at this paper, helt at LDAPcon 2013

http://fr.slideshare.net/ldapcon/benchmarks-on-ldap-directories

-Dieter

--
Dieter Klünter | Systemberatung
http://dkluenter.de
GPG Key ID: E9ED159B
53°37'09,95"N
10°08'02,42"E