[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Antw: Re: Slow to add 1 million items
Hi!
It's complete nonsense, but did you try to put a database in a RAM disk? This
would show the delay that actual secondary storage access costs in addition to
primary storage access.
Regards,
Ulrich
>>> Andrew Eross <eross@locatrix.com> schrieb am 04.02.2014 um 21:52 in
Nachricht
<CAL_tfFdPYK-vePsyPqSya1xsjDAOqHOcS_SEjOaf-noUTXWMsw@mail.gmail.com>:
> Thanks, Dieter, Quanah.
>
> I've been doing some experimenting with those mdb options.
>
> I ran a few tests with inserting 10,000 records, wiping the DB in between,
> and changing just the one option at a time:
>
> Base-line, no extra options: 4m8s
> With "writemap" enabled: 8m55s
> With "writemap+mapasync" enabled: 5m12s
> With "dbnosync+checkpoint 0kb/1min": 0m14s
>
> Oddly I didn't have much luck with writemap, which by default actually
> slowed things down for me.
>
> The magic bullet of the bunch is dbnosync, which completed so quickly
> because it didn't even hit the 1 minute mark where it would've written to
> disk.
>
> That seems a bit risky to me, though. You'd have to be willing to have 1
> minute worth of data be theoretically expendable with that option.
>
> Cheers,
> Andrew
>
>
> Andrew Eross
> CTO
> Locatrix Communications
> Office: +61 7 3123 1469
> Mobile: +55 37 9858 9815
> eross@locatrix.com
>
>
> On Tue, Feb 4, 2014 at 2:04 PM, Dieter KlÃnter <dieter@dkluenter.de> wrote:
>
>> Am Tue, 4 Feb 2014 08:45:47 -0200
>> schrieb Andrew Eross <eross@locatrix.com>:
>>
>> > Thanks, Chris.
>> >
>> > Yeah, I hear you on that, but sorry, to be more specific, I was
>> > running this test to get an idea of performance for regular LDAP use,
>> > and slapadd is a purely offline solution.
>> >
>> > It would be helpful for a restore, of course, but not equivalent to
>> > when our application will be adding/modifying records (which is what
>> > I'm really trying to simulate).
>> >
>> > Running the same test of inserting 1M rows into postgres with the
>> > same type of data record on the same machine goes about 3x faster,
>> > which just doesn't sound right, since LDAP should be way faster than
>> > Postgres, right?
>> >
>> > Cheers,
>> > Andrew
>> >
>> >
>> >
>> >
>> > Andrew Eross
>> > CTO
>> > Locatrix Communications
>> > Office: +61 7 3123 1469
>> > Mobile: +55 37 9858 9815
>> > eross@locatrix.com
>> >
>> >
>> > On Tue, Feb 4, 2014 at 8:25 AM, Chris Card <ctcard@hotmail.com> wrote:
>> >
>> > > ________________________________
>> > >
>> > > > Hello all,
>> > > >
>> > > > I've been Google'ing around and searching the archives, but I
>> > > > haven't quite been able to find an answer, so I wanted to ask the
>> > > > list.
>> > > >
>> > > > I've been experimenting with OpenLDAP adds to see how quickly we
>> > > > can get data inserted into the DB.
>> > > >
>> > > > I'm using Ubuntu 10.04, and I've tried both the packaged OpenLDAP
>> > > > 2.4.21 using hdb, and just recently the latest OpenLDAP 2.4.39
>> > > > using lmdb, both with relatively similar results.
>> > > >
>> > > > The short version is: to insert 1 million records, it's taking
>> > > > about 8 hours on a machine with 2GB RAM / 3Ghz / SSD, which seems
>> > > > like a long time to me.
>> > > >
>> > > > The insert method is to use a single big ldiff file like this:
>>
>> You may have a look at this paper, helt at LDAPcon 2013
>>
>> http://fr.slideshare.net/ldapcon/benchmarks-on-ldap-directories
>>
>> -Dieter
>>
>> --
>> Dieter KlÃnter | Systemberatung
>> http://dkluenter.de
>> GPG Key ID: E9ED159B
>> 53Â37'09,95"N
>> 10Â08'02,42"E
>>
>>