[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: Slow to add 1 million items

Brent Bice wrote:

On 02/05/2014 11:45 AM, Quanah Gibson-Mount wrote:

Ok.  I had some wierd perf issues with Ubuntu10 and Ubuntu12.  I was
able to resolve them on Ubuntu12 by upgrading to a newer 3.x kernel
(currently 3.10.11-031011-generic on my Ubuntu12 box).

Also, it isn't necessary to use ext4 (I use ext2 for the ldap data with
mdb).  ext4 in the 2.x kernel has some serious perf problems as well
(when using writemap).


     Yeah, what he said. :-)  It's also worth compiling from source
rather than just using whatever version of whatever binaries some distro
provides.  It's not hard and that way you have control over exactly what
version of OpenLDAP you run regardless of distro.

     BTW, I was using xfs for the OpenLDAP instance I sorta optimized for
writes that I benchmarked recently.  I think Howard and/or Quanah did a
bunch of benchmarking of the MDB back-end on a whole bunch of
filesystems and ext2 won.  I use XFS simply cuz a modern version of XFS
is decently fast, yields more usable space than ext4, and I really
wanted a journaled filesystem.  OpenLDAP on XFS was good 'nuff for my
purposes, but it's prob'ly significantly faster on ext2 (from what I read).
Journaling may be good for other uses, but it's completely redundant for LMDB. If you have an entire partition to dedicate to the purpose, you're still better off just using ext2. xfs and some of these other filesystems can be faster when allocating new space (extent-based allocation) but if you manually preallocate the space for the DB file that's also moot.
With an ext2 filesystem, mounted with noatime, and preallocated space, LMDB can run with zero metadata update overhead. That makes all writes faster and makes it immune to filesystem corruption, thus rendering fsck irrelevant. 

--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/