[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: clarifications on cachesize, preferred db, et. al. from admin guide


All-

In the email that started this thread, I asked several questions about
clarifications or updates to the Admin Guide or man pages.
Quanah's first response is below with my status updates inline.
I received further clarifications from Marco Pizzoli and Howard.

I have status updates on three of the items and a draft I would like
feedback on for one of the other items.

In regard to: Re: clarifications on cachesize, preferred db, et. al. from...:
--On Thursday, April 07, 2011 6:46 PM -0500 Tim Mooney <Tim.Mooney@ndsu.edu> wrote: 


- Admin Guide, section 21.4.1.1.  The tuning chapter is a godsend and


[snip]


Every single one of my index .bdb files is of type Btree, though, not
Hash.  Is that section of the docs outdated, and all indexed attributes
are now in Btree databases (for back-bdb and presumably back-hdb), or am
I fundamentally misunderstanding what the index-related cache calculations
are saying?


They are btrees.  You can find this is corrected in the FAQ:

<http://www.openldap.org/faq/data/cache/1075.html>


I submitted a patch last week to fix up that section of the Admin Guide,
it's ITS #6906.


If you agree that it would be useful to explicitly list which backends
would block the use of slapd-config and someone can provide me with the
list of blockers, I would be happy to file an ITS and provide a patch to
the current docs to spell things out.  I personally think it will help
adoption of slapd-config.
Check against OpenLDAP 2.4.25. I believe nearly all backends and overlays support slapd-config now. 

With Marco's tip and a bit of subsequent reading, it looks like only meta
and sql backends block the use of slapd-config.  I've updated the intro
section of both chapter 5 and chapter 6 to make it more explicit that

	* chapter 6 is about slapd.conf, which is on its way out
	* chapter 5 and chapter 6 show two different but roughly equivalent
	  ways to configure slapd.
	* only slapd-meta and slapd-sql block the use of slapd-config
	* you should never edit the slapd-config LDIF files directly, you
	  should use ldap commands instead.

The patch for the updates to the intro for chapters 5 & 6 is ITS #6905.


- man page for slapd.backends(5).  The man page entry states that
bdb is the preferred backend.  I've seen enough hints and comments on
the mailing list to suggest that it will eventually be supplanted by hdb.
How soon is that going to happen (2.5?), and is it worth mentioning that
hdb is as good as bdb now and will be the new preferred backend soon?
Again, I'll submit the ITS with the doc patch if it's worth making that
assertion in the docs now.


It was done with OpenLDAP 2.4.  The man page needs updating.


There are two separate patches as part of ITS #6904 that correct this.


- Admin Guide, chapter 21.  The tuning chapter doesn't mention the
potential benefits of using an alternate memory allocator on Linux, as
Quanah clued me in to on the mailing list last month.  Should it?  If
people feel it would be worthwhile to mention, I would be happy to write
the first draft and supply the patch in an ITS.


Probably.  It doesn't just apply to Linux.  It applies to all *nix systems.


I haven't touched this yet, probably won't get to it for a week or so.


- Admin Guide, chapter 21.  The tuning chapter doesn't mention the
potential benefits of using sysv shared memory vs. mmap'ed files on
some platforms.  Should it?  Same offer for documentation patch applies,
though I expect this one will need more feedback from the experts.


Do you mean the shared memory keys, or something else?


For tuning information on shared memory keys, I added that to the guide
as section 21.4.4, basically right after the section on the IDL cache
and before the section on threads.

The draft I have for shared memory keys is below.  I'll submit it as a
patch once I've integrated any feedback.  I'm of course most interested
in making certain the information provided is actually correct, but any
and all feedback is welcome.

------------

21.4.4. Berkeley DB shared memory regions: mmap(2) vs. System V shared
memory key

Berkeley DB uses shared memory regions to hold the per-process and
per-thread shared information for each environment. On most modern
platforms, the default for shared memory regions is to use
filesystem-backed memory, via the mmap(2) system call. It's this use of
mmap(2) that results in the creation of the __db.00N files in the database
directory for back-bdb or back-hdb.

Another option for Berkeley DB shared memory regions is to instead use
System V style interprocess communication (IPC) shared memory. On some
platforms, shared memory may be much faster than using mmap(2) for
Berkeley DB shared memory regions. This is true on Solaris 10 and earlier,
as well as some releases of the 2.6 Linux kernel.

To use a shared memory key with a particular bdb or hdb backend, specify
the shm_key configuration option with an integer key id.

       database hdb
       suffix "dc=example,dc=com"
       shm_key 21

If you have multiple bdb or hdb databases and you wish to use shared
memory for each of them, you should specify a different key id for each.
Because the key id you specify is actually the base segment ID that is
passed to the Berkeley DB set_shm_key function, there will be additional
shared memory segments created beyond the key id you specify.

 # ipcs -m

 ------ Shared Memory Segments --------
 key        shmid      owner      perms      bytes      nattch     status
 0x7402ce23 3473408    root      600        4          0
 0x7402cdf9 4161537    root      600        4          0
 0x00000000 4521986    root      644        80         2
 0x7402cdf8 4128771    root      600        4          0
 0x00000000 4554756    root      644        16384      2
 0x00000000 4587525    root      644        280        2
 0x00000015 6094854    ldap      600        24576      1
 0x00000016 6127623    ldap      600        30105600   1
 0x00000017 6160392    ldap      600        536870912  1
 0x00000018 6193161    ldap      600        2359296    1
 0x00000019 6225930    ldap      600        811008     1
 0x0000001a 6258699    ldap      600        32768      1

In the example output shown above, shared memory key 0x00000015 owned by
the ldap user corresponds to the shm_key 21 specified in the slapd
configuration. The other subsequent shared memory segments owned by the
ldap user correspond to the additional shared memory regions created
automatically by Berkeley DB.

If you elect to use a shared memory key with one or more Berkeley DB
databases, be certain the System V shared memory on your server has
suitable settings for the number of shared memory segments, the maximum
amount of memory usable system-wide for shared memory, and especially the
maximum size of any one shared memory segment. These tunables are
frequently called shmmni, shmall, and shmmax, respectively, but vary by
platform. Consult the documentation for your operating system for
information on how to configure shared memory.

----------------


Tim
--
Tim Mooney                                             Tim.Mooney@ndsu.edu
Enterprise Computing & Infrastructure                  701-231-1076 (Voice)
Room 242-J6, IACC Building                             701-231-8541 (Fax)
North Dakota State University, Fargo, ND 58105-5164