[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: [Solved] Poor performance on Solaris

Juergen.Sprenger@swisscom.com wrote:

Hi Howard,

On-chip ethernet interfaces are currently not used for slapd,
They are currently used for management access and netbackup.
Zones running slapd use nxge type ethernet interfaces.

Did some quick tests. Increasing threads to 64 caused
higher cpu utilization but also a dramatic decrease
in performance on OpenLDAP releases 2.4.2[34].



That could not be compensated by increasing listener_threads.
Have tested values default, 4 and 8 for listener threads.
Thanks for taking the time to try this out. Yes, increasing the number of threads beyond a certain threshold typically causes a performance decrease due to inefficiencies in the system's scheduler. This is why we use a relatively low number of threads by default.
When I set up tests for maximum performance, I usually have to bind individual slapd threads to specific CPU cores to avoid this problem. (Look into the pbind or psrset commands on Solaris.) 


Staying with 32 threads and setting listener_threads to 4
Increased searches/s from 37000 to 48000.
More listener_threads had no significant impact.


30% gain, not bad.


CPU utilization by slapd was at about 65%.
If you needed to, I believe you can coax a fair bit more out of this using pbind. You generally can't get slapd above 90% utilization because the network device driver will consume the rest. (Which is why I was curious about whether the on-chip ethernet is more efficient.) I.e., on an optimally tuned setup, the network will be the performance limiter, not slapd. 

Juergen Sprenger

-----Original Message-----
From: Howard Chu [mailto:hyc@symas.com]
Sent: Monday, March 07, 2011 10:25 PM
To: Sprenger Jürgen, ITS-SDL-SO-WXS-USO-BE1
Cc: openldap-technical@openldap.org
Subject: Re: [Solved] Poor performance on Solaris

Juergen.Sprenger@swisscom.com wrote:

Hi,

first I wish to thank all those who supplied helpful hints to solve the problem, especially Quanah Gibson-Mount and Howard Chu.

My performance issue was solved by switching from memory mapped keys to shared memory keys for hdb as suggested by Quanah and Howard.
Putting 'shm_key 10' into slapd.conf and restart of slapd solved the problem.
Performance improvement was about factor 10 on the Solaris box.

As the problem was gone connection logging was switched off, which additionally doubled throughput as Solaris sylog can't do asynchronous logging.

During further tests slapd stopped responding when many concurrent connections were active.

This behaviour was caused by default settings in /etc/system.
Adding two lines to /etc/system and reboot solved the problem:
set rlim_fd_max = 16368
set rlim_fd_cur = 8192

Should be enough for about 8000 connections.

Performance before:
box1:
hardware: Sun Microsystems  sun4v SPARC Enterprise T5120
memory:32 GB RAM
os: Solaris 10 s10s_u9wos_14a
searches (avg/second): 1521

Performance after:
box1:
hardware: Sun Microsystems  sun4v SPARC Enterprise T5120
memory:32 GB RAM
os: Solaris 10 s10s_u9wos_14a
searches (avg/second): 37000


What is the CPU utilization that you see at this point? The T5120 supposedly
has 64 hardware threads, but in my tests on this architecture, we never got
better than 50% utilization (a couple years ago). I'm curious if you can make
use of the multiple listener feature added in 2.4.24. I no longer have access
to any machines of this architecture to test for myself. Also are you using
the on-chip ethernet interface?


My next step will be tuning network parameters.





--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/