[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: n-way multi master and mirror mode

On 12/04/2016 01:22 PM, Quanah Gibson-Mount wrote:
--On Saturday, December 03, 2016 6:42 PM -0500 Brendan Kearney <bpk678@gmail.com> wrote: 


where do i chase down the source of roughly 10k messages a week, given
that this is my home network?  do i have a bottleneck in ldap? is
bind-dyndb-ldap pounding on ldap for some unknown reason? i dont believe 
this to an appropriate amount of logs for what seems to be a low volume
network.


Hi Brendan,

a) This has nothing to do with MMR/Mirror Mode
b) It's not uncommon, and it rarely indicates a problem. Again, you would need to show that things are actually being deferred for a significant amount of time. Most often it is only for milliseconds. I.e., it's an informational message that may or may not be indicative of a real issue.
Overall, you haven't really provided a lot of the necessary information to help you move forward, such as: 

OpenLDAP release
OpenLDAP backend(s) in use
Thread settings
concurrency settings
Specific messages of the problem

etc
For example, it could be the application(s) are poorly written, and fail to use persistent connections, causing binds to be constantly be deferred. Or it could be searches being deferred, etc. I.e., we have no idea /which/ operations are being deferred because you haven't provided that information.
I personally prefer applying the patch to OpenLDAP that provides timing information for every operation so that I can actually tell whether or not real problems are occurring. If that interests you, I can point you at it. 

Hope that helps!

Regards,
Quanah




--

Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
my line of thinking was that if mirror mode was not appropriate for multi master replication, then i would remove the setting and eliminate the item from my list of potential contributing factors. if mirror mode is not needed, there are potential efforts and cpu cycles being spent on doing work that is unnecessary or even detrimental to the performance of slapd. i am looking make sure mirror mode is safe to remove from my configs, so i can test the theory.
the below shows 12 log events over a 6 second duration from the same connection. 

ReceivedAt                    SysLogtag        Message
2016-11-15 20:35:05 slapd[1514]: connection_input: conn=6836 deferring operation: too many executing 2016-11-15 20:35:05 slapd[1514]: connection_input: conn=6836 deferring operation: pending operations 2016-11-15 20:35:11 slapd[1514]: connection_input: conn=6836 deferring operation: too many executing 2016-11-15 20:35:11 slapd[1514]: connection_input: conn=6836 deferring operation: too many executing 2016-11-15 20:35:11 slapd[1514]: connection_input: conn=6836 deferring operation: too many executing 2016-11-15 20:35:11 slapd[1514]: connection_input: conn=6836 deferring operation: too many executing 2016-11-15 20:35:11 slapd[1514]: connection_input: conn=6836 deferring operation: too many executing 2016-11-15 20:35:11 slapd[1514]: connection_input: conn=6836 deferring operation: too many executing 2016-11-15 20:35:11 slapd[1514]: connection_input: conn=6836 deferring operation: too many executing 2016-11-15 20:35:11 slapd[1514]: connection_input: conn=6836 deferring operation: too many executing 2016-11-15 20:35:11 slapd[1514]: connection_input: conn=6836 deferring operation: too many executing 2016-11-15 20:35:11 slapd[1514]: connection_input: conn=6836 deferring operation: too many executing 

i am running OpenLDAP: slapd 2.4.44 on fedora 24 using lmdb.

listener threads are the default of 1, and the value is not set.
concurrency is the default value, is not set, and i dont know how to query for the current setting.
i will have to turn up the log level to stats, in order to collect the log events associated with the deferred connections. i will do that and reply later with those details. in the meantime, so details about the environment:
2 hp n54l microservers with 2.2 GHz Athlon II dual core cpu, 16 GB RAM, 250 GB Samsung SSD with a separate mount point for /var/lib/ldap that has noatime, nodiratime and noexec specified in the mount options to reduce i/o. boxes also run dns, dhcp, kerberos, sasl, squid, radius, mariadb, rsyslog over tcp, privoxy, httpd. each has a 4 port Gb NIC, bonded with increased ring buffers and a bunch of sysctl tweaks for tcp, udp and ipv4 performance. tcp low latency is also set, to favor low latency over high bandwidth. what can be load balanced, run in High Availability, etc is being done behind HAProxy or natively (dhcpd, etc).
with everything i am doing, how do i determine how many listeners and what concurrency level i should be setting, if at all? are there best practices? i am thinking that with the slapd environment validated, and tuned if need be, i have my "ducks in a row" when starting the conversation on the freeipa mailing list about the bind-dyndb-ldap package, and potential issues to work through in my configs there. 

thanks for the help, input and patience,

brendan