Re: performance very differently for two hosts that are configured identical

[Zhang Weiwu <zhangweiwu@realss.com>]

Hi. Thanks for sharing your ideas and experience!

On Mon, 12 Oct 2009, Emmanuel Lecharny wrote:
> Two ideas :
>
> 1) Check the code that send requests to the prod server. It might receive 
> many more requests than expected

Why? I don't know how to check the code and already asked the web dev to 
do so without clue. I guess ldap commandline output can exclude potential 
issue from code writer. By the way the same code run on prod and dev 
server.

> 2) Check the network latency. This is most certainly what kills your 
> performances : firewalls, load balancers, etc are adding some latency you 
> don't see on your dev server because you are not using them on dev.

Couldn't be. As a new prod, no firewall nor load balancing thing running.
I just double checked iptables kernel module is not loaded.

This information might be helpful. I can manaully track down the 
slow-down happens because of two pauses.

the small pause (around 0.1 second) is marked here:

	# extended LDIF
	#
	# LDAPv3
	# base <ou=contacts,ou=realss.com,dc=eoa,dc=cn> with scope
	# subtree
	# filter: (uidNumber=7)
	# requesting: dn
	#
-> 0.1s PAUSE
	# zhangweiwu, contacts, realss.com, eoa.cn
	dn: uid=zhangweiwu,ou=contacts,ou=realss.com,dc=eoa,dc=cn

	# search result
	search: 2
	result: 0 Success

	# numResponses: 2
	# numEntries: 1


the big pause (around 0.3 second) is marked here:

-0.3s PAUSE
	# extended LDIF
	#
	# LDAPv3
	# base <ou=contacts,ou=realss.com,dc=eoa,dc=cn> with scope
	# subtree
	# filter: (uidNumber=7)
	# requesting: dn
	#

	# zhangweiwu, contacts, realss.com, eoa.cn
	dn: uid=zhangweiwu,ou=contacts,ou=realss.com,dc=eoa,dc=cn

	# search result
	search: 2
	result: 0 Success

	# numResponses: 2
	# numEntries: 1

To get these pauses I had to run ldapsearch(1) multiple times and try 
capture the pausing with my staring eyes!

Since there is a pause before each ldapsearch command I first think of 
speed of hostname resolve and time needed to establish a socket. I 
tested domain name resolve using code snipplet offered by 
http://paulschreiber.com/blog/2005/10/28/simple-gethostbyname-example/ 
and found prod is not slow at resolving its own name as given by 
hostname(1) or 'localhost', excluding resolving issue. I do not know how 
to test time needed to establish tcp/ip socket connection but a manual 
'telnet localhost ldap' does not feel slow.

This is really frustrating...