Re: performance very differently for two hosts that are configured identical

[Zhang Weiwu <zhangweiwu@realss.com>]

I found the problem. Stupid problem, this took us several days!

Since it has been so many days, I quote the full original question for
your reference, see the bottom for the last jigsaw of the puzzle

Zhang Weiwu wrote:
> We have a development server 'emerson' with roughly one third of the
> performance of our productional server 'bossdog', it runs ldap query 10
> times faster than the productional server. I checked everything I can to
> find out the difference, it seems both are equally configured.
>
> Test method:
>
> emerson:~# time for (( i=0; i<100; i++)); do ldapsearch -xD cn=manager,dc=eoa,dc=cn -w masked -b ou=contacts,ou=realss.com,dc=eoa,dc=cn '(uidNumber=7)' dn; done;
> [snip]
> # zhangweiwu, contacts, realss.com, eoa.cn
> dn: uid=zhangweiwu,ou=contacts,ou=realss.com,dc=eoa,dc=cn
>
> # search result
> search: 2
> result: 0 Success
>
> # numResponses: 2
> # numEntries: 1
>
> real	0m1.047s
> user	0m0.320s
> sys	0m0.352s
>
> bossdog:~# time for (( i=0; i<100; i++)); do ldapsearch -xD cn=manager,dc=eoa,dc=cn -w masked -b ou=contacts,ou=realss.com,dc=eoa,dc=cn '(uidNumber=7)' dn; done;
> [snip]
>
> real	0m41.790s
> user	0m0.232s
> sys	0m0.464s
>
>
> What I have compared:
>
>    1. Both server uses Debian Lenney
>    2. Both server have same DB_CONFIG in /var/lib/ldap
>    3. Both server have same ACL settings. In fact we tried to reduce the
>       ACL setting on the slow preforming productional server without
>       improvement result.
>    4. Both server uses bdb
>    5. Compare syslog at log level 256 both server produce same log messages
>    6. The productional server shows harddisk is twice as fast as
>       development server (hdparm), free memory more than the development
>       server, and its dual-core xeon cpu should outperform the VIA CPU
>       used in the development server, yet slower. The productional
>       server average load is 0.5 which is not high for dual-CPU (with
>       hyperthreading display as 4 CPUs to Linux).
>    7. Productional server DB is then re-built (by rm -rf & slapadd)
>       without noticeable change in performance.
>
> This performance difference resulted the productional server user
> interface much slower than user's patience.
>
> What would you check further in this case? Thanks in advance

The problem is we have been looking into openldap and system
resource/kernel so much that we forgot other software directly related
to performance. There had been something wrong made syslog low
performance, change loglevel to none instantly solves the problem.