performance very differently for two hosts that are configured identical

[Zhang Weiwu <zhangweiwu@realss.com>]

We have a development server 'emerson' with roughly one third of the
performance of our productional server 'bossdog', it runs ldap query 10
times faster than the productional server. I checked everything I can to
find out the difference, it seems both are equally configured.

Test method:

emerson:~# time for (( i=0; i<100; i++)); do ldapsearch -xD cn=manager,dc=eoa,dc=cn -w masked -b ou=contacts,ou=realss.com,dc=eoa,dc=cn '(uidNumber=7)' dn; done;
[snip]
# zhangweiwu, contacts, realss.com, eoa.cn
dn: uid=zhangweiwu,ou=contacts,ou=realss.com,dc=eoa,dc=cn

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1

real	0m1.047s
user	0m0.320s
sys	0m0.352s

bossdog:~# time for (( i=0; i<100; i++)); do ldapsearch -xD cn=manager,dc=eoa,dc=cn -w masked -b ou=contacts,ou=realss.com,dc=eoa,dc=cn '(uidNumber=7)' dn; done;
[snip]

real	0m41.790s
user	0m0.232s
sys	0m0.464s


What I have compared:

   1. Both server uses Debian Lenney
   2. Both server have same DB_CONFIG in /var/lib/ldap
   3. Both server have same ACL settings. In fact we tried to reduce the
      ACL setting on the slow preforming productional server without
      improvement result.
   4. Both server uses bdb
   5. Compare syslog at log level 256 both server produce same log messages
   6. The productional server shows harddisk is twice as fast as
      development server (hdparm), free memory more than the development
      server, and its dual-core xeon cpu should outperform the VIA CPU
      used in the development server, yet slower. The productional
      server average load is 0.5 which is not high for dual-CPU (with
      hyperthreading display as 4 CPUs to Linux).
   7. Productional server DB is then re-built (by rm -rf & slapadd)
      without noticeable change in performance.

This performance difference resulted the productional server user
interface much slower than user's patience.

What would you check further in this case? Thanks in advance!