search returns no data under load (ITS#3147)

[schaeferc@logica.com]

Full_Name: Craig Schaefer
Version: 2.1.3
OS: HPUX
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (158.234.250.66)


Our open LDAP is backed by Berkeley DB 4.0.14. 
HPUX is N-Class :
HP9000/800/N4000-44
GBM4118844
 HPUX B.11.00 64-bit 4x440Mhz CPU 3Gb 24x18Gb 

We have populated the LDAP db with a set of constrained entries (subscriber
objects). During testing the set is not added to or modified or deleted from.
Our test simply causes lookups to known subscribers at increasing rates. Thus we
start at a rate of 200 searches per second and slowly move up to 600 searches
per second. On the client side we use connection pooling as commonly documented
and it is configured as such:

-Dcom.sun.jndi.ldap.connect.pool.maxsize=100 \
       -Dcom.sun.jndi.ldap.connect.pool.initsize=50 \
       -Dcom.sun.jndi.ldap.connect.pool.timeout=60000 

Our code is very simple but is executed thread locally by up to 100 threads.
i.e. we can have 100 concurrent searches (note we are using connection
pooling):

Hashtable env = new Hashtable();
env.put("com.sun.jndi.ldap.connect.pool", "true");
env.put(Context.PROVIDER_URL, url);
env.put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.ldap.LdapCtxFactory");
DirContext ctx = new InitialDirContext(env);
...
SearchControls searchControls = new SearchControls();
...
NamingEnumeration lookupResult = ctx.search(dn, filter, searchControls)
...
ctx.close(); // throw away
...
// examine lookupResult

Our spread of subscriber records is 10000 and no search for the same subscriber
record happens when we notice our problem.

We notice that a search can return an empty NamingEnumeration one instant and
then when another thread searches for the same record some time later (minutes
later) it gets a fully populated NamingEnumeration.

Initially we thought something was wrong in the application code or the
JNDI/LDAP SPI code for JDK 1.4.1. However we turned on trace on openLDAP and
find that it is reporting nentries=0 for the problematic searches in question.
The incoming search is as expected in all cases. nentries=1 for all other
successful searches. Looking at the failed searches we double checked what we
were searching for after the load test and an LDAP browser shows these entries
as definitely present.

Why is LDAP thinking it has no data for a search one instant and then thinks
there is data for another instant?