back-bdb deadlocks?

[Howard Chu <hyc@symas.com>]

I've been chasing down some strange crash/deadlock situations in 
back-bdb (HEAD) which occur when the entry cache is full. To duplicate 
the problem, I'm running test008 with cachesize 4 in the slapd.conf 
file. It usually takes thousands of iterations for any problem to show up.

The deadlock isn't too much of a surprise, but I'm seeing something else 
that makes no sense at all - getting SEGV due to accessing a freed 
EntryInfo in bdb_cache_find_id. I finally caught an occurrence of this 
problem using valgrind, but it makes no sense:

==11370== Thread 15:
==11370== Invalid read of size 4
==11370==    at 0x80D1C5F: bdb_id_cmp (cache.c:237)
==11370==    by 0x80D23B3: bdb_cache_find_id (cache.c:641)
==11370==    by 0x80B24FF: bdb_do_search (search.c:923)
==11370==    by 0x80B0ED0: bdb_search (search.c:361)
==11370==  Address 0x755A8458 is 4 bytes inside a block of size 40 free'd
==11370==    at 0x7501ED79: free (vg_replace_malloc.c:127)
==11370==    by 0x8073740: ch_free (ch_malloc.c:159)
==11370==    by 0x80D1BEE: bdb_cache_entryinfo_destroy (cache.c:191)
==11370==    by 0x80D1D43: bdb_entryinfo_add_internal (cache.c:269)

The stack trace is strange because bdb_id_cmp is not called directly 
from bdb_cache_find_id; it's called from avl_find and yet avl_find is 
missing from the stack trace. Also, it's "impossible" for 
bdb_cache_find_id to see a freed EntryInfo because EntryInfo's are 
removed from the AVL tree before they are freed, and the AVL tree is 
protected by a rdwr lock.

I don't know what to think of this at this point. I've duplicated the 
problem on a Linux 2.2.25 kernel with glibc 2.3.2 and on a SuSE 9.1 
install (Linux 2.6.4, glibc 2.3.3). If anyone else can reproduce this 
problem I'd like to hear about it. You're more likely to just encounter 
a deadlock, but if you get a SEGV I'd like to see the details.
--
  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support