slapd memory alloc/free functions

[Hallvard B Furuseth <h.b.furuseth@usit.uio.no>]

#define LDAP_MEMORY_DEBUG and SLAP_NO_SL_MALLOC breaks slapd in a
number of places.  Some of the fixes are straightforward, but others
require more sweeping changes - or leaving LDAP_MEMORY_DEBUG useless,
which it has been for some time anyway.

However the LDAP_MEMORY_DEBUG troubles match a Windows bug: According
to ITS#4901, Windows gets heap corruption unless mallocs done in one
DLL are balanced by frees done from the same DLL.

(I'll summarize memory management at the end, so the uninitiated can
have some clue what I'm talking about.)

The main problem is memory allocated with malloc(), either in
slapd or a library - in particular librewrite.  proto-slap.h does
  #define free ch_free
unless CH_FREE is #defined.  ch_free(malloc(x)) ends up calling
ber_memfree, and LDAP_MEMORY_DEBUG breaks ber_memfree(malloc(x)).
With Windows, the freeing DLL will be liblber instead of librewrite
or whatever.

The simplest fix looks like a global search-and-replace with malloc &
co -> ber_memalloc & co except leaving free (-> ch_free) alone in
slapd.  (Or use LDAP_MALLOC, LBER_MALLOC and SLAP_MALLOC and friends
instead.  They all expand to ber_memalloc & co.)  Is there a reason
why we use both malloc() and ber_memalloc()?  Functionality aside,
it also makes the code harder to read (and to search for bugs).

I think it should be done in librewrite at least, since it is used
to insert data into ber_memalloced data structures in slapd's.

I suppose such a change could break some Windows module which by
somehow explicitly uses _current_ free vs ber_memfree vs ber_memfree_x
correctly - in particular if it uses librewrite as a library outside
of slapd.  Should we worry?  Maybe librewrite should have its own
memory handler functions which can be overridden like liblber's.


Regarding the '#define free ch_free' in slapd, I'm not convinced that
is a good idea.  I'm sure it has saved some code which wrote free
where it should have written ch_free, but it also makes it even harder
to see what is going on - and apparently it can break slapd modules
that use malloc() - free() on Windows.  Maybe we should begin to phase
it out.  (First stage: /free/ch_free/ in slapd, #define CH_FREE in all
slapd .c files, and make the #define free ch_free in slap.h cause some
warning.  Then wait indefinitely for modules to follow suit.)


Summary of OpenLDAP memory management, as I understand it --
  Some memory is allocated with plain malloc, but most via ber_mem*()
  functions in liblber.  Of those, the ones ending with "_x" take an
  extra "context" argument.  If you pass a non-NULL context and you
  have set malloc handler functions in liblber, ber_*_x() call these
  with the context.  Otherwise they call plain malloc() co.

  slapd also has thread-local malloc heaps, used by slap_sl_malloc() &
  co.  A thread can have a context, which has a reference to the
  thread's heap.  Threads must not free/alloc in other threads' heaps.
  This avoids locks or mutexes which malloc() needs to ensure that two
  concurrent malloc()s do not corrupt malloc()'s heap.  Slapd can use
  them for memory which gets allocated and freed while handling a
  single request (which gets handled by a single thread), I think.

  slapd sets slap_sl_free() co or wrappers around them as memory
  handlers, and a special ch_free() for the free() handler.

  Since it can be a bit hard to keep track of which memory comes from
  where, ch_free() fetches the current thread's context if passed a
  NULL context.  Then it calls slap_sl_free if there is a context,
  ber_memfree otherwise.  slap_sl_free in turn checks if the memory to
  be freed came from the context's heap, otherwise it passes it on to
  ber_memfree() without a context.  With the '#define free ch_free', a
  free/ch_free/slap_sl_free/ber_memfree_x() ends up in the right
  place.  Except if you try to free another thread's memory, or use
  ber_memfree[_x]() without a context to free a thread's memory,
  or try to free something from a plain malloc() on Windows.

  Oh, and finally there are wrapper functions and macros.
  LBER_MALLOC, LDAP_MALLOC, SLAP_MALLOC and friends, which expand to
  ber_memalloc & co.  ch_malloc() and friends in slapd, which check
  that the malloc was successful and exit otherwise.  (Naughty, we
  really ought to phase out that someday.)

-- 
Regards,
Hallvard