Re: better malloc strategies

[Howard Chu <hyc@symas.com>]

David Boreham wrote:
> Howard Chu wrote:
>
> > Kinda interesting - with hoard this shows us processing 23000 entries 
> > per second in the single-threaded case, vs only 3521 per second with 
> > glibc malloc.
>
> It is possible that you are seeing the syndrome that I wrote about here:
> http://www.bozemanpass.com/info/linux/malloc/Linux_Heap_Contention.html
> AFAIK the poorly behaving code is still present in today's glibc malloc.
> (the problem affects UMich-derived LDAP servers because entries in
> the cache comprise a significant proportion of the heap traffic, and they
> tend to get allocated by a different thread than frees them when the
> cache fills up). malloc exhibits lock contention when blocks are freed
> by a different thread than allocated them (more exactly when the threads
> hash to different arenas).

Yes, that's a factor I had considered. One of the approaches I had 
tested was tagging each entry with the threadID that allocated it, and 
deferring the frees until the owning thread comes back to dispose of 
them. I wasn't really happy with this approach because it meant the 
entry cache size would not be strictly regulated, though on a busy 
server it's likely that all of the frees would happen in reasonably 
short time.

Still, free() contention alone doesn't explain the huge performance gap 
in the single-threaded case. However, judging from the fact that the 
very first glibc search often runs faster than the second and subsequent 
ones, I'd say that there are other significant costs in glibc free(). 
Most likely would be that glibc returns memory back to the OS too 
frequently. Hoard and Umem also return memory back to the OS, but they 
also keep more extensive free lists. Tcmalloc never returns memory back 
to the OS.

What's also interesting is that for Hoard, Umem, and Tcmalloc, the 
multi-threaded query times are consistently about 2x slower than the 
single-threaded case. The 2x slowdown makes sense since it's only a 
dual-core CPU and it's doing 4x as much work. This kinda says that the 
cost of malloc is overshadowed by the overhead of thread scheduling.

But for glibc the multi-threaded times are actually faster than the 
single-threaded case. Since glibc's costs are being partially hidden it 
seems that the multiple threads are actually getting some benefit from 
the entry cache here. Still the difference between glibc and the other 
allocators is so dramatic that this little difference is just academic.
--
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  OpenLDAP Core Team            http://www.openldap.org/project/