Re: back-mdb - futures...

[Howard Chu <hyc@symas.com>]

Howard Chu wrote:
> The basic idea is to construct a database that is always mmap'd to a fixed
> virtual address, and which returns its mmap'd data pages directly to the
> caller (instead of copying them to a newly allocated buffer). Given a fixed
> address, it becomes feasible to make the on-disk record format identical to
> the in-memory format. Today we have to convert from a BER-like encoding into
> our in-memory format, and while that conversion is fast it still takes up a
> measurable amount of time. (Which is one reason our slapd entry cache is still
> so much faster than just using BDB's cache.) So instead of storing offsets
> into a flattened data record, we store actual pointers (since they all simply
> reside in the mmap'd space).

One stumbling block: on Little-Endian machines, of which we seem to be cursed 
with an overabundance these days, the in-memory format for integers makes a 
terrible format for database keys. Byte-swapping them between on-disk and 
in-memory would completely defeat the mmap'ing scheme. So there's two choices: 
store them Little-Endian on disk, and use a reverse-order key comparison 
function (which we did back in OpenLDAP 2.1). This would break portability of 
the database files to other machines using Big-Endian format.

The other alternative is to store them in Big-Endian format, and just use them 
in their reversed order in memory. That would allow the database files to 
remain portable and eliminate the need for alternate key comparison functions. 
But it would require a custom iterator to do in-order traversals and entryID 
sorting comparisons.

At this point I'm leaning toward the former choice: store in native byte order 
and sacrifice portability. The alternative will have too big an ipmact on 
runtime performance. With the native byte order choice, this means if you ever 
want to cluster a bunch of servers on the same database they will all need to 
use the same byte order. (And of course, the same word size, which is the same 
requirement we have today.)

(Too bad C doesn't give us a "byteswapped" data attribute; some CPU 
architectures have instructions that can load a word from memory in a byte 
order that you choose. That would make life easier here, but if your CPU was 
that smart, it probably wouldn't be using brain-damaged byte order in the 
first place. Oh well...)

> (And yes, by the way, we have planning for LDAPCon2009 this September in the
> works; I imagine the Call For Papers will go out in a week or two. So now's a
> good time to pull up whatever other ideas you've had in the back of your mind
> for a while...)

Reminder: LDAPCon2009 is just a couple weeks away!

--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/