Re: back-bdb performance

[Marijn Meijles <marijn@bitpit.net>]

You wrote:
> nodes/pointers (4 bytes per pointer). Plus 400MB in RDN storage. 560MB, not
> 5GB. Sure, I only have 256MB physical RAM in my build system, but 560MB is
> not a ridiculous requirement for a database of this size. To put this into
> perspective, how much dbcache would you need to configure to get good
> performance out of a standard backend for this size of database?
> 
My mistake ;) 560mb is reasonable. However, to read 10 million entries at
startup is still silly in my point of view. 
560mb is also without locks which you will need for a bit concurrency.

> With a dn2id-based database, look at the extreme growth rate of the dn2id
> database itself as entry counts rise and the tree depth increases. Turn that
> into disk thrashing because there's no way you could cache it all. On top of
> that, the dn2id index itself becomes next-to-useless because at most levels
> the subtree IDLs will have already overflowed and been replaced with ALLIDS.
> That means there will be no scoping on your subtree searches at all, they
> will all be based over the entire 10 million entries. A lot of your
In openldap2 this is indeed the case. We changed the scheme a long time ago,
Subtree idls suck big time. our dn2id and id2children are just O(log n)

> 
> If you've ever used a MiniDisc then you've already used such an fs. This is
> another example of a filesystem that reads the entire directory structure at
> start-time (when a disc is inserted into a drive) and keeps it completely in
> memory. The approach works extremely well, especially in cases where write
> speed to the physical media is ridiculously slow (as on optical media).
> 
True, but then again, it's all about tradeoffs. To be honest, I think
there's a better solution to this than either of us has thought of.
Maybe I'll think about it during christmas ;)


-- 
Marijn@bitpit.net
---
Why don't sheep shrink when it rains?