Re: Columnar databases

[Kristoffer Sjögren <stoffe@gmail.com>]

My idea is to have one bit index per logical column+value stored in a
value where the key also have some means of partitioning the data,
maybe over time. So high cardinality columns will generate lots of
keys+values.

I was thinking of storing multiple indexes in each key+value, making
values bigger but fewer. Say around a few hundreds kilobytes each.

Are there any trade offs between many smaller key+value vs fewer
larger ones? I'm more concerned about read performance.

On Sun, Jul 12, 2015 at 12:17 PM, Howard Chu <hyc@symas.com> wrote:
> Kristoffer Sjögren wrote:
>>
>> Hi
>>
>> I'm doing some experiments with LMDB trying to emulate a columnar
>> storage database using roaring bitmaps and other tricks.
>>
>> The initial results are promising, but I ask myself, is a row based
>> storage like LMDB appropriate for implementing a columnar database or
>> are there better, more efficient ways/formats?
>
>
> What makes you think LMDB is either row- or column-based? It has no concept
> of either, that's purely an abstraction created by higher level code.
>
> --
>   -- Howard Chu
>   CTO, Symas Corp.           http://www.symas.com
>   Director, Highland Sun     http://highlandsun.com/hyc/
>   Chief Architect, OpenLDAP  http://www.openldap.org/project/