Re: Real alternatives to BDB?

[Buchan Milne <bgmilne@staff.telkomsa.net>]

On Wednesday 27 February 2008 18:42:05 Marten Lehmann wrote:
> Hello,
>
> we are currently running openldap 2.2.13 with a bdb-4.2.52 backend as it
> comes with RHEL4. We have about 50.000 cn's

How many cn's you have is irrelevant,  so I assume you mean entries.

> each having about 5-10 
> attributes with a length of no longer than 100 characters. So our
> dataset is not very big.

However, it is most definitely bigger that what you can run on the default BDB 
cache (256kB). You didn't provide any details of BDB tuning (the contents of 
your DB_CONFIG file, for example), so I assume you have none, in which case 
you can expect problems.

> We have a lot of concurrent reads since or ldap provides the data for
> incoming and pop3/imap servers. But we have only about 100 changes/adds
> a day, not more.

We briefly ran 2.2.13 (before I joined, and for a short time thereafter), and 
there weren't huge problems, on a database with (at that time) ~ 800 000 
entries, each server doing > 100 operations/sec, with about 2 writes/sec.

We currently run 2.3.40 (plus one patch ...) with 1.5 million entries, each 
slave doing > 200 operations/sec, with about 4 writes/sec.

In neither case have we had significant problems. But, we always had an 
appropriately tuned DB_CONFIG 

> Yesterday we had the second bdb problem in our 1 year ldap setup. The
> problem is, that we cannot really detect the crash, since ldap queries
> just hang, but neither they timeout nor openldap is crashing. We just
> notice the mailserver issues and have to track it down to openldap and
> bdb.

So, on a 50,000 user mail system, you don't monitor all the components of the 
mail system?

> As with last time, we had to shutdown openldap and recover bdb.

The 2nd may be as a result of the first, which could be as a result of your 
lack of tuning, or a bug.

> Fortunately we have been able to recover the data-directory both times
> using dv_recover.

Red Hat should have included automatic database recovery in their init script, 
as other a number of other distributions did for 2.1 and 2.2. Then you would 
probably not have noticed much ...

> But in fact, we want to have a backend, that doesn't 
> crash twice a year.

Are you sure it was the backend? Or, was the fact that you needed to recover 
the result of shutting down slapd uncleanly?

> And as our setup is growing and we want to start 
> with replication, we want a backend to whom we can trust.
>
> I read through the backend documentation of openldap and berkeley db is
> praised as very fast and efficient. Amazing, that berkeleydb is not
> called stable anywhere in the documentation. And from my experience, bdb
> is not stable. Whenever you here about bdb, you here about it because a
> database went corrupt. bdb is just a key/value database that seems to
> work fine as long it is read-only. Different projects, e.g. subversion,
> have turned away from bdb and use a different backend.

You are assuming that the reason they stopped using BDB was problems in BDB 
itself, whereas it could have been in the way the software used the 
library ...

>
> Often they use SQLite. Since SQLite handles complete table layouts, it
> would also be possible to create one table with two columns (key and
> value as in bdb). SQLite shall also be transactional.

BDB is transactional. OpenLDAP has no need of SQL support etc. AFAIK, the 
biggest driver for people to use SQLite is related to licensing issues.

> But why is openldap sticking on bdb? Does bdb have any other important
> features the SQLite doesn't have? Benchmark issues? Replication?

OpenLDAP doesn't use BDB replication features.

> I have to admit, that I'm not using the latest releases of bdb. But I'm
> using and watching bdb for years and I hardly believe, that it has
> become stable in the latest release.

I really don't think it is beneficial for you to criticise the software ... if 
you want solutions to your problems, provide more detail on your 
configuration, and maybe you will have a quicker resolution ...

You may want to upgrade. This may not fix all your issues (as upgrading 
doesn't magically tune your configuration for you), however:
-There have been lots of fixes since 2.2.13
-Features have been added that would mitigate your problems, even if you 
didn't do tuning (such as slapd doing automatic database recovery when 
necessary)
-Warning messages when no DB_CONFIG file is found
etc.

Upgrading may not be very much work ... you may want to look at:

http://staff.telkomsa.net/packages/rhel4/openldap/
http://staff.telkomsa.net/packages/ (for a .repo file for yum, if you use yum)


However, I am curious as to why this question ("Why does OpenLDAP on RHEL4 
suck") gets asked so often. How did you come to the conclusion that BDB 
should be dropped in favour of some other database library? What searches did 
you do to find solutions to your problem? Maybe we can improve the content on 
the FAQ-o-matic to ensure people don't end up in your situation, or at least 
find real solutions sooner.

Regards,
Buchan