Re: updating a tree from a full dataset

[Bjorn Nordbo <bn@nextra.com>]

Daniel Tiefnig wrote:
> 
> Bjorn Nordbo wrote...:
> 
> > I have a program which reads a bunch of router configs, extracts a
> > lot of information and writes it to a file database every hour. To
> > make the world a better place to live, I'd like to store this infor-
> > mation in an LDAP directory instead. However, this makes the current
> > update-everything approach very inefficient (the file database has
> > about 1500 entries).
> 
> so you have a write-often read-once-if-at-all structure..? are you sure 
> you want to use LDAP for this..?

That's the current approach, except for that it is read quite often.
The current database is written (full copy) every hour, but read by
numerous applications all the time.

As we are moving some of the services using this database to separate
boxes, I'll have to make it available to a distributed set of appli-
cations.

In this context, maybe my investigation of OpenLDAP makes more sense.

The characteristics are:

- add/modify/delete every hour (approx 50 ops/day)
- applications reading data    (approx 1M ops/day)
- entries in the database this year:   30.000
- entries in the database next year: ~160.000
- attributes per entry: 13

(I was completely off on my previous estimate of the number of entries)

> > The approach which first comes to mind is to retrieve the entire
> > branch from LDAP, compare this to the data recently extracted from
> > the router configs and write only changed entries back to LDAP.
> 
> you may retrieve entry by entry from LDAP and use ldapmodify to write 
> back attributes/values that changed.. improves update speed 
> significantly if there aren't much changes in the router-configs, and 
> even a bit (or bunch) if there are..

Mm, I like that a lot better (programming wise) than my previous
retrieve-everything-then-compare approach. But how will this improve
the performance?

> > Are there any better approaches to this? How does it scale if every
> > entry has about 20 attributes?
> 
> your database is very small, i don't think performance will become a 
> problem here.

Ok. It will probably grow a lot, but I have a lot of things I can do
to minimize the number of read operations (it is quite high today,
at least compared to the number of entries).

Thank you very much for your help!

-- 
Bjørn Nordbø  -  Developer / IP Net NMS  -  Nextra Norway