RE: benchmarks &real world

["Kurt D. Zeilenga" <Kurt@OpenLDAP.org>]

At 12:17 AM 4/15/2004, John Smith wrote:
>Well, what other/better benchmark do you suggest then ? 

Well, that depends on what one is trying to measure
and why one is trying to measure it.

 From a developers' perspective, we're after improving the
software in a series of incremental changes, verifying that
each incremental change, as well as sets of incremental,
changes behavior as expected.  This requires tools which
allow us to measure small differences in overall behavior,
and often small differences in isolated behaviors.  Generally
speaking, one needs to utilize custom tools (or use existing
tools in a custom manner) as what you are trying to measure
is often different from change to change.

For evaluation purposes between multiple products, one needs
tools which reasonably mimic their real world use.  Generally
speaking, one is looking for large differences in overall
behavior here.

For marketing purposes, one needs a benchmark which favors
your product over your competitors.  Generally, one is trying
to fabricate a favorable large difference here.

Now, if you like to develop tools to aid in the development
of OpenLDAP Software, I suggest you look at the kinds of
changes being developed and devise ways of testing those
changes (individual or in sets) to see if they behave as
intended.

If you like to develop a general benchmark tool, well, feel
free.  But if you are going to truly be general, you likely
should move discussion of such to a more general forum.

And if you want to develop a tool for marketing purposes,
feel free to development something and submit it to our
marketing department (/dev/null) for consideration.

Kurt