RE: benchmarks &real world

["Howard Chu" <hyc@highlandsun.com>]

> -----Original Message-----
> From: John Smith [mailto:temp201@hotmail.com]

> "Howard Chu" wrote:
> >
> >I've played with DirectoryMark a few times, after seeing it
> >bandied about.
> >In
> >fact, the data set that it uses is randomly generated, so there is no
> >canonical fixed data set. Nor is there a standardized job
> >mix or load
> >factor.
> >In articles I've seen that report DirectoryMark figures,
> >these critical
> >input
> >parameters are not included. As such, none of the published
> >results I've
> >ever
> >seen could be compared to any concrete reference, which
> >makes the usage of
> >DirectoryMark rather meaningless.

> Well, what other/better benchmark do you suggest then ?

The DirectoryMark tools are basically OK. Improving it is a simple matter of
applying the scientific method: make it consistent and repeatable.

Using a single data set for all tests would be a good step there. Since the
data set could be quite large, it would be acceptable to continue to
"randomly" generate it, as long as
	A) the pseudo-random number generation algorithm is fixed
and	B) the input seed is fixed

for all invocations of the test. (This is just a cheat to be able to
distribute a constant dataset without having to actually store it in expanded
LDIF form.)

Likewise, the hardware environment for the test must be fixed. The client
machines must be fully specified - CPU, RAM, network interface, OS version,
etc. as well as the servers. The operation mix must be fixed. Again, the job
mix can be "randomly" generated as long as the sequence can be perfectly
repeated - i.e., use the known pseudo-random number generator and input seed.
All of these parameters must be either published with the DirectoryMark
report, or everyone must agree to a canonical set of parameters so that they
can be explicitly referenced/linked to when publishing a report.

The fact that certain vendors' servers are not available on certain machines
or OS revisions complicates matters, because you can't get an
apples-to-apples comparison then. In that case, whatever numbers you get are
basically useless for product comparisons.

The notion of benchmarking is surely a well-understood topic, it surprises me
that no one working in this space has demonstrated any actual understanding
of it thus far.

  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support