Registered: Sep 2000
Location: Raleigh, NC
Posts: 832

Are Benchmark's Accurate

I ran a benchmark program I got somewhere just for kicks the other day. I have a Prism, which should be faster than the VDx. The speed for a VDx was 21, the same as my Prism, how accurate are these things?
By the way the name of the program was Speedy.
Sterling

Registered: Jul 2000
Location: Plant City, FL
Posts: 385

Re: Are Benchmark's Accurate

quote:

---

Originally posted by dampeoples
I ran a benchmark program I got somewhere just for kicks the other day. I have a Prism, which should be faster than the VDx. The speed for a VDx was 21, the same as my Prism, how accurate are these things?

By the way the name of the program was Speedy.

Sterling

---

Perhaps a better way to phrase the question is "are benchmarks meaningful" Writing good benchmarks is not easy. Most benchmarks are "synthetic" - meaning that they perform a contrived sequence of operations as opposed to actually timing tasks in real applications. The operations a benchmark measures must be carefully selected by the author to give meaningful indications of particular aspects of system performance. But there are a lot of different things that can be tested such as database access, computations, graphics speed, text speed, form operations, etc. and figuring out how each of these factor in to "overall" system performance is tricky. To make matters works, performance aspects that are critical to some apps may not be nearly as important to others.

For those reasons it's important for benchmarks to explain what they are testing and how they arrive at their overall numbers. For example, Kopsis' VFSMark benchmark utility shows benchmark scores for all the individual test components and tells how they all factor into the overall result. On top of that, we provide the source code for the benchmark app so that other developers can analyze what our tests are doing and tell us if there are areas that need improvement.

If a benchmark doesn't tell you what it's testing or how it's getting its numbers, and it doesn't provide source code, dump it There are plenty of good ones out there that do.

__________________
<ul><li>Dave Kessler<br>President - Kopsis, Inc.</li></ul>

Registered: Oct 2000
Location:
Posts: 380

Exactly correct

As Dave says, benchmarks are generally accurate about what they are testing, but you have to be able to evaluate how meaningful that is to your application.