Paul Monckton

As the manager of VNU Labs I frequently face a perplexing array of seemingly worthy but dull tasks, not least of which is benchmarking.

I hate it, especially benchmarking PCs. Loading up benchmarking software is tedious. Checking and double checking configurations is excruciatingly unbearable. Then there's the seemingly endless repetition - PC after PC, group test after group test.

But this isn't why I hate benchmarking. In fact we have a semi-automated system to be obsessive and compulsive for us. The part I really don't like is the output of all this testing: the results.

Why? Because more often than not, they tell us just one thing. The PCs all perform pretty much as they should and there's no humanly-detectable difference between them, at least not in terms of raw speed.

Print the results properly, that is with all the bars starting at zero, and you end up with a chart the shape of a house brick.

The temptation is to use this brick to build a story. A PC at the top of the chart is not necessarily faster in any useful or even noticeable way than the one at the bottom.

What we want to know is what makes each PC special or different. All the graph provides is a stark representation of everything that's identical. Wow!

Let's face it, if you're buying a replacement computer, even the slowest PC in the test is likely to 'whip the ass' of your current heap of junk.

You may be interested to know by how much, but how accurately do you really need to know this? All other things being equal (price, features, styling), you may plump for the faster of two machines but it's unlikely to be your primary purchasing criterion.

Of course benchmarks have their uses. Sometimes systems don't work as they should and we can spot them and deal with them accordingly. It is possible to make an unacceptably slow PC but, thankfully, it doesn't happen very often.

My main PC is a Pentium III system. OK, it's got dual CPUs, but in benchmark terms it'll offer no challenge to anything we've tested in the lab over the last couple of years at least.

I don't play games on it and for me it's easily fast enough. Its level of performance goes up and down depending on how recently I defragged the hard drive, which pointless utilities or desktop enhancements I'm running, and the sheer unpredictability of Windows.

But, you know what? I don't really care. I have a faster PC next to me - just sitting there - but I don't use it.

What's important to me about my PC is what makes it different from other PCs and probably utterly unusable to anyone else. I, of course, have the mandatory four monitors, eye-popping optical-illusion wallpaper and blue sound-sensitive case illumination to help me fish out any bits of lunch that fall into the open case.

The fact that I haven't moved that lot to the faster PC suggests that raw performance isn't really that crucial anyway.

It's usually the parts that don't get tested that matter to me - the monitor(s), a keyboard that's coffee-resistant and as loud and annoying as possible, a reset switch that can't be knee-activated, a case that can be opened without risk of personal injury ... the list goes on.

Any decent review will contain this information but this is what I want to see at a glance, right up there with the photo.

This is why I feel uncomfortable with the current state of PC benchmarking. Not because our figures are wrong, but because it's what people do (or don't do) with them once they leave VNU Labs that worries me.

PC Magazine Best Buys

Like this story? Spread the news by clicking below:

 del.icio.us     Digg this     reddit!