3 ms·
It's a pretty serious problem with how we benchmark though. EDIT: when looking up what I vaguely remembered I somehow managed to come across a similar article
by vanderZwan 14y ago
It's a pretty serious problem with how we benchmark though.
EDIT: when looking up what I vaguely remembered I somehow managed to come across a similar article that was published just today[1], even though I was referring to an older one[2] which was about microstuttering (basically: a high standard deviation in frame rate). The point still stands - in fact it applies to both cases in somewhat different ways.
To give an example: Crossfire and SLI graphics card setups a few years ago[1]. It turned out that while both gave a similar performance increase in average framerates. Then was it discovered that one of them had a significantly lower minimum framerate than the other. A high minimum framerate is probably more important in shooters than peak performance, but that's not what we've been testing all of these years, is it? That's exactly the problem highlighted in the article by Zed.
I know this is a gaming example, but I'm sure that in user perception of the performance this matters just as much for the responsiveness of webpages.
[1] http://www.tomshardware.com/reviews/graphics-card-benchmarking-frame-rate,3466.html http://www.tomshardware.com/reviews/graphics-card-benchmarki...
[2] http://www.tomshardware.com/reviews/radeon-geforce-stutter-crossfire,2995.html http://www.tomshardware.com/reviews/radeon-geforce-stutter-c...
- spartango 14y agoIndeed, it's important to look not just at the average performance and performance extremes, but also the distribution of performance. Standard deviation helps with this. Also, often times looking at the latency at the 50, 90, and 99th percentiles is valuable as you can see events that would make your users unhappy. They're a very tangible series of metrics.