3 ms·
I strongly disagree - I care about how the thing will perform in practice, which includes what people often consider "noise." The minimum value for a metric is
by scott_s 9y ago
I strongly disagree - I care about how the thing will perform in practice, which includes what people often consider "noise." The minimum value for a metric is just when all of the stars, sun, moon and planets lined up juuuuust right, and is often not representative of real performance. What if, for example, the technique performs great when everything is in the cache, but it also uses the data in such a way that it has horrible cache locality? Your min will be great, but your average, max and standard deviation will be terrible.
- deleted 9y ago[deleted]
- barrkel 9y agoMicrobenchmarks specifically try to hold the rest of the world constant while measuring a microscopic piece of code, in the realms of countable cpu clock cycles. In that case, noise really is noise. And of course the data is of limited use. Things like cache locality, branch prediction, prefetch etc. should be flushed out with a good microbenchmark prologue.
- scott_s 9y agoAgain, disagreed when the point of your technique is better cache and memory hierarchy performance. Much of my own work aims to perform better given the realities of modern systems. Trying to factor such things out removes the thing being measured!
- enqk 9y agoI don't think you can say that microbenchmarks are anywhere realistic. First because the machine workload is different while you run them. Second because the compiling or VM state is also quite different. A value of microbenchmarks is to establish a stable environment when a baseline can be set, where predictions can be tested. It's a development tool, not a qualification tool. You will have anyway to measure the real performance impact in the complete environment.