4 ms·

For microbenchmarks, I've seen it recommended that you just take the min value. The idea behind it is that any extra time from the min is very likely noise so t

by slaymaker1907 9y ago

For microbenchmarks, I've seen it recommended that you just take the min value. The idea behind it is that any extra time from the min is very likely noise so the min is closest to what you want to measure.

scott_s 9y ago
I strongly disagree - I care about how the thing will perform in practice, which includes what people often consider "noise." The minimum value for a metric is just when all of the stars, sun, moon and planets lined up juuuuust right, and is often not representative of real performance. What if, for example, the technique performs great when everything is in the cache, but it also uses the data in such a way that it has horrible cache locality? Your min will be great, but your average, max and standard deviation will be terrible.
- deleted 9y ago
  [deleted]
- barrkel 9y ago
  Microbenchmarks specifically try to hold the rest of the world constant while measuring a microscopic piece of code, in the realms of countable cpu clock cycles. In that case, noise really is noise. And of course the data is of limited use. Things like cache locality, branch prediction, prefetch etc. should be flushed out with a good microbenchmark prologue.
  - scott_s 9y ago
    Again, disagreed when the point of your technique is better cache and memory hierarchy performance. Much of my own work aims to perform better given the realities of modern systems. Trying to factor such things out removes the thing being measured!
    - enqk 9y ago
      I don't think you can say that microbenchmarks are anywhere realistic. First because the machine workload is different while you run them. Second because the compiling or VM state is also quite different. A value of microbenchmarks is to establish a stable environment when a baseline can be set, where predictions can be tested. It's a development tool, not a qualification tool. You will have anyway to measure the real performance impact in the complete environment.

scott_s 9y ago

I strongly disagree - I care about how the thing will perform in practice, which includes what people often consider "noise." The minimum value for a metric is just when all of the stars, sun, moon and planets lined up juuuuust right, and is often not representative of real performance. What if, for example, the technique performs great when everything is in the cache, but it also uses the data in such a way that it has horrible cache locality? Your min will be great, but your average, max and standard deviation will be terrible.

deleted 9y ago
[deleted]
barrkel 9y ago
Microbenchmarks specifically try to hold the rest of the world constant while measuring a microscopic piece of code, in the realms of countable cpu clock cycles. In that case, noise really is noise. And of course the data is of limited use. Things like cache locality, branch prediction, prefetch etc. should be flushed out with a good microbenchmark prologue.
- scott_s 9y ago
  Again, disagreed when the point of your technique is better cache and memory hierarchy performance. Much of my own work aims to perform better given the realities of modern systems. Trying to factor such things out removes the thing being measured!
  - enqk 9y ago
    I don't think you can say that microbenchmarks are anywhere realistic. First because the machine workload is different while you run them. Second because the compiling or VM state is also quite different. A value of microbenchmarks is to establish a stable environment when a baseline can be set, where predictions can be tested. It's a development tool, not a qualification tool. You will have anyway to measure the real performance impact in the complete environment.