3 ms·

SPEC is a geometric mean, so 0.5% is very significant. (Assuming you are accurately measuring the parts)

chrisseaton 11y ago
Does using the geometric mean do something to the size of the error? If the geometric mean of the original implementation varies 1% under normal conditions then this is still not a significant result is it? I have benchmarks where my geomean easily varies 1% even in the most stable conditions I know to give it.
- exDM69 11y ago
  It's a benchmark engineered to measure the performance of compilers and CPUs. It's healthy to be skeptical but in general if you get a better score, it's a real improvement. You should be able to safely exclude measurement errors and other mundane issues. These benchmarks have a slight issue that they might not represent real world performance, but compilers and CPUs tend to be optimized against them - because their timing and measurement is generally more reliable than ad-hoc real world measurement. It might lead to slight "over-optimization" which doesn't reflect real performance but at least it's reliable and repeatable.
gus_massa 11y ago
To know if it's statically significant,it's not important if the benchmark use the arithmetic or the geometric mean. Did they repeat the benchmark? How many times? Is that information available? (Perhaps it's available for the master branch and not for the new commit).
- DannyBee 11y ago
  "To know if it's statically significant,it's not important if the benchmark use the arithmetic or the geometric mean. " Of course, but i was answering whether 0.5% matters, not whether it is valid. This is explicitly why i said "assuming you are accurately measuring the parts". It seems you are just making a side point for the heck of it, despite completely agreeing with what i said.
  - chrisseaton 11y ago
    I'm not sure what you mean by 'valid' or 'matters' or the difference between them though. I'm asking if it is more likely that the 0.5% difference is due to the change in the implementation than normal variation in the original implementation. I don't know how you can say that this is the case based only on the information that it's a geometric mean. You need to know the variation in the reference implementation, and you don't. The facts that he measured an increase of performance of 0.5% and he used a geometric mean are not enough information to say that it's statistically significant.
    - DannyBee 11y ago
      I understand exactly what you are saying. I'm saying you are being super pedantic here, since it's pretty obvious I meant valid to mean statistically significant, and matters took mean whether improving spec by 0.5% translates into real world application performance benefit. I'm not sure why you want to be so pedantic, but it doesn't seem worthwhile to me since nobody was confused.
      - zodiac 11y ago
        > it doesn't seem worthwhile to me since nobody was confused. fwiw I was confused about what you meant by valid and matters

Does using the geometric mean do something to the size of the error? If the geometric mean of the original implementation varies 1% under normal conditions then this is still not a significant result is it? I have benchmarks where my geomean easily varies 1% even in the most stable conditions I know to give it.

exDM69 11y ago
It's a benchmark engineered to measure the performance of compilers and CPUs. It's healthy to be skeptical but in general if you get a better score, it's a real improvement. You should be able to safely exclude measurement errors and other mundane issues. These benchmarks have a slight issue that they might not represent real world performance, but compilers and CPUs tend to be optimized against them - because their timing and measurement is generally more reliable than ad-hoc real world measurement. It might lead to slight "over-optimization" which doesn't reflect real performance but at least it's reliable and repeatable.

gus_massa 11y ago

To know if it's statically significant,it's not important if the benchmark use the arithmetic or the geometric mean. Did they repeat the benchmark? How many times? Is that information available? (Perhaps it's available for the master branch and not for the new commit).

DannyBee 11y ago
"To know if it's statically significant,it's not important if the benchmark use the arithmetic or the geometric mean. " Of course, but i was answering whether 0.5% matters, not whether it is valid. This is explicitly why i said "assuming you are accurately measuring the parts". It seems you are just making a side point for the heck of it, despite completely agreeing with what i said.
- chrisseaton 11y ago
  I'm not sure what you mean by 'valid' or 'matters' or the difference between them though. I'm asking if it is more likely that the 0.5% difference is due to the change in the implementation than normal variation in the original implementation. I don't know how you can say that this is the case based only on the information that it's a geometric mean. You need to know the variation in the reference implementation, and you don't. The facts that he measured an increase of performance of 0.5% and he used a geometric mean are not enough information to say that it's statistically significant.
  - DannyBee 11y ago
    I understand exactly what you are saying. I'm saying you are being super pedantic here, since it's pretty obvious I meant valid to mean statistically significant, and matters took mean whether improving spec by 0.5% translates into real world application performance benefit. I'm not sure why you want to be so pedantic, but it doesn't seem worthwhile to me since nobody was confused.
    - zodiac 11y ago
      > it doesn't seem worthwhile to me since nobody was confused. fwiw I was confused about what you meant by valid and matters