3 ms·

Plenty of other data points in the table show reduced stddev with turbo enabled. The 768x line looks like an anomaly. Without knowing how many runs were made a

Plenty of other data points in the table show reduced stddev with turbo enabled. The 768x line looks like an anomaly. Without knowing how many runs were made and the other test conditions, it's difficult to know what actually changed. There's far too much noise in the table to come to the conclusion that turbo was the culprit. We're only talking about a 30% difference in clock rate for 3.2GHz vs 4.2GHz. If there's more change than that across turbo vs. non-turbo, something else in the benchmark setup is messed up.

agallego 8y ago
~20%. The difference however is a max of ~2.3x in that direction. 100% of the code is there, what do you suggest is 'messed' up. I probably ran the benchmarks 100s of times over 7 days trying to understand. That particular I posted seemed aligned with the others, but that was the stddev of 3 runs provided by Google Benchmark --benchmark_repetitions=3 I made it 10 and not much difference there. That was the point of 768X anomalies. The values are never like that w/ capped frequencies.
- ianhowson 8y ago
  Hang on, so we've got: - SD increase of 768x - Mean increase of 1.2 to 2.3x - But n=3? You need a lot more samples to make a reasonable statement about whether Turbo had an impact or not. Noise is completely overpowering your measurements. AFAICT, you're measuring SD of the time (nanoseconds) to run your task, and mean is around 9000ns. 1ns SD is abnormally low, and 768ns SD seems about right (this is a desktop machine, after all). But I couldn't really figure this out from the article.
  - agallego 8y ago
    No no. - capped freqs: stddev 768x. in the non-turbo-to-turbo direction - in the other direction it is 2.3x. So was just acknowledging that there are 20% of samples in that table with your previous point
    - ianhowson 8y ago
      25% of 'samples' in the stddev table show absolutely no difference between Turbo and non-Turbo. Assuming that this line of argument makes sense (it doesn't) that leaves merely 55% of samples that support your case. I appreciate what you're trying to do, and I'm intensely interested in this issue (I have a feeling that we face very similar issues in our day jobs) -- but I can't see a reasonable way to interpret your data that supports your assertion that Turbo meaningfully increases variance. There's just not enough data.
      - agallego 8y ago
        I see what you are saying. I did run it 100s of times, but didn't record it. Problem with benchmarking is, that it is incredibly slow haha. I'll run it for a few hours with a couple other tips ppl sent me and have a follow up on that post.
      - BeeOnRope 8y ago
        Turbo and Hyperthreding gernate a ton of noise when benchmarking. Disabling both results in a one or two order of magnitude reduction in outlier results (maybe you want to call it noise - but it is not symmetric like classic noise). The primary way turbo introduces variance is due to the turbo transitions forced by the "max turbo ratio varies by active core count" behavior. Eg on my Skylake chip, the turbo speed is either 3.5, 3.3, 3.2 or 3.1 GHz depending on whether 1, 2, 3 or 4 cores are active. Let's say you are running a threaded benchmark with "nothing else" running on the box. Of course other cores will still occasionally fire up, to handle scheduler ticks, interrupts from you network card, background processes, whatever. Every time this happens, the chip has to immediately undergo a frequency transition which leaves it running at 3.3 GHz. The main problem isn't the lowered speed, it's the fact that the transition itself puts the chip into a halted state for 10 to 20 us, presumably to allow the multiplier transition to occur, for voltages to stabilize, etc. Especially for short, precise benchmarks, this shows up as a lot of noise. Turning off turbo fixes it, but so does just setting the max turbo ratio to the "all cores" value (3.1 GHz in my case) since then you don't have these forced transitions.
DannyBee 8y ago
Sure. It should not be that much. There is definitely some weird stuff with turbo, mind you. Particularly because using heavy AVX2 or AVX512 loads changes the max turbo frequency for the cores they are running on (in broadwell) or all cores (in haswell). https://images.anandtech.com/doci/11544/freqbehavioravx.png https://images.anandtech.com/doci/11544/freqbehavioravx.png It's quite significant for AVX512. This really screws with benchmarking unless you are on something newer than haswell and isolate the core you are executing on.

agallego 8y ago

~20%. The difference however is a max of ~2.3x in that direction. 100% of the code is there, what do you suggest is 'messed' up. I probably ran the benchmarks 100s of times over 7 days trying to understand. That particular I posted seemed aligned with the others, but that was the stddev of 3 runs provided by Google Benchmark --benchmark_repetitions=3 I made it 10 and not much difference there. That was the point of 768X anomalies. The values are never like that w/ capped frequencies.

ianhowson 8y ago
Hang on, so we've got: - SD increase of 768x - Mean increase of 1.2 to 2.3x - But n=3? You need a lot more samples to make a reasonable statement about whether Turbo had an impact or not. Noise is completely overpowering your measurements. AFAICT, you're measuring SD of the time (nanoseconds) to run your task, and mean is around 9000ns. 1ns SD is abnormally low, and 768ns SD seems about right (this is a desktop machine, after all). But I couldn't really figure this out from the article.
- agallego 8y ago
  No no. - capped freqs: stddev 768x. in the non-turbo-to-turbo direction - in the other direction it is 2.3x. So was just acknowledging that there are 20% of samples in that table with your previous point
  - ianhowson 8y ago
    25% of 'samples' in the stddev table show absolutely no difference between Turbo and non-Turbo. Assuming that this line of argument makes sense (it doesn't) that leaves merely 55% of samples that support your case. I appreciate what you're trying to do, and I'm intensely interested in this issue (I have a feeling that we face very similar issues in our day jobs) -- but I can't see a reasonable way to interpret your data that supports your assertion that Turbo meaningfully increases variance. There's just not enough data.
    - agallego 8y ago
      I see what you are saying. I did run it 100s of times, but didn't record it. Problem with benchmarking is, that it is incredibly slow haha. I'll run it for a few hours with a couple other tips ppl sent me and have a follow up on that post.
    - BeeOnRope 8y ago
      Turbo and Hyperthreding gernate a ton of noise when benchmarking. Disabling both results in a one or two order of magnitude reduction in outlier results (maybe you want to call it noise - but it is not symmetric like classic noise). The primary way turbo introduces variance is due to the turbo transitions forced by the "max turbo ratio varies by active core count" behavior. Eg on my Skylake chip, the turbo speed is either 3.5, 3.3, 3.2 or 3.1 GHz depending on whether 1, 2, 3 or 4 cores are active. Let's say you are running a threaded benchmark with "nothing else" running on the box. Of course other cores will still occasionally fire up, to handle scheduler ticks, interrupts from you network card, background processes, whatever. Every time this happens, the chip has to immediately undergo a frequency transition which leaves it running at 3.3 GHz. The main problem isn't the lowered speed, it's the fact that the transition itself puts the chip into a halted state for 10 to 20 us, presumably to allow the multiplier transition to occur, for voltages to stabilize, etc. Especially for short, precise benchmarks, this shows up as a lot of noise. Turning off turbo fixes it, but so does just setting the max turbo ratio to the "all cores" value (3.1 GHz in my case) since then you don't have these forced transitions.

DannyBee 8y ago

Sure. It should not be that much. There is definitely some weird stuff with turbo, mind you. Particularly because using heavy AVX2 or AVX512 loads changes the max turbo frequency for the cores they are running on (in broadwell) or all cores (in haswell). https://images.anandtech.com/doci/11544/freqbehavioravx.png https://images.anandtech.com/doci/11544/freqbehavioravx.png It's quite significant for AVX512. This really screws with benchmarking unless you are on something newer than haswell and isolate the core you are executing on.