5 ms·

> Intel i7-12700K processor Wait! You can’t benchmark on a CPU with unpredictable speed and mix of slow and fast cores. All kinds of effects come into play her

> Intel i7-12700K processor Wait! You can’t benchmark on a CPU with unpredictable speed and mix of slow and fast cores. All kinds of effects come into play here, and the code itself is not the most prominent among them. To measure which _code_ is better, you should use real SMP machine with fixed clock speed and turned off HT. On the machine from TFA you are just fighting with Intel’s thermal smarts and OS scheduling shenanigans. (edit: you can use the same machine, but configure it in BIOS. I, myself, use i9-12900k fixed to 5ghz@8P-cores as "Intel testing machine")

sspiff 3y ago
Except then you're not testing on the hardware configurations people will actually run the software on. People do run software with HyperThreading on and on E cores and with Turbo boost and throttling. Having code that behaves better in these dynamic, heterogeneous environments is a net benefit to the user.
- omgtehlion 3y ago
  It is easier to deal with all the issues separately. First your algo (in single thread), then threading, then memory accesses and cache effects. After you sort everything in your control, you try to deal with the real world (like counting only “real” cores, ignoring/or enjoying hyperthreads, OS scheduling, but most of these are quite unpredictable, and if you get your code run faster in 12700k, the same setting will be slower on, say, 5800X or on server machines, while basic stuff speeds up the program on any hardware).
  - spenczar5 3y ago
    Those issues are not separable. The performance of memory and caches can determine which algorithm is best. This is the basis of the entire field of cache-aware algorithms, which routinely beat the pants off of theoretically superior algorithms. In my experience (which is in video transcoding and in research astrophysics - both domains where it matters!), if you really need to squeeze out performance, you have to design with the target platform available for profiling and benchmarking from the beginning. Edit to add: I agree wholeheartedly with your top-level comment! I just am perhaps more extreme than you; I don’t think “laptop benchmarks” can ever be twisted into being useful.
    - the8472 3y ago
      You usually don't design an algorithm for a single CPU. Most software has to run on different tiers and generations from Intel, AMD, a whole herd of different ARM processors. Even the very hottest code paths will mostly do CPU feature detection but not "if cpu == i7-12700K { ... }"
      - spenczar5 3y ago
        I get what you are saying, but in some domains, you really do design for a single SKU, beyond even just the CPU. Supercomputers and computing centers like SLAC have a very constrained set of SKUs that your software will ever run on. I know this is not how most cloud or consumer software works, but that stuff usually isnt as performance-sensitive.
        the8472 3y ago
        There's plenty of CPU-cycle-munching work in the pro segment that does care about performance and can't assume fixed hardware. CI, compression, CGI, ...
      - pletnes 3y ago
        «Most software» - lots of software is run on cloud compute / backend servers and can be built or tuned specifically for the box/vm/container you’re running it on. I would expect AWS/Azure/Google to do that for their PaaS offerings, for cost savings.
- dijit 3y ago
  youre missing the point. you need to remove as many variables as possible when benchmarking because benchmarks are always relative to each other. in absolute terms testing is required on a standard config, but benchmarks should mostly be conducted with a minimum of external variants if you want to draw meaningful conclusions
hinkley 3y ago
I barely even try to run micro benchmarks on my work laptop, and I’m the guy who introduced them. It’s just so random. Pretty much I’m looking for a slowdown that’s more than 25% and anything less could just be noise. When you’re reaching for 5% improvements, that means you have to let the CI system tell you, and production response times are the final arbiter of truth.
- hmottestad 3y ago
  Do you have a system for running benchmarks in your CI?
  - hinkley 3y ago
    I did a reply and then I somehow lost the whole thing. Nodejs has microbenchmarking tools. They are borrowed from other languages though. Mostly what I did was cook the results to send them to telemetry. Not unlike exporting your unit test results.
adfgiknio 3y ago
Software should be tested on a system like the one that will run it. All modern machines use dynamic frequency scaling. It should rarely be disabled in production. Many modern machines have heterogeneous cores. Your testing procedure might be reproducible but it's unrealistic.
deleted 3y ago
[deleted]
- deleted 3y ago
  [deleted]

sspiff 3y ago

Except then you're not testing on the hardware configurations people will actually run the software on. People do run software with HyperThreading on and on E cores and with Turbo boost and throttling. Having code that behaves better in these dynamic, heterogeneous environments is a net benefit to the user.

omgtehlion 3y ago
It is easier to deal with all the issues separately. First your algo (in single thread), then threading, then memory accesses and cache effects. After you sort everything in your control, you try to deal with the real world (like counting only “real” cores, ignoring/or enjoying hyperthreads, OS scheduling, but most of these are quite unpredictable, and if you get your code run faster in 12700k, the same setting will be slower on, say, 5800X or on server machines, while basic stuff speeds up the program on any hardware).
- spenczar5 3y ago
  Those issues are not separable. The performance of memory and caches can determine which algorithm is best. This is the basis of the entire field of cache-aware algorithms, which routinely beat the pants off of theoretically superior algorithms. In my experience (which is in video transcoding and in research astrophysics - both domains where it matters!), if you really need to squeeze out performance, you have to design with the target platform available for profiling and benchmarking from the beginning. Edit to add: I agree wholeheartedly with your top-level comment! I just am perhaps more extreme than you; I don’t think “laptop benchmarks” can ever be twisted into being useful.
  - the8472 3y ago
    You usually don't design an algorithm for a single CPU. Most software has to run on different tiers and generations from Intel, AMD, a whole herd of different ARM processors. Even the very hottest code paths will mostly do CPU feature detection but not "if cpu == i7-12700K { ... }"
    - spenczar5 3y ago
      I get what you are saying, but in some domains, you really do design for a single SKU, beyond even just the CPU. Supercomputers and computing centers like SLAC have a very constrained set of SKUs that your software will ever run on. I know this is not how most cloud or consumer software works, but that stuff usually isnt as performance-sensitive.
      - the8472 3y ago
        There's plenty of CPU-cycle-munching work in the pro segment that does care about performance and can't assume fixed hardware. CI, compression, CGI, ...
    - pletnes 3y ago
      «Most software» - lots of software is run on cloud compute / backend servers and can be built or tuned specifically for the box/vm/container you’re running it on. I would expect AWS/Azure/Google to do that for their PaaS offerings, for cost savings.
dijit 3y ago
youre missing the point. you need to remove as many variables as possible when benchmarking because benchmarks are always relative to each other. in absolute terms testing is required on a standard config, but benchmarks should mostly be conducted with a minimum of external variants if you want to draw meaningful conclusions

hinkley 3y ago

I barely even try to run micro benchmarks on my work laptop, and I’m the guy who introduced them. It’s just so random. Pretty much I’m looking for a slowdown that’s more than 25% and anything less could just be noise. When you’re reaching for 5% improvements, that means you have to let the CI system tell you, and production response times are the final arbiter of truth.

hmottestad 3y ago
Do you have a system for running benchmarks in your CI?
- hinkley 3y ago
  I did a reply and then I somehow lost the whole thing. Nodejs has microbenchmarking tools. They are borrowed from other languages though. Mostly what I did was cook the results to send them to telemetry. Not unlike exporting your unit test results.

adfgiknio 3y ago

Software should be tested on a system like the one that will run it. All modern machines use dynamic frequency scaling. It should rarely be disabled in production. Many modern machines have heterogeneous cores. Your testing procedure might be reproducible but it's unrealistic.

deleted 3y ago

[deleted]

deleted 3y ago
[deleted]