3 ms·

Except then you're not testing on the hardware configurations people will actually run the software on. People do run software with HyperThreading on and on E

by sspiff 3y ago

Except then you're not testing on the hardware configurations people will actually run the software on. People do run software with HyperThreading on and on E cores and with Turbo boost and throttling. Having code that behaves better in these dynamic, heterogeneous environments is a net benefit to the user.

omgtehlion 3y ago
It is easier to deal with all the issues separately. First your algo (in single thread), then threading, then memory accesses and cache effects. After you sort everything in your control, you try to deal with the real world (like counting only “real” cores, ignoring/or enjoying hyperthreads, OS scheduling, but most of these are quite unpredictable, and if you get your code run faster in 12700k, the same setting will be slower on, say, 5800X or on server machines, while basic stuff speeds up the program on any hardware).
- spenczar5 3y ago
  Those issues are not separable. The performance of memory and caches can determine which algorithm is best. This is the basis of the entire field of cache-aware algorithms, which routinely beat the pants off of theoretically superior algorithms. In my experience (which is in video transcoding and in research astrophysics - both domains where it matters!), if you really need to squeeze out performance, you have to design with the target platform available for profiling and benchmarking from the beginning. Edit to add: I agree wholeheartedly with your top-level comment! I just am perhaps more extreme than you; I don’t think “laptop benchmarks” can ever be twisted into being useful.
  - the8472 3y ago
    You usually don't design an algorithm for a single CPU. Most software has to run on different tiers and generations from Intel, AMD, a whole herd of different ARM processors. Even the very hottest code paths will mostly do CPU feature detection but not "if cpu == i7-12700K { ... }"
    - spenczar5 3y ago
      I get what you are saying, but in some domains, you really do design for a single SKU, beyond even just the CPU. Supercomputers and computing centers like SLAC have a very constrained set of SKUs that your software will ever run on. I know this is not how most cloud or consumer software works, but that stuff usually isnt as performance-sensitive.
      - the8472 3y ago
        There's plenty of CPU-cycle-munching work in the pro segment that does care about performance and can't assume fixed hardware. CI, compression, CGI, ...
    - pletnes 3y ago
      «Most software» - lots of software is run on cloud compute / backend servers and can be built or tuned specifically for the box/vm/container you’re running it on. I would expect AWS/Azure/Google to do that for their PaaS offerings, for cost savings.
dijit 3y ago
youre missing the point. you need to remove as many variables as possible when benchmarking because benchmarks are always relative to each other. in absolute terms testing is required on a standard config, but benchmarks should mostly be conducted with a minimum of external variants if you want to draw meaningful conclusions

by sspiff 3y ago

omgtehlion 3y ago

It is easier to deal with all the issues separately. First your algo (in single thread), then threading, then memory accesses and cache effects. After you sort everything in your control, you try to deal with the real world (like counting only “real” cores, ignoring/or enjoying hyperthreads, OS scheduling, but most of these are quite unpredictable, and if you get your code run faster in 12700k, the same setting will be slower on, say, 5800X or on server machines, while basic stuff speeds up the program on any hardware).

spenczar5 3y ago
Those issues are not separable. The performance of memory and caches can determine which algorithm is best. This is the basis of the entire field of cache-aware algorithms, which routinely beat the pants off of theoretically superior algorithms. In my experience (which is in video transcoding and in research astrophysics - both domains where it matters!), if you really need to squeeze out performance, you have to design with the target platform available for profiling and benchmarking from the beginning. Edit to add: I agree wholeheartedly with your top-level comment! I just am perhaps more extreme than you; I don’t think “laptop benchmarks” can ever be twisted into being useful.
- the8472 3y ago
  You usually don't design an algorithm for a single CPU. Most software has to run on different tiers and generations from Intel, AMD, a whole herd of different ARM processors. Even the very hottest code paths will mostly do CPU feature detection but not "if cpu == i7-12700K { ... }"
  - spenczar5 3y ago
    I get what you are saying, but in some domains, you really do design for a single SKU, beyond even just the CPU. Supercomputers and computing centers like SLAC have a very constrained set of SKUs that your software will ever run on. I know this is not how most cloud or consumer software works, but that stuff usually isnt as performance-sensitive.
    - the8472 3y ago
      There's plenty of CPU-cycle-munching work in the pro segment that does care about performance and can't assume fixed hardware. CI, compression, CGI, ...
  - pletnes 3y ago
    «Most software» - lots of software is run on cloud compute / backend servers and can be built or tuned specifically for the box/vm/container you’re running it on. I would expect AWS/Azure/Google to do that for their PaaS offerings, for cost savings.

dijit 3y ago

youre missing the point. you need to remove as many variables as possible when benchmarking because benchmarks are always relative to each other. in absolute terms testing is required on a standard config, but benchmarks should mostly be conducted with a minimum of external variants if you want to draw meaningful conclusions