5 ms·

Half the benchmarks i see posted here are done on AWS instances or the authors laptop with power stepping/cstates, unisolated cores etc etc likely all enabled t

Half the benchmarks i see posted here are done on AWS instances or the authors laptop with power stepping/cstates, unisolated cores etc etc likely all enabled too, let alone basic rigour around hardware specifics. We are very much entering the age where mechanical sympathy is waning

gpderetta 5y ago
Also the benchmark do not seem to use the same set of sizes for all benchmarks (nor use a fixed random seed) so repeatability and comparability seem questionable. I guess if you run it often enough it could still give useful numbers, but I understand the author is picking the best run. Also I don't see to be any attempt at avoiding compiler optimisations. Benchmarking is hard
- nadav256 5y ago
  Grep the code for rand_reset. The code uses a fixed seed, fixed sizes for all programs, stable nop baseline, etc. also the pointer indirection blocks compiler optimizations.
  - gpderetta 5y ago
    Indeed. i looked at the benchmark itself, but not the RNG.
nadav256 5y ago
I guess that you are referring to other projects because the benchmarks in this repo use a stable seed, turbo disabled, physical machine, both random and stable sizes, etc.
- ungamed 5y ago
  Havn't read the article, but c-states and p-states are different to turbo disabled.
- eternalban 5y ago
  What’s the story with the spikes for folly and your code in memcpy plots? They get chopped off and it’s not clear what number they hit. (Nice work, btw)
astrobe_ 5y ago
CPU makers add so many complicated features to let users optimize their programs that it takes too much knowledge for mere mortals to optimize their programs - That you have to rely on heuristics and measures, as it is usually and wisely advised, is a bit unsatisfying. That's sort of ironic, in a way. I used to do assembly and count cycles, but now I wouldn't dare; it's hardcore compiler and library makers stuff. It's like "don't do your own crypto (optimization)". Everyone knows why it is so, though - we cannot solve the problem by throwing more Gigahertz at it.
- cout 5y ago
  I have hand-written asm for use in production code, and I would do it again. I also knew exactly what cpu my code would be running on. If I were publishing the code to run on a variety of hardware, I would be very cautious. Writing your own crypto is very different; the stakes are higher if you get it wrong.
  - pjmlp 5y ago
    Including the microcode version currently installed into the CPU and gone through the release notes, alongside a profile like V-Tune?
- WalterBright 5y ago
  A good heuristic is simply minimize the number of instructions.
  - mhh__ 5y ago
    I doubt it'd be an issue for memcpy but minimizing the size of a loop body can lead to very counterintuitive speedups now processors have LSDs (loop stream detectors).
    - jeffbee 5y ago
      If you make them too small it can lead to counterintuitive slowdowns.
dietr1ch 5y ago
IMO there's too many things to get right and no easy standard way of doing it. Is there at least a checklist? It should ideally be automated. IIRC there's some library, maybe BLAS, that at compile time computes some machine-specific constants and also refuses to do it if a few things are not right.

gpderetta 5y ago

Also the benchmark do not seem to use the same set of sizes for all benchmarks (nor use a fixed random seed) so repeatability and comparability seem questionable. I guess if you run it often enough it could still give useful numbers, but I understand the author is picking the best run. Also I don't see to be any attempt at avoiding compiler optimisations. Benchmarking is hard

nadav256 5y ago
Grep the code for rand_reset. The code uses a fixed seed, fixed sizes for all programs, stable nop baseline, etc. also the pointer indirection blocks compiler optimizations.
- gpderetta 5y ago
  Indeed. i looked at the benchmark itself, but not the RNG.

nadav256 5y ago

I guess that you are referring to other projects because the benchmarks in this repo use a stable seed, turbo disabled, physical machine, both random and stable sizes, etc.

ungamed 5y ago
Havn't read the article, but c-states and p-states are different to turbo disabled.
eternalban 5y ago
What’s the story with the spikes for folly and your code in memcpy plots? They get chopped off and it’s not clear what number they hit. (Nice work, btw)

astrobe_ 5y ago

CPU makers add so many complicated features to let users optimize their programs that it takes too much knowledge for mere mortals to optimize their programs - That you have to rely on heuristics and measures, as it is usually and wisely advised, is a bit unsatisfying. That's sort of ironic, in a way. I used to do assembly and count cycles, but now I wouldn't dare; it's hardcore compiler and library makers stuff. It's like "don't do your own crypto (optimization)". Everyone knows why it is so, though - we cannot solve the problem by throwing more Gigahertz at it.

cout 5y ago
I have hand-written asm for use in production code, and I would do it again. I also knew exactly what cpu my code would be running on. If I were publishing the code to run on a variety of hardware, I would be very cautious. Writing your own crypto is very different; the stakes are higher if you get it wrong.
- pjmlp 5y ago
  Including the microcode version currently installed into the CPU and gone through the release notes, alongside a profile like V-Tune?
WalterBright 5y ago
A good heuristic is simply minimize the number of instructions.
- mhh__ 5y ago
  I doubt it'd be an issue for memcpy but minimizing the size of a loop body can lead to very counterintuitive speedups now processors have LSDs (loop stream detectors).
  - jeffbee 5y ago
    If you make them too small it can lead to counterintuitive slowdowns.

dietr1ch 5y ago

IMO there's too many things to get right and no easy standard way of doing it. Is there at least a checklist? It should ideally be automated. IIRC there's some library, maybe BLAS, that at compile time computes some machine-specific constants and also refuses to do it if a few things are not right.