3 ms·

I totally buy your explanation. Consumer devices are extremely noisy. Back when I worked on embedded, the only tests that I could trust were running on the boxe

• cratermoon 5y ago
  I lost the link, but I remember reading an article in the last year or so about tricks for benchmarking on a laptop. Various things you could do to avoid the thermal throttling. The catch was that it was only possible to do approximate, comparative benchmarks, but never get a result that showed the "true" unconstrained performance.
  • the8472 5y ago
    The most important thing to do would be disabling turbo boost. On linux that's echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo # intel echo 0 > /sys/devices/system/cpu/cpufreq/boost # amd together with disabling noisy background services that's the most significant reduction in noise. There are further steps that can be taken but for CPU-bound microbenchmarks they only provide incremental improvements in my experience. IO benchmarks are trickier. SSDs have thermal throttling too, there's the OS write cache, the SLC cache performance cliff and other factors.
    • cratermoon 5y ago
      This discussion is starting to make me think that the industry needs benchmarks that take into account modern hardware behavior better, somehow. The only benchmarking I've done is on deployed software for comparative performance tuning, determining if a given change runs faster or slower as deployed. I've long been a skeptic of benchmarks that try to test across heterogenous systems. My dad, who was a systems programmer, helped teach me how benchmarks could be deceptive.