4 ms·
No, it isn't. My motivation for running on EC2 was that benchmarking on a laptop is somewhat noisy due to thermal constraints [1]. I specifically used a bare-me
by Svetlitski 5y ago
No, it isn't. My motivation for running on EC2 was that benchmarking on a laptop is somewhat noisy due to thermal constraints [1]. I specifically used a bare-metal EC2 instance in an attempt to have a less noisy environment (since VM performance is less predictable and depends on the other workloads running on the same hardware) with proper cooling.
[1] https://lemire.me/blog/my-sayings/ https://lemire.me/blog/my-sayings/
Don't take my word for it, Prof. Lemire is the expert on performance.
- beebeepka 5y agoI totally buy your explanation. Consumer devices are extremely noisy. Back when I worked on embedded, the only tests that I could trust were running on the boxes
- cratermoon 5y agoI lost the link, but I remember reading an article in the last year or so about tricks for benchmarking on a laptop. Various things you could do to avoid the thermal throttling. The catch was that it was only possible to do approximate, comparative benchmarks, but never get a result that showed the "true" unconstrained performance.
- the8472 5y agoThe most important thing to do would be disabling turbo boost. On linux that's echo 1 > /sys/devices/system/cpu/intel_pstate/no_turbo # intel echo 0 > /sys/devices/system/cpu/cpufreq/boost # amd together with disabling noisy background services that's the most significant reduction in noise. There are further steps that can be taken but for CPU-bound microbenchmarks they only provide incremental improvements in my experience. IO benchmarks are trickier. SSDs have thermal throttling too, there's the OS write cache, the SLC cache performance cliff and other factors.
- cratermoon 5y agoThis discussion is starting to make me think that the industry needs benchmarks that take into account modern hardware behavior better, somehow. The only benchmarking I've done is on deployed software for comparative performance tuning, determining if a given change runs faster or slower as deployed. I've long been a skeptic of benchmarks that try to test across heterogenous systems. My dad, who was a systems programmer, helped teach me how benchmarks could be deceptive.
- pizza234 5y agoWhile that's correct, it's misleading in the big picture. On full load, any machine hits thermal constraints, so, considering clock changes a problem, a desktop is problematic as well. A desktop will certainly have a higher ceiling, and/or a laptop may have too much of a low one, but it's incorrect to assume that the problem is inherent to laptops. The solution is therefore, which is canonical in serious benchmarks and it's not mentioned, is to ensure that the machine runs on a fixed clock rate. This is easily accomplished, at least on desktops, by changing the BIOS. I wasn't personally able to do this from the O/S (Linux). Using a machine without control on the clock (which a cloud one can be, even if bare metal) is subject to generate inaccurate benchmark results.
- celrod 5y agoDepending on the scaling governor you're using, you should be able to adjust behavior from the command line: https://www.kernel.org/doc/html/v4.12/admin-guide/pm/cpufreq.html https://www.kernel.org/doc/html/v4.12/admin-guide/pm/cpufreq... https://www.kernel.org/doc/html/v4.12/admin-guide/pm/intel_pstate.html https://www.kernel.org/doc/html/v4.12/admin-guide/pm/intel_p... E.g., if you're using the Intel pstate driver in active mode, you can disable turbo via: echo "1" | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo (and re-enable by writing "0".)
- pizza234 5y agoThanks! I'll check the reference document to search for something that works. In the past, I had tried (to no avail) with the `userspace` governor, `cpufreq-set`, `cpupower` and also `/sys/*/cpufreq/scaling_max_freq`; hopefully, I'll find other info in the document.