3 ms·

Writing (compiler) microbenchmarks is notoriously hard. Overall no benchmark should run fewer than two mins or so to account for context switching noises at lea

by xxs 6y ago

Writing (compiler) microbenchmarks is notoriously hard. Overall no benchmark should run fewer than two mins or so to account for context switching noises at least. It should run on fixed CPU frequency (no turbo boost, etc), fixed CPU core(s) too. It should be aware of L1/L2 cache sizes - data crossing L1/L2 boundaries on different processors results in disproportional results. The difference in compilers is too high and the author should just disassembly of the generated code. Imo, a bad article.

rocqua 6y ago
At what performance differential are those factors relevant? For a 5% difference I would share your sceptisism. However, the effect size is a 4x difference (3x slowdown vs 12x slowdown). To me, it seems very unlikely to be a result of inconsistent benchmarking. However, I am not an expert on the subject.
- titzer 6y ago
  Frequency scaling can absolutely make a 3x difference on small timescales.
- xxs 6y ago
  The test is too short to measure anything of use. Also you want the same code run multiple times and the results collected, at the least you'd like standard std dev between tests.
mhh__ 6y ago
I think unless you own the entire machine almost all microarchitecture level benchmarking is playing with fire because of all the things you've mentioned. Even if you use perf_event to get as low as the OS allows you to account for things that'll slow you down there's still overhead associated with that AFAIK Also, code disassembly is a fairly awful way of assessing performance unless you mean for the purposes of the article
- xxs 6y ago
  Article in this case, hence the new paragraph. I used to routinely do for Java... with -XX:+PrintAssembly to ensure certain bound checks were eliminated by enforcing the index fell within the lattice [0-array.length). A simple test that always succeeded improved zip compression by 15% for instance. Not going in details but verifying results of microbenchmarks with assembly is quite useful - of course usually limited to very hot spots. Of course about owning the hardware and doing on bare metal with fixed freq (core/uncore) + set voltage(s). Removing as many variables as possible is important just as knowing what to test and how it gets affected by the hardware.
BeeOnRope 6y ago
Two minutes!? You can write good microbenchmarks that run in as little as a few nanoseconds.

by xxs 6y ago

rocqua 6y ago

At what performance differential are those factors relevant? For a 5% difference I would share your sceptisism. However, the effect size is a 4x difference (3x slowdown vs 12x slowdown). To me, it seems very unlikely to be a result of inconsistent benchmarking. However, I am not an expert on the subject.

titzer 6y ago
Frequency scaling can absolutely make a 3x difference on small timescales.
xxs 6y ago
The test is too short to measure anything of use. Also you want the same code run multiple times and the results collected, at the least you'd like standard std dev between tests.

mhh__ 6y ago

I think unless you own the entire machine almost all microarchitecture level benchmarking is playing with fire because of all the things you've mentioned. Even if you use perf_event to get as low as the OS allows you to account for things that'll slow you down there's still overhead associated with that AFAIK Also, code disassembly is a fairly awful way of assessing performance unless you mean for the purposes of the article

xxs 6y ago
Article in this case, hence the new paragraph. I used to routinely do for Java... with -XX:+PrintAssembly to ensure certain bound checks were eliminated by enforcing the index fell within the lattice [0-array.length). A simple test that always succeeded improved zip compression by 15% for instance. Not going in details but verifying results of microbenchmarks with assembly is quite useful - of course usually limited to very hot spots. Of course about owning the hardware and doing on bare metal with fixed freq (core/uncore) + set voltage(s). Removing as many variables as possible is important just as knowing what to test and how it gets affected by the hardware.

BeeOnRope 6y ago

Two minutes!? You can write good microbenchmarks that run in as little as a few nanoseconds.