3 ms·
Nobody should do benchmarking like this anymore. Memory layout of your code has such a big impact on performance on modern computers that measuring performance
by j-pb 3y ago
Nobody should do benchmarking like this anymore.
Memory layout of your code has such a big impact on performance on modern computers that measuring performance without removing that variable leads to wild goose chases, where you think you improved something, but in reality you incidentally got the compiler to move the code around a bit.
Emery Berger has an excelent talk on this [1], and a causal profiler that they developed called Coz[2].
Branch miss-predicts might be somewhat invariant to that, but still, one of the main points of the talk is that people do too much eyeball statistics, mistaking the variance of the underlying stochastic process with actual signal.
Coz is pretty trivial to set up with Zig too.
1. https://m.youtube.com/watch?v=r-TLSBdHe1A https://m.youtube.com/watch?v=r-TLSBdHe1A
2. https://github.com/plasma-umass/coz https://github.com/plasma-umass/coz
- ororroro 3y agoSometimes you just need to check Poop to make sure your apps still healthy.
- deleted 3y ago[deleted]
- krona 3y agoCausal profiling still uses hypothesis testing, so I don't get your point. Ultimately, one needs to know if software A is 'better' or 'worse' than software A' for any given definition of those terms.
- adrian_b 3y agoCoz appears to use a very good method for profiling multi-threaded applications, but which seems to be not applicable to single-threaded applications and I do not see how it would help to avoid the variability caused by different memory layouts, which, as you rightly point out, may confound profiling results.
- dundarious 3y agoYes, just because it's from a team connected with the old Stabilizer tool, it doesn't mean it shares the insights and benefits of that kind of testing. At least, that's my understanding based on my limited knowledge of coz, and a little bit more knowledge of https://github.com/ccurtsinger/stabilizer https://github.com/ccurtsinger/stabilizer
- MattPalmer1086 3y agoWhat a great talk. I've been working on algorithm benchmarking software, and I observed that small changes to algorithm code, even in parts that would not execute, would have a large effect on execution speed. Even better they seem to have a solution to the problem. Going to explore further to see how we can use this technique.