4 ms·
Is this a good method to use while trying out a lot of ideas? Usually when I am optimizing code, most of the ideas don't work out and performance remains roughl
by zawerf 8y ago
Is this a good method to use while trying out a lot of ideas? Usually when I am optimizing code, most of the ideas don't work out and performance remains roughly the same (or so I think - I don't really know and want a better workflow here).
But if you do this test repeatedly, even if the code had identical performance, you'll get a false positive 5% of the time. And depending on the spread of the timings you might not get a clear XXXYYY win even when it is indeed a minor improvement.
- ma2rten 8y agoIt is not a good method to use in that case.
- eloff 8y agosqlite famously squeezed out a ~40% performance improvement (I think from v3 to v4?) by just combining tons of micro-optimizations of this kind where it wasn't obvious if each change even made an improvement. They measured the performance with cachegrind in order to identify very small improvements that get lost in the normal measurement noise.
- zawerf 8y agoI would love to read more about this. Do you have a link?
- SQLite 8y agohttps://www.sqlite.org/cpu.html https://www.sqlite.org/cpu.html
- rytill 8y agoThat’s a really awesome idea. Also, maybe a few of the performance boosts in the final combination are mutualistic, but if measured independently would not make a difference.
- thomasahle 8y agoAlso check fishcooking: http://tests.stockfishchess.org/tests http://tests.stockfishchess.org/tests which is how Stockfish became the indisputably strongest chess engine: Everyone can commit patches which are then tested in thousands of games, and if they make Stockfish stronger, they are accepted.
- hinkley 8y agoStatistics is a tricky beast. If 3 runs gives you a 95% chance that your change is an improvement, and you use this process on ten different improvements, there's a 40% chance that at least one of them is bogus.
- hinkley 8y agoWould love to see a refutation from a downvoter.
- agonmon 8y agoI believe you have it right. Odds that all changes are improvements in 10 changes is: (.95)^10 Odds that at least one change is not improvement is: 1-(.95)^10 = 0.40 One could probably make an argument that not all trials are independent, but not sure what else down-voters saw.