I worked on this problem 25 years ago. It hasn't gone away. In general, performance comparisons are hard to reproduce. For instance, when benchmarking network

I worked on this problem 25 years ago. It hasn't gone away. In general, performance comparisons are hard to reproduce. For instance, when benchmarking network protocols, often a tiny change in configuration makes a big change in a results. You might change the size of a buffer from 150 packets to 151 packets and see performance double (or halve.) Instead of making measurements with some arbitrary choices for parameters, you can take lots of measurements with parameters randomly varied to show a distribution of measurements. It's hard work to track down all the possible parameters and decide on a reasonable range for them, so it's rarely done. I found many 10x variances in network protocol performance (like how fairly competing TCP streams can sharing bandwidth). The big idea was to show that by randomizing some decisions in the protocol (like discarding packets with some probability as the buffer gets full) you can make the performance less sensitive to small changes. ie, more reproducible. Less sensitivity is especially good when you care about the worst-case performance rather than average. It can also make tuning a protocol much easier, since you aren't constantly being fooled by unstable performance. Performance sensitivity analysis is hard work, so most papers are just like "we ran our new thing 3 times and got similar numbers so there you go."