This reminds me a different rule of three [0] as described by u/cperciva: > If you want to compare two things (e.g., "is my new code faster than my old code"),

This reminds me a different rule of three [0] as described by u/cperciva: > If you want to compare two things (e.g., "is my new code faster than my old code"), a very simple approach is to measure each three times. If the all three measurements of X are smaller than all three measurements of Y, you have X < Y with 95% confidence. > This works because the probability of the ordering XXXYYY happening by random chance is 1/(6 choose 3) = 1/20 = 5%. It's quite a weak approach -- you can get more sensitivity if you know something about the measurements (e.g., that errors are normally distributed) -- but for a quick-and-dirty verification of "this should be a big win" I find that it's very convenient. [0] https://news.ycombinator.com/item?id=18463449 https://news.ycombinator.com/item?id=18463449