Sorry, I still believe that a better approach is in https://news.ycombinator.com/item?id=10484602 https://news.ycombinator.com/item?id=10484602 That post show

Sorry, I still believe that a better approach is in https://news.ycombinator.com/item?id=10484602 https://news.ycombinator.com/item?id=10484602 That post shows that what PG is doing is a first-cut effort at a statistical hypothesis test but with being vague on assumptions and without any information on false alarm rate. In particular, in my post, get to compare sample averages without making a distribution assumption. Indeed make no distribution assumptions at all. Yes, distributions exist, but that does not mean that we have to consider their details in all applications! Come on guys, this is distribution-free statistical hypothesis testing, and we should be able to use that.