1. the point of doing statistics properly is to try to remove your bias about what you think the results are: every developer who has worked hard enough on a pr

1. the point of doing statistics properly is to try to remove your bias about what you think the results are: every developer who has worked hard enough on a problem, probably thinks they made an improvement. it's a good idea to test this, statistically, as you might be surprised. 2. the article didn't repeat anything, at all. it's not even on average (at least, as it's reported). as far as i can tell, he ran a test once and reported the result. 3. the point is that judging from one sample or a mean on a small sample does not answer any reasonable question because the (unrepresented) uncertainty in the answer could actually mean the result goes either way in general. btw, what i'm suggesting isn't some deep, complicated statistics! i'm suggesting the most basic, elementary statistics. this is super, super basic stuff. high school kids here know about the mean and standard deviation here. if you want to be more principled about things kind of thing, you could either do some kind of hypothesis test (e.g., a two-sample t-test adjusting for variance would be appropriate here) or use a Bayes factor to compare the two hypothese (better, not better).