When you say: "A/B tests will tell you nothing you did not already know, they will just PROVE it if you need to convince someone else." you are wrong. Hypothes

When you say: "A/B tests will tell you nothing you did not already know, they will just PROVE it if you need to convince someone else." you are wrong. Hypothesis testing is a way to distinguish, with a given level of confidence, those events that happen due to chance from those that do not. It can very easily tell you something you did not know, because you mistook noise for signal for example, or because you failed to see a pattern that is obscured by noise. Furthermore hypothesis testing does not need a very large user base. You can get significant results with tens of samples, though obviously the more users you have the stronger the power of your tests.