22 ms·
My belief is that p-values are popular exactly BECAUSE they can be misunderstood so easily as the answer to the question people really want answered. Namely, "
by btilly 11y ago
My belief is that p-values are popular exactly BECAUSE they can be misunderstood so easily as the answer to the question people really want answered. Namely, "What are the odds that this is the right answer?"
I many years of experience in helping people understand A/B testing. And I have found that no matter how many times you give a correct explanation, people will have this exact misunderstanding. Repeatedly.
So if you're A/B testing, here is something to consider. If you successfully set up a testing culture where you decide at a 95% confidence level, and are running one A/B test per week, you will make an average of 2-3 mistakes per year. In the long run, mistakes are unavoidable. You will make mistakes. The only question is how often you make mistakes, and how bad those mistakes are.
P-values manage to bound how often you make mistakes with the trick of most of the time saying you couldn't figure out an answer. THIS IS USELESS FOR A BUSINESS. A business needs to decide what text to use, even if they're not confident.
Your A/B tests should always produce a usable business answer. The question of interest is how often you make bad mistakes and how bad they are. And p-values are useless for that.