4 ms·
I think the problem with p-values is that it trains us to think about uncertainty without nuance. It hides the inherent trade off between the cost of taking on
by twelfthnight 7y ago
I think the problem with p-values is that it trains us to think about uncertainty without nuance. It hides the inherent trade off between the cost of taking on risk and the cost of reducing uncertainty, since it sets the threshold at p=0.05. Taken to the extreme, with a large enough sample we can nearly always find significant differences between populations, the difference will just be very small and n size will be enormous.
Recently I worked with a client to interpret results from an A/B test where A performed better than B with 85% confidence (based on credible intervals, accounting for multiple comparisons). We therefore recommended A. In a group phone call, the client told her colleagues that our company doesn't know what we're talking about because 85% confidence of a difference isn't statistically significant (i.e. isn't 95% confident). We lost their business.
This was a shame because gathering the data for the experiment was expensive and the downside of making the wrong choice was low. It is often the case that taking on more risk makes more sense than hitting diminishing returns on shrinking p-values with extra sample.