4 ms·
The root problem is even simpler. The question that everyone wants from statistics is, "Is this true?" The question that data properly answers is, "Here is ho
by btilly 9y ago
The root problem is even simpler.
The question that everyone wants from statistics is, "Is this true?" The question that data properly answers is, "Here is how to update your beliefs." Therefore every incentive exists to pose questions that can produce answers which are MISUNDERSTOOD as being answers to the question we wanted answered which we couldn't.
And once a particular way of answering it has become popular, every incentive exists to push it to draw conclusions because conclusions are better than the alternative.
That's why p-values are so popular and misused. That's why you can't keep people from p-hacking their way to something publishable.
My preference is to not fight it, but accept it. Make null results publishable. Have businesses focus on decision procedures, not statistics. You can argue all day with things like https://www.evanmiller.org/how-not-to-run-an-ab-test.html https://www.evanmiller.org/how-not-to-run-an-ab-test.html that people shouldn't misuse statistics. Or you can decide the maximum number of conversions that you'll spend, call that N, stop if one version gets sqrt(N) ahead or if you get to N without a clear answer. (In the latter case, go with the one ahead as the best available answer.)
The second procedure has a rigorous statistical interpretation as "the best answer available with the effort we were willing to put in". But it isn't a definitive answer, and doesn't pretend to be. Complex conversations are avoided. Get an answer, go with it.