4 ms·
The author lost credibility by describing statistical significance as unecessary for making informed decisions. In this context, that’s just not correct.
by arthurofbabylon 8y ago
The author lost credibility by describing statistical significance as unecessary for making informed decisions.
In this context, that’s just not correct.
- gwern 8y agoNo, he's correct. This is a decision theory problem. Controlling false positives or false negatives at some arbitrary significance level like 0.05 is completely and utterly irrelevant. All of the debates above about which proportion test to use are irrelevant because none of them answer the question about which decision (keep current template or revert) maximizes utility (revenue). The hypothetical long-run rejection rate of a particular test in a world in which there is no difference between templates (which is a priori known to be false), has little or nothing to do with that. Here, since he has no strong prior probability of the new version being bad, the switching costs are already sunk, and the evidence increases the posterior probability of the new version being better, and experimentation involves a high opportunity cost while being unlikely to yield a result indicating the new version is noticeably worse, keeping the new template probably is the correct decision.