No, he's correct. This is a decision theory problem. Controlling false positives or false negatives at some arbitrary significance level like 0.05 is completely

No, he's correct. This is a decision theory problem. Controlling false positives or false negatives at some arbitrary significance level like 0.05 is completely and utterly irrelevant. All of the debates above about which proportion test to use are irrelevant because none of them answer the question about which decision (keep current template or revert) maximizes utility (revenue). The hypothetical long-run rejection rate of a particular test in a world in which there is no difference between templates (which is a priori known to be false), has little or nothing to do with that. Here, since he has no strong prior probability of the new version being bad, the switching costs are already sunk, and the evidence increases the posterior probability of the new version being better, and experimentation involves a high opportunity cost while being unlikely to yield a result indicating the new version is noticeably worse, keeping the new template probably is the correct decision.