4 ms·
I think these kinds of articles generally overstate the "schism." I wrote about this here: http://blog.keithw.org/2013/02/q-what-is-difference-between-bayesian.
by keithwinstein 11y ago
I think these kinds of articles generally overstate the "schism." I wrote about this here: http://blog.keithw.org/2013/02/q-what-is-difference-between-bayesian.html http://blog.keithw.org/2013/02/q-what-is-difference-between-... and here: http://qr.ae/70H3k7 http://qr.ae/70H3k7
Three points:
(1) I couldn't follow the author's Bayesian analysis or what "standard assumptions" were used. Does anybody know what is meant?
Here's a plausible Bayesian analysis using what are arguably standard assumptions: "assume, a priori, that the success rate of the new drug was drawn uniformly between 0 and 1. Given the outcome of 83/100 i.i.d. successes, the probability that the new drug has a success rate worse than the old drug is Integrate[PDF[BetaDistribution[84,18],x],{x,0,0.7}]. In other words, p(worse|outcome,uniform prior) = 0.0018."
Here's a plausible classical analysis: "Before doing the experiment, let's specify how to calculate the p-value at the end. For whatever outcome we get, we'll calculate the one-sided probability that the old drug produces an outcome that good or better. Now perform the experiment: our outcome is 83/100. Per the method that we pre-specified, the probability that the old drug would have produced an outcome that good or better is Sum[Binomial[100,i] (0.7)^i (1-0.7)^(100-i), {i, 83, 100}]. In other words, p<0.0022."
Here's a plausible takeaway: "What do you know, the p-value from the classical analysis and the posterior probability from the Bayesian analysis are almost the same. There isn't much difference in this case, and contrary to a point emphasized in the article, the posterior probability was actually slightly smaller. The claim that p-values overstate the certainty of findings compared with Bayesian methods is not supported and probably not true -- often the accusation is the reverse, that p-values are too conservative!"
(2) The Bayesian analysis and the classical one are trying to achieve different things. Speaking very generally, classical methods are about designing an experiment that has a cap on the rate of false positives, even in the worst-case input (and then running that experiment). Depending on who you are, this may or may not be what you really want to do. Bayesian methods are (again very generally) about calculating the conditional probability of some event, given a particular observation and well-stated prior assumptions. Again, this may or may not be what you really want to do.
The difference is sort of like the difference between saying that the running time of QuickSort is O(n^2) on adversarial input, compared with saying that the running time is O(n lg n) in expectation, assuming the input order is uniformly distributed. Both of these statements can be useful.
You don't have to pick a side and declare yourself a worstcaster or an expecterian, any more than you have to call yourself a Bayesian or a Frequentist. These are families of mathematically-sound techniques, not religions.
(3) Ultimately, statistics doesn't really matter until somebody starts making a decision based on the results. And once you start putting a cost on bad decisions and designing a decision rule to maximize utility (the domain of decision theory), methods based on p-values and posterior probabilities end up reaching THE SAME DECISIONS. This makes sense since there can really only be one utility-maximizing decision theory.
This was understood in the early 1940s when they had to decide how to balance the cost of mistakenly shooting down an Allied aircraft versus the cost of mistakenly letting a Nazi aircraft off the hook. It was understood in the late 1940s when Shannon and others worked out the mathematical theory of communication. These fields do not have squabbles about a Bayesian vs. frequentist schism.