3 ms·
> random guessing (of priors) is not a reasonable replacement for science. Priors in Bayesian statistics are not randomly guessed. The prior is supposed to ref
by pdonis 7y ago
> random guessing (of priors) is not a reasonable replacement for science.
Priors in Bayesian statistics are not randomly guessed. The prior is supposed to reflect your state of knowledge prior to looking at the data from the experiment/study/test/whatever (hence the name "prior"). For example, in the mammogram example, it is assumed that the doctor knows the base rate of breast cancer in the population (which is a very reasonable assumption since there are mountains of data on such rates) and the false positive and false negative rates of the test (which come from data on previous tests). Those are the prior. The doctor doesn't just make up those numbers; they come from prior knowledge.
> if you can't find a justifiable prior in a scientific work, using Bayes' rule just replaces one statistical fallacy for another one
In other words, if you have no prior knowledge about something, you can't expect to do statistics on some small set of data and get reliable answers. Yes, that's true. And Bayesianism tells you this is true, by telling you that you can't find a justifiable prior. (Technically, you can always find a maximum entropy prior, but in most cases that's not really any better than having no prior at all since it is basically telling you you need a lot more data before you can conclude anything.) Whereas statistical methods as they are currently done in many scientific fields simply ignore this issue completely and pick arbitrary thresholds for statistical significance. Which is the article's point.
In other words, Bayesianism, properly viewed, is not a magic bullet for extracting statistical answers from nowhere. It is a tool for exercising discipline, so that you know when you have to say "sorry, not enough data for a meaningful answer".
- mcguire 7y ago"It is a tool for exercising discipline, so that you know when you have to say "sorry, not enough data for a meaningful answer"." As opposed to the frequentist approach of saying, "We have data D and hypothesis H. If the probability of D when not-H is true is below a threshold, H is true," which hides an awful lot of assumptions. The advantage of Baysian statistics is that you have to explicitly state your assumptions in terms of a prior.
- BeetleB 7y ago>As opposed to the frequentist approach of saying, "We have data D and hypothesis H. If the probability of D when not-H is true is below a threshold, H is true," which hides an awful lot of assumptions. My textbook (what would be called a frequentist book) is very explicit: You either reject the null hypothesis, or you fail to reject the null hypothesis. A significance level test never concludes that the null hypothesis is true. It's nuanced, but is not quite the same as your statement.
- pdonis 7y agoYes, and under this description, the equivalent of the Bayesian prior is picking what the null hypothesis is. Frequentists simply fail to admit (or perhaps fail to understand) that that choice is just as much of a subjective judgment as the Bayesian choice of prior.
- BeetleB 7y ago>Frequentists simply fail to admit (or perhaps fail to understand) that that choice is just as much of a subjective judgment as the Bayesian choice of prior. Again, this claim is something I see only from self described Bayesians. I've not met a professional statistician ever fail to admit it. My textbook (written in the 90's), in at least two places, talks about it. In one place it explicitly warns against assuming the "traditional" approach is more objective, and points out that the assumptions exist in the model and the practitioner should be aware of those assumptions. And really, when it comes to p-values, I've never seen any stats textbook not give a proper description and interpretation. No statistics textbook I've read describes significance testing as a mere prescription from which you can deduce binary answers. My book discusses factors in picking your p values, and the implications, and that the appropriate value is very dependent on the problem.
- pdonis 7y agoI didn't say "picking the p-values". I said "picking the null hypothesis". (Although p-value hacking is certainly a major contributing factor to the replication crisis.) It's good that the textbooks you were taught from point out that assumptions exist and should be acknowledged.
- crdrost 7y agoIt’s also not absolutely necessary to use Bayesian reasoning to come up with a concrete final answer, but still allow for scientific style “we have a significant discovery, but it may not change our view from the status quo just yet” types of publications—significant changes in probability which nevertheless do not make unlikely things likely or vice versa. So there is a very convenient way to view Bayes’ theorem by looking at odds ratios, Pr(A)/Pr(A'), where A' is the complement of A (the event which obtains precisely when A does not), which are equivalent to probabilities but span the range from 0-∞ rather than the range 0-1: Pr(A | E) Pr(A E) Pr(E|A) Pr(A) ------------ = ---------- = ---------- x -------- Pr(A' | E) Pr(A' E) Pr(E|A') Pr(A') In other words the odds after obtaining some evidence E are a multiplicative factor times the prior odds. Taking the logarithm, one essentially can measure the impact of any new evidence in decibels as a linear result; the core of Bayes’ theorem can be stated as saying that evidence is measured in decibels. For example in the breast cancer example in the original article, you have a +13 dB evidence—Pr(E|A)/Pr(E|A') = 20—but the prior log-odds are given as -20dB and thus the shocking conclusion is that after the evidence your log-odds are still negative, -7dB, and you likely do not have cancer. For a life-or-death situation, resolving the relative nature of evidence to a specific probability is clearly essential. But in terms of p-values, if you have an experiment which gives +20dB evidence to something, I would say that this should still be meaningfully publishable as a relative increment even if it hits something whose prior is -30dB and therefore it still is only a ~1/10 chance that the thing is actually true. Like, I think that the key is having a vocabulary for relative results versus absolute results and I think that the Bayesian language can be used to take one step towards that, but perhaps it would be nice if we could have a standardized way in our language to indicate that numbers are relative rather than absolute, the way that we do habitually with decibels and pressures and such all the time...
- BeetleB 7y ago>Priors in Bayesian statistics are not randomly guessed. And frequentism (if there is such a thing) does not prohibit using base rates in their analysis. The article is setting up false dichotomies. Also, this in the article is a "random guess": >Maybe we’re not so dogmatic as to rule out “The Thinker” hypothesis altogether, but a prior probability of 1 in 1,000, somewhere between the chance of being dealt a full house and four-of-a-kind in a poker hand, could be around the right order of magnitude.