8 ms·

The article says: > The main reason scientists have historically been resistant to using Bayesian inference instead is that they are afraid of being accused of

The article says: > The main reason scientists have historically been resistant to using Bayesian inference instead is that they are afraid of being accused of subjectivity. The prior probabilities required for Bayes’ rule feel like an unseemly breach of scientific ethics. Where do these priors come from? It's not just "being afraid"; the problem is that random guessing (of priors) is not a reasonable replacement for science. Bayes' rule is great if you can find a reasonable justification for a prior. Bayes is widely used for decision-making (for example), where you need an answer quickly & you aren't trying to make general scientific claims. But if you can't find a justifiable prior in a scientific work, using Bayes' rule just replaces one statistical fallacy for another one. After all, Bayes' rule will give nonsense answers if you give it a nonsense prior! Bayes' rule is a great tool in many circumstances! But it has a great weakness: it requires a prior. That doesn't make it useless; few tools are useful in all circumstances. But requiring "everyone to use Bayes' rule, even though we have no reasonable way to find a good estimate of the priors," is unlikely to ever happen (and rightly so). The article rightly points out a serious problem with the typical application of statistics, but there needs to be a better justification for priors than is suggested in this article. I could imagine systemic worldwide ways to deal with this. For example, perhaps the scientific community could allow people to propose initial priors, and then allow multiple different papers to improve the estimation of the probability over time. But that would require much more than articles repeatedly saying "there's no serious problem with priors"; having a justifiable way to estimate and update priors is the fundamental problem with Bayesian analysis in the scientific community.

cwyers 7y ago
> Bayes' rule is a great tool in many circumstances! But it has a great weakness: it requires a prior. That doesn't make it useless; few tools are useful in all circumstances. But requiring "everyone to use Bayes' rule, even though we have no reasonable way to find a good estimate of the priors," is unlikely to ever happen (and rightly so). The article rightly points out a serious problem with the typical application of statistics, but there needs to be a better justification for priors than is suggested in this article. If you don't have good information on what your prior should be, you can just use a really weak prior. And you probably know more about your data than you think you do -- if you are doing counts, that leads you to a different prior than if you are doing things that can be in increments smaller than a single integer.
kansface 7y ago
> the problem is that random guessing (of priors) is not a reasonable replacement for science. I disagree! > After all, Bayes' rule will give nonsense answers if you give it a nonsense prior! You are absolutely correct here, but draw the wrong conclusion. Bayes rule isn't just some convenience that you ought to ignore without a solid prior. Fundamentally, what can any study say about a subject whose prior likely ranges over a few orders of magnitude among the experts? That subject represents a gaping hole in our knowledge. If a single study were to narrow the band by a whole decade, it would still leave a decade sized hole... All of this should be explicit and out in the open. > I could imagine systemic worldwide ways to deal with this. For example, perhaps the scientific community could allow people to propose initial priors, and then allow multiple different papers to improve the estimation of the probability over time. Yeah, this is the way to go. Researchers should preregister their studies with their priors. Future studies will revise them. After all, its easy to argue about numbers and those who disagree can easily supply their own.
- DataWorker 7y ago
  This ignores the labor market economics involved in research. If there was lots of pre-registration and always multiple replication attempts then there wouldn’t be much of a replication crisis in the first place. It’s not frequentist stats that’s to blame, it’s the labor market for researchers.
  - kansface 7y ago
    I appreciate your point, but surely both are problems? One is bad math while the other is a sort of a prisoners dilemma where everyone has already defected. I don't think changing the incentives would necessarily lead to better math or even that we have to change the incentives before changing the math.
  - tomp 7y ago
    No, the problem is much wider than just researchers. If for example the public, or at least the media, didn’t buy the bullshit they were selling (with high p values, non-public data and other shoddy statistical hacks), there wouldn’t be any replication crisis either. The general problem is that just stamping the word “science” on a black box that smells of poo gives it much more credibility than it should, and it often also prevents people from thinking critically.
pdonis 7y ago
> random guessing (of priors) is not a reasonable replacement for science. Priors in Bayesian statistics are not randomly guessed. The prior is supposed to reflect your state of knowledge prior to looking at the data from the experiment/study/test/whatever (hence the name "prior"). For example, in the mammogram example, it is assumed that the doctor knows the base rate of breast cancer in the population (which is a very reasonable assumption since there are mountains of data on such rates) and the false positive and false negative rates of the test (which come from data on previous tests). Those are the prior. The doctor doesn't just make up those numbers; they come from prior knowledge. > if you can't find a justifiable prior in a scientific work, using Bayes' rule just replaces one statistical fallacy for another one In other words, if you have no prior knowledge about something, you can't expect to do statistics on some small set of data and get reliable answers. Yes, that's true. And Bayesianism tells you this is true, by telling you that you can't find a justifiable prior. (Technically, you can always find a maximum entropy prior, but in most cases that's not really any better than having no prior at all since it is basically telling you you need a lot more data before you can conclude anything.) Whereas statistical methods as they are currently done in many scientific fields simply ignore this issue completely and pick arbitrary thresholds for statistical significance. Which is the article's point. In other words, Bayesianism, properly viewed, is not a magic bullet for extracting statistical answers from nowhere. It is a tool for exercising discipline, so that you know when you have to say "sorry, not enough data for a meaningful answer".
- mcguire 7y ago
  "It is a tool for exercising discipline, so that you know when you have to say "sorry, not enough data for a meaningful answer"." As opposed to the frequentist approach of saying, "We have data D and hypothesis H. If the probability of D when not-H is true is below a threshold, H is true," which hides an awful lot of assumptions. The advantage of Baysian statistics is that you have to explicitly state your assumptions in terms of a prior.
  - BeetleB 7y ago
    >As opposed to the frequentist approach of saying, "We have data D and hypothesis H. If the probability of D when not-H is true is below a threshold, H is true," which hides an awful lot of assumptions. My textbook (what would be called a frequentist book) is very explicit: You either reject the null hypothesis, or you fail to reject the null hypothesis. A significance level test never concludes that the null hypothesis is true. It's nuanced, but is not quite the same as your statement.
edflsafoiewq 7y ago
Why can't the result itself be the map prior->posterior (instead of the result of applying this map to a fixed prior)?
- comex 7y ago
  Well, that’s essentially what frequentist statistics gives you: the probability of the results given the hypothesis, which Bayes’ rule lets you combine with a prior to get the probability of the hypothesis given the results. But people don’t usually actually perform that calculation...
  - 6gvONxR4sf7o 7y ago
    >Well, that’s essentially what frequentist statistics gives you: the probability of the results given the hypothesis No. The whole point of NHST is that it gives the probability of the results given the null hypothesis. It's whole point is that it doesn't touch probability of the results given other hypotheses (except when designing the experiment, where you need other subjective inputs).
- jbay808 7y ago
  This is known as the "evidence". Usually we expect scientific papers to not only present their data, but also some analysis and some kind of conclusion. If we eliminate that second half, it probably won't do much to solve the problem. More importantly, your priors help you elect plausible explanations for data that let you delve deeper and identify the root cause. If you pretend not to have priors, you won't have any idea how to understand the data. For example, you run an experiment that shows that several communities using pesticide X average a 1% higher rate of lung cancer than comminities using pesticide Y. There's a lot of variance though, and some communities X have much lower cancer rates. You could publish the data with no further analysis and leave the broader scientific community to do the work of trying to draw conclusions. Or you could remember that you expected the opposite effect, because pesticide Y has a more direct route to enter the lungs, and then you go back and check the smoking rates in those communities and find a nearly perfect correlation. Accounting for the smoking rates, pesticide Y appears to increase lung cancer rates by 1% relative to pesticide X. Yes, you could have just published the raw data and other scientists could have figured that out. But realistically, they won't be nearly as involved as you and your peer reviewers are in the work. Even assuming you identify the communities in your published dataset, few scientists doing a meta-review would go deep enough to cros correlate your data against smoking rates if you didn't already do it for them. And if they actually do it, it's only because their priors are telling them that there's an important effect that could alter the conclusion... In other words, it's your priors that give you a nagging feeling that no matter how big the sample size D is comparing cancer rates among X and Y, you just won't feel P(C|X,~Y,D)-P(C|~X,Y,D) collapse to a narrow distribution in your mind, because you'll know that the outcome hinges critically on a third factor that you haven't seen yet.
cultus 7y ago
You can't "not assume" a prior. Frequentist statistics does indeed use (sometimes improper) implicit priors, and thus implicitly assumes subjective information. Bayesian inference makes it explicit. Anyone who thinks they can evaluate evidence objectively in the first place is fooling themselves. There is always subjectivity. If you really want to assume "minimum information," use the Jefferys prior for your domain. This minimizes the square root of the determinant of the Fisher information matrix. The Jefferys has the key property of being invariant upon changes of parameters. This property is not generally true of implicit frequentist priors (unless they correspond with the Jefferys prior), but yet is essential for uninformativeness. If your model depends on the units you choose for your parameters, you can hardly call it objective! https://en.wikipedia.org/wiki/Jeffreys_prior https://en.wikipedia.org/wiki/Jeffreys_prior
- dwheeler 7y ago
  Explicit nonsense is still nonsense. I don't believe that either Bayesian or frequentist approaches are the "one true way"; they are merely tools that help us understand the world, and both approaches have limitations. I suggest looking at the essay "Beyond Bayesians and Frequentists" by Jacob Steinhardt http://cs.stanford.edu/~jsteinhardt/stats-essay.pdf http://cs.stanford.edu/~jsteinhardt/stats-essay.pdf , who says, "The essential difference between Bayesian and frequentist decision theory is that Bayes makes the additional assumption of a prior... and optimizes for average-case performance rather than worst-case performance. It follows, then, that Bayes is the superior method whenever we can obtain a good prior and when good average-case performance is sufficient. However, if we have no way of obtaining a good prior, or when we need guaranteed performance, frequentist methods are the way to go." The same author also has an essay that tries to explain why Bayesians shouldn't be so confident in their approach: "A Fervent Defense of Frequentist Statistics", 18th Feb 2014, https://www.lesswrong.com/posts/KdwP5i6N4E4q6BGkr/a-fervent-defense-of-frequentist-statistics https://www.lesswrong.com/posts/KdwP5i6N4E4q6BGkr/a-fervent-... Note that the author isn't against Bayesian approaches, but against the dogmatic assumption that frequentist approaches are always worse.
  - pdonis 7y ago
    > the essay "Beyond Bayesians and Frequentists" by Jacob Steinhardt This essay is about decision theory, and the things he is calling "Bayesian" and "frequentist" are more than just statistical methods, which is what the replication crisis is about. Decision theory, particularly when other agents are present, cannot be handled by any method that only considers statistics; game theory is involved. The Steinhardt article is basically claiming that "Bayesians" can't use game theory while "frequentists" can, which is nonsense.
jdietrich 7y ago
IMO the most significant problem isn't any particular statistical approach, but the incredibly archaic nature of scientific publication. I read a paper and think that there's something wonky with the statistics. Can I go to the journal's website and download the full dataset? No. Can I download the source code used to compute those statistics? No. I can e-mail one of the authors and ask nicely for that information, but what are the odds that they'll respond if they know that their work is weak or outright fraudulent? Science is supposed to be transparent, it's supposed to be about sharing information, but there's a big opaque wall concealing anything that won't fit into the archaic size limitations of a paper journal. There's no One Weird Trick to fix the replication crisis, there's no obvious Bayesian replacement for p-values or confidence intervals, but there's an obvious shortcoming in scientific publishing that is a huge impediment to secondary research. A high schools student can't get full credit on a math exam without showing their working - why should scientists be held to a lower standard?
- lisper 7y ago
  It's even worse than that. The whole peer-review process we have today is stuck in the 19th century. The whole idea that publication and the credibility that is associated with it should be a binary quality -- a paper is either "published in a peer-reviewd publication", and therefore credible, or it isn't and so it isn't -- is absurd on its face. If there are any shades of grey, they are associated with the "prestige" of the publication, which again, if you think about it, is absurd, and flies in the face of the scientific ideal: an idea is either supported by the data or it isn't, and what two randomly selected anonymous referees on the review board of a journal, people who likely have ulterior motives in terms of their own career advancement, have to say about it should be the least of anyone's concern. The right model IMHO is open publication a la arxiv with an associated open review process so that anyone who wants to can weigh in. The wheat can be separated from the chaff by doing a "reviewer-rank" analogous to page rank so that the opinions of the people who know what they're doing count more than the crackpots. But this idea that there's an elite priesthood of anointed Scientists, a.k.a. corporate researchers and tenure-track university faculty, who pass final judgement on all scientific truth really has to end. (And I say this as someone who made their living working successfully within the current system for 15 years. I quit in no small measure because I saw firsthand just how corrupt and dysfunctional it could become, and I was working in a field that is generally considered "hard" science!)
mcguire 7y ago
"It's not just "being afraid"; the problem is that random guessing (of priors) is not a reasonable replacement for science." As the article goes on to argue, ignoring (or hiding) the existence of priors is not science, either. (Two incidences of SIDS in one family are probably not independent, even if they aren't murder.)
725686 7y ago
I don't know the first thing about statistics, other than noticing that each time there is a "fight" between Bayesians and Frequentists (on internet forums) there are always "experts" on both sides dismissing the other without ever showing a systematic, scientific argument, and without ever reaching a conclusion.
- BeetleB 7y ago
  The thing is, hard core frequentists don't really exist. You have Bayesians, and you have people who happily use Bayes theorem as appropriate. Take this article as an example. Reading it, I got the sense that the author at some point in life discovered Bayesian statistics, and is now on a crusade to twist everything into a pro and anti-Bayesian stance. His example of the base rate fallacy (breat cancer diagnosis) is probably in every "frequentist" text book out there. Frequentists are well aware of it, and have no aversion to using Bayes Theorem. You will not find a frequentist objecting to taking into account the base rate when applying statistics. The difference, as the GP mentioned, is that the base rate is fairly well known and not subject to much debate. Whereas this: >Maybe we’re not so dogmatic as to rule out “The Thinker” hypothesis altogether, but a prior probability of 1 in 1,000, somewhere between the chance of being dealt a full house and four-of-a-kind in a poker hand, could be around the right order of magnitude. Is a number he pulled out of his rear end, and his subsequent calculation is not meaningful to people who don't agree with the prior. Sure, anyone can manufacture a prior if they wanted. And part of me is merely wondering: If he wanted a prior of 1 in 1000, why not simply require a p value of 0.001 instead of the 0.03 the paper used? The problem with the paper is that the sample size is small (n=57), and small samples are a lot more likely to give extreme results. I'll be OK with a p=0.03 if n=10000, but not if n < 100.
  - jhbadger 7y ago
    I don't know your age, but from my experience, at least in the 1980s and 1990s, there really were real warring camps of Bayesians and Frequentists. People working on the same scientific topic who were Frequentists wouldn't even cite the papers of their Bayesian colleagues. Frequentist textbooks like A.W.F Edward's "Likelihood" would spend pages disparaging Bayesian methods. But I agree that things are much calmer these days with most people being pragmatists that don't care about being "pure" but use a mixture of methods from both camps.
thom 7y ago
Is it not valid to do a multiverse analysis using all conceivable priors in this case? There’s no real need to make an arbitrary decision, unless you truly know nothing about what you’re looking at in which case, shrug.
patientplatypus 7y ago
Yeah, I don't agree. How does one go about assessing the validity of any given prior a priori? All priors are based on some sort of logical reasoning based on past experience, but much of the time when investigating something novel there's no guarantee that past experience is a good predictor. Often we don't even know to what degree we can or cannot rely on past experience. I just don't buy that there's something magically more "mathy" about just conjuring assumptions you want to test.
- jbay808 7y ago
  If you follow the derivation of frequentist formulas, you'll find they always start from some kind of assumption, such as a uniform prior. You can't escape having a prior. The only way to avoid having a prior is by pulling a formula from a textbook and applying it without looking at its derivation.
6gvONxR4sf7o 7y ago
This acts like there isn't just as much subjectivity in frequentist analysis. Ever run a power analysis? The same p value with an underpowered study can be more misleading, so you still need some subjective inputs to your design. This is a subjective frequentist fix for a real issue. 80% power with 5% significance with a presumed effect size of x contains three totally subjective numbers (and one reeeeeally subjective one)? They are crucial in the same way, and subjective in the same way. You need a lot more than a p-value to do frequentist science, and it's not that objective. What moved me over was the practical learning that your results often aren't that sensitive to your prior and that dependence is usually easy enough to characterize.
nonbel 7y ago
For almost all the typical use cases a Frequentist 95% confidence interval will be nearly the same as a Bayesian 95% credible interval determined using a uniform prior. In every case I have seen where this is not true the Frequentist interval gives nonsensical results. So your complaint is a total non-issue. Most people are interpreting their confidence intervals as credible intervals anyway... Essentially, for nearly a hundred years researchers have already been using Bayesian stats with flat priors.
harimau777 7y ago
Couldn't you plot a graph of prior probability versus Bayesian significance? So that someone could see how significant it is if you assume that the prior probability is 1 in 100 vs 1 in 1000 vs 1 in a million, etc.

cwyers 7y ago

> Bayes' rule is a great tool in many circumstances! But it has a great weakness: it requires a prior. That doesn't make it useless; few tools are useful in all circumstances. But requiring "everyone to use Bayes' rule, even though we have no reasonable way to find a good estimate of the priors," is unlikely to ever happen (and rightly so). The article rightly points out a serious problem with the typical application of statistics, but there needs to be a better justification for priors than is suggested in this article. If you don't have good information on what your prior should be, you can just use a really weak prior. And you probably know more about your data than you think you do -- if you are doing counts, that leads you to a different prior than if you are doing things that can be in increments smaller than a single integer.

kansface 7y ago

> the problem is that random guessing (of priors) is not a reasonable replacement for science. I disagree! > After all, Bayes' rule will give nonsense answers if you give it a nonsense prior! You are absolutely correct here, but draw the wrong conclusion. Bayes rule isn't just some convenience that you ought to ignore without a solid prior. Fundamentally, what can any study say about a subject whose prior likely ranges over a few orders of magnitude among the experts? That subject represents a gaping hole in our knowledge. If a single study were to narrow the band by a whole decade, it would still leave a decade sized hole... All of this should be explicit and out in the open. > I could imagine systemic worldwide ways to deal with this. For example, perhaps the scientific community could allow people to propose initial priors, and then allow multiple different papers to improve the estimation of the probability over time. Yeah, this is the way to go. Researchers should preregister their studies with their priors. Future studies will revise them. After all, its easy to argue about numbers and those who disagree can easily supply their own.

DataWorker 7y ago
This ignores the labor market economics involved in research. If there was lots of pre-registration and always multiple replication attempts then there wouldn’t be much of a replication crisis in the first place. It’s not frequentist stats that’s to blame, it’s the labor market for researchers.
- kansface 7y ago
  I appreciate your point, but surely both are problems? One is bad math while the other is a sort of a prisoners dilemma where everyone has already defected. I don't think changing the incentives would necessarily lead to better math or even that we have to change the incentives before changing the math.
- tomp 7y ago
  No, the problem is much wider than just researchers. If for example the public, or at least the media, didn’t buy the bullshit they were selling (with high p values, non-public data and other shoddy statistical hacks), there wouldn’t be any replication crisis either. The general problem is that just stamping the word “science” on a black box that smells of poo gives it much more credibility than it should, and it often also prevents people from thinking critically.

pdonis 7y ago

> random guessing (of priors) is not a reasonable replacement for science. Priors in Bayesian statistics are not randomly guessed. The prior is supposed to reflect your state of knowledge prior to looking at the data from the experiment/study/test/whatever (hence the name "prior"). For example, in the mammogram example, it is assumed that the doctor knows the base rate of breast cancer in the population (which is a very reasonable assumption since there are mountains of data on such rates) and the false positive and false negative rates of the test (which come from data on previous tests). Those are the prior. The doctor doesn't just make up those numbers; they come from prior knowledge. > if you can't find a justifiable prior in a scientific work, using Bayes' rule just replaces one statistical fallacy for another one In other words, if you have no prior knowledge about something, you can't expect to do statistics on some small set of data and get reliable answers. Yes, that's true. And Bayesianism tells you this is true, by telling you that you can't find a justifiable prior. (Technically, you can always find a maximum entropy prior, but in most cases that's not really any better than having no prior at all since it is basically telling you you need a lot more data before you can conclude anything.) Whereas statistical methods as they are currently done in many scientific fields simply ignore this issue completely and pick arbitrary thresholds for statistical significance. Which is the article's point. In other words, Bayesianism, properly viewed, is not a magic bullet for extracting statistical answers from nowhere. It is a tool for exercising discipline, so that you know when you have to say "sorry, not enough data for a meaningful answer".

mcguire 7y ago
"It is a tool for exercising discipline, so that you know when you have to say "sorry, not enough data for a meaningful answer"." As opposed to the frequentist approach of saying, "We have data D and hypothesis H. If the probability of D when not-H is true is below a threshold, H is true," which hides an awful lot of assumptions. The advantage of Baysian statistics is that you have to explicitly state your assumptions in terms of a prior.
- BeetleB 7y ago
  >As opposed to the frequentist approach of saying, "We have data D and hypothesis H. If the probability of D when not-H is true is below a threshold, H is true," which hides an awful lot of assumptions. My textbook (what would be called a frequentist book) is very explicit: You either reject the null hypothesis, or you fail to reject the null hypothesis. A significance level test never concludes that the null hypothesis is true. It's nuanced, but is not quite the same as your statement.

edflsafoiewq 7y ago

Why can't the result itself be the map prior->posterior (instead of the result of applying this map to a fixed prior)?

comex 7y ago
Well, that’s essentially what frequentist statistics gives you: the probability of the results given the hypothesis, which Bayes’ rule lets you combine with a prior to get the probability of the hypothesis given the results. But people don’t usually actually perform that calculation...
- 6gvONxR4sf7o 7y ago
  >Well, that’s essentially what frequentist statistics gives you: the probability of the results given the hypothesis No. The whole point of NHST is that it gives the probability of the results given the null hypothesis. It's whole point is that it doesn't touch probability of the results given other hypotheses (except when designing the experiment, where you need other subjective inputs).
jbay808 7y ago
This is known as the "evidence". Usually we expect scientific papers to not only present their data, but also some analysis and some kind of conclusion. If we eliminate that second half, it probably won't do much to solve the problem. More importantly, your priors help you elect plausible explanations for data that let you delve deeper and identify the root cause. If you pretend not to have priors, you won't have any idea how to understand the data. For example, you run an experiment that shows that several communities using pesticide X average a 1% higher rate of lung cancer than comminities using pesticide Y. There's a lot of variance though, and some communities X have much lower cancer rates. You could publish the data with no further analysis and leave the broader scientific community to do the work of trying to draw conclusions. Or you could remember that you expected the opposite effect, because pesticide Y has a more direct route to enter the lungs, and then you go back and check the smoking rates in those communities and find a nearly perfect correlation. Accounting for the smoking rates, pesticide Y appears to increase lung cancer rates by 1% relative to pesticide X. Yes, you could have just published the raw data and other scientists could have figured that out. But realistically, they won't be nearly as involved as you and your peer reviewers are in the work. Even assuming you identify the communities in your published dataset, few scientists doing a meta-review would go deep enough to cros correlate your data against smoking rates if you didn't already do it for them. And if they actually do it, it's only because their priors are telling them that there's an important effect that could alter the conclusion... In other words, it's your priors that give you a nagging feeling that no matter how big the sample size D is comparing cancer rates among X and Y, you just won't feel P(C|X,~Y,D)-P(C|~X,Y,D) collapse to a narrow distribution in your mind, because you'll know that the outcome hinges critically on a third factor that you haven't seen yet.

cultus 7y ago

You can't "not assume" a prior. Frequentist statistics does indeed use (sometimes improper) implicit priors, and thus implicitly assumes subjective information. Bayesian inference makes it explicit. Anyone who thinks they can evaluate evidence objectively in the first place is fooling themselves. There is always subjectivity. If you really want to assume "minimum information," use the Jefferys prior for your domain. This minimizes the square root of the determinant of the Fisher information matrix. The Jefferys has the key property of being invariant upon changes of parameters. This property is not generally true of implicit frequentist priors (unless they correspond with the Jefferys prior), but yet is essential for uninformativeness. If your model depends on the units you choose for your parameters, you can hardly call it objective! https://en.wikipedia.org/wiki/Jeffreys_prior https://en.wikipedia.org/wiki/Jeffreys_prior

dwheeler 7y ago
Explicit nonsense is still nonsense. I don't believe that either Bayesian or frequentist approaches are the "one true way"; they are merely tools that help us understand the world, and both approaches have limitations. I suggest looking at the essay "Beyond Bayesians and Frequentists" by Jacob Steinhardt http://cs.stanford.edu/~jsteinhardt/stats-essay.pdf http://cs.stanford.edu/~jsteinhardt/stats-essay.pdf , who says, "The essential difference between Bayesian and frequentist decision theory is that Bayes makes the additional assumption of a prior... and optimizes for average-case performance rather than worst-case performance. It follows, then, that Bayes is the superior method whenever we can obtain a good prior and when good average-case performance is sufficient. However, if we have no way of obtaining a good prior, or when we need guaranteed performance, frequentist methods are the way to go." The same author also has an essay that tries to explain why Bayesians shouldn't be so confident in their approach: "A Fervent Defense of Frequentist Statistics", 18th Feb 2014, https://www.lesswrong.com/posts/KdwP5i6N4E4q6BGkr/a-fervent-defense-of-frequentist-statistics https://www.lesswrong.com/posts/KdwP5i6N4E4q6BGkr/a-fervent-... Note that the author isn't against Bayesian approaches, but against the dogmatic assumption that frequentist approaches are always worse.
- pdonis 7y ago
  > the essay "Beyond Bayesians and Frequentists" by Jacob Steinhardt This essay is about decision theory, and the things he is calling "Bayesian" and "frequentist" are more than just statistical methods, which is what the replication crisis is about. Decision theory, particularly when other agents are present, cannot be handled by any method that only considers statistics; game theory is involved. The Steinhardt article is basically claiming that "Bayesians" can't use game theory while "frequentists" can, which is nonsense.

jdietrich 7y ago

IMO the most significant problem isn't any particular statistical approach, but the incredibly archaic nature of scientific publication. I read a paper and think that there's something wonky with the statistics. Can I go to the journal's website and download the full dataset? No. Can I download the source code used to compute those statistics? No. I can e-mail one of the authors and ask nicely for that information, but what are the odds that they'll respond if they know that their work is weak or outright fraudulent? Science is supposed to be transparent, it's supposed to be about sharing information, but there's a big opaque wall concealing anything that won't fit into the archaic size limitations of a paper journal. There's no One Weird Trick to fix the replication crisis, there's no obvious Bayesian replacement for p-values or confidence intervals, but there's an obvious shortcoming in scientific publishing that is a huge impediment to secondary research. A high schools student can't get full credit on a math exam without showing their working - why should scientists be held to a lower standard?

lisper 7y ago
It's even worse than that. The whole peer-review process we have today is stuck in the 19th century. The whole idea that publication and the credibility that is associated with it should be a binary quality -- a paper is either "published in a peer-reviewd publication", and therefore credible, or it isn't and so it isn't -- is absurd on its face. If there are any shades of grey, they are associated with the "prestige" of the publication, which again, if you think about it, is absurd, and flies in the face of the scientific ideal: an idea is either supported by the data or it isn't, and what two randomly selected anonymous referees on the review board of a journal, people who likely have ulterior motives in terms of their own career advancement, have to say about it should be the least of anyone's concern. The right model IMHO is open publication a la arxiv with an associated open review process so that anyone who wants to can weigh in. The wheat can be separated from the chaff by doing a "reviewer-rank" analogous to page rank so that the opinions of the people who know what they're doing count more than the crackpots. But this idea that there's an elite priesthood of anointed Scientists, a.k.a. corporate researchers and tenure-track university faculty, who pass final judgement on all scientific truth really has to end. (And I say this as someone who made their living working successfully within the current system for 15 years. I quit in no small measure because I saw firsthand just how corrupt and dysfunctional it could become, and I was working in a field that is generally considered "hard" science!)

mcguire 7y ago

"It's not just "being afraid"; the problem is that random guessing (of priors) is not a reasonable replacement for science." As the article goes on to argue, ignoring (or hiding) the existence of priors is not science, either. (Two incidences of SIDS in one family are probably not independent, even if they aren't murder.)

725686 7y ago

I don't know the first thing about statistics, other than noticing that each time there is a "fight" between Bayesians and Frequentists (on internet forums) there are always "experts" on both sides dismissing the other without ever showing a systematic, scientific argument, and without ever reaching a conclusion.

BeetleB 7y ago
The thing is, hard core frequentists don't really exist. You have Bayesians, and you have people who happily use Bayes theorem as appropriate. Take this article as an example. Reading it, I got the sense that the author at some point in life discovered Bayesian statistics, and is now on a crusade to twist everything into a pro and anti-Bayesian stance. His example of the base rate fallacy (breat cancer diagnosis) is probably in every "frequentist" text book out there. Frequentists are well aware of it, and have no aversion to using Bayes Theorem. You will not find a frequentist objecting to taking into account the base rate when applying statistics. The difference, as the GP mentioned, is that the base rate is fairly well known and not subject to much debate. Whereas this: >Maybe we’re not so dogmatic as to rule out “The Thinker” hypothesis altogether, but a prior probability of 1 in 1,000, somewhere between the chance of being dealt a full house and four-of-a-kind in a poker hand, could be around the right order of magnitude. Is a number he pulled out of his rear end, and his subsequent calculation is not meaningful to people who don't agree with the prior. Sure, anyone can manufacture a prior if they wanted. And part of me is merely wondering: If he wanted a prior of 1 in 1000, why not simply require a p value of 0.001 instead of the 0.03 the paper used? The problem with the paper is that the sample size is small (n=57), and small samples are a lot more likely to give extreme results. I'll be OK with a p=0.03 if n=10000, but not if n < 100.
- jhbadger 7y ago
  I don't know your age, but from my experience, at least in the 1980s and 1990s, there really were real warring camps of Bayesians and Frequentists. People working on the same scientific topic who were Frequentists wouldn't even cite the papers of their Bayesian colleagues. Frequentist textbooks like A.W.F Edward's "Likelihood" would spend pages disparaging Bayesian methods. But I agree that things are much calmer these days with most people being pragmatists that don't care about being "pure" but use a mixture of methods from both camps.

thom 7y ago

Is it not valid to do a multiverse analysis using all conceivable priors in this case? There’s no real need to make an arbitrary decision, unless you truly know nothing about what you’re looking at in which case, shrug.

patientplatypus 7y ago

Yeah, I don't agree. How does one go about assessing the validity of any given prior a priori? All priors are based on some sort of logical reasoning based on past experience, but much of the time when investigating something novel there's no guarantee that past experience is a good predictor. Often we don't even know to what degree we can or cannot rely on past experience. I just don't buy that there's something magically more "mathy" about just conjuring assumptions you want to test.

jbay808 7y ago
If you follow the derivation of frequentist formulas, you'll find they always start from some kind of assumption, such as a uniform prior. You can't escape having a prior. The only way to avoid having a prior is by pulling a formula from a textbook and applying it without looking at its derivation.

6gvONxR4sf7o 7y ago

This acts like there isn't just as much subjectivity in frequentist analysis. Ever run a power analysis? The same p value with an underpowered study can be more misleading, so you still need some subjective inputs to your design. This is a subjective frequentist fix for a real issue. 80% power with 5% significance with a presumed effect size of x contains three totally subjective numbers (and one reeeeeally subjective one)? They are crucial in the same way, and subjective in the same way. You need a lot more than a p-value to do frequentist science, and it's not that objective. What moved me over was the practical learning that your results often aren't that sensitive to your prior and that dependence is usually easy enough to characterize.

nonbel 7y ago

For almost all the typical use cases a Frequentist 95% confidence interval will be nearly the same as a Bayesian 95% credible interval determined using a uniform prior. In every case I have seen where this is not true the Frequentist interval gives nonsensical results. So your complaint is a total non-issue. Most people are interpreting their confidence intervals as credible intervals anyway... Essentially, for nearly a hundred years researchers have already been using Bayesian stats with flat priors.

harimau777 7y ago

Couldn't you plot a graph of prior probability versus Bayesian significance? So that someone could see how significant it is if you assume that the prior probability is 1 in 100 vs 1 in 1000 vs 1 in a million, etc.