4 ms·

So how does one go about choosing reasonable bayesian priors for an experiment?

afthonos 7y ago
It becomes part of your experimental design. Just like people can quibble with your setup, with your questions, with your procedures, they can quibble with your priors. The difference is that it's out there, explicit. Note that a big weakness of Bayes rule is that you can look at any data and specify a prior that will make it look good. To continue with the mammogram example, suppose the doctor says "We really don't know if you are likely to have cancer or not. So we're just going to give 50-50 odds, and see what the test comes back with." That's a very different prior from the known base rate. The results would be, where C means "Cancer" and "R" means "Positive Result": P(C|R) = P(R|C) * P_prior(C) / P(R) = 1.0 * 0.5 / (1.0 * 0.5 + 0.05 * 0.5) = 0.5 / 0.55 = 0.9 A much higher probability. As you can imagine, you can do that in a paper as well: you know the data you have, and you come up with a "plausible" prior to make the data seem important. In my opinion, in any switch to using Bayesian analysis in scientific work, pre-registering priors will be essential.
- mike_ivanov 7y ago
  Taking 50/50 as a prior indicates that you don't know the true prevalence of C. Note that here 0.9 is not a probability, but rather a degree of belief. In this concrete example it tells you how much you should be worried upon observing R. If the prevalence of C is unknown (even if C is actually rare!), then given a positive R one should be worried a lot, which is exactly what 0.9 says. Also note the language change and its implications for policy making, etc: Instead of saying "there is [not enough] evidence that .." we say "given the observations and these assumptions we should [not] believe/expect that .."
  - afthonos 7y ago
    All of this is true. I was writing to mainly illustrate that Bayesian analysis is still open to abuse—though as the sibling commenter says, it's hopefully clearer when it's being abused.
- SolarNet 7y ago
  It's also nice in that it's a more accessible statistical number than significance testing. It's easier to look at a prior and know the science is suspect (since when is that a 5 in 100,000 chance) than it is a P value test (so this data would fall outside of 8 standard deviations 10% of the time...? I'm not sure how that maps).
- pdonis 7y ago
  > a big weakness of Bayes rule is that you can look at any data and specify a prior that will make it look good This isn't a weakness in Bayes' rule, it's a weakness in your experimental protocol. You're supposed to pick the prior before doing the experiment and seeing the data. > In my opinion, in any switch to using Bayesian analysis in scientific work, pre-registering priors will be essential. Pre-registering statistical criteria and assumptions should already be essential, whether you're a Bayesian or not. The fact that it isn't is a key factor behind the replication crisis.
  - afthonos 7y ago
    > This isn't a weakness in Bayes' rule, it's a weakness in your experimental protocol. You're supposed to pick the prior before doing the experiment and seeing the data. Sure, if the goal is to get to something true. If the goal is to publish or to maintain your position, though, you’ll work differently. You don’t even need to have seen the data. If I set my priors for the earth being flat extreme enough, it’ll take a long time for even good faith updating to converge to reality. As I said in another reply, I was simply pointing out that Bayesian analysis can also be abused, and that proper protocols still need to be followed. A point on which I believe we agree. :-)
    - pdonis 7y ago
      > Bayesian analysis can also be abused, and that proper protocols still need to be followed. A point on which I believe we agree. :-) Yes, indeed. :-)

It becomes part of your experimental design. Just like people can quibble with your setup, with your questions, with your procedures, they can quibble with your priors. The difference is that it's out there, explicit. Note that a big weakness of Bayes rule is that you can look at any data and specify a prior that will make it look good. To continue with the mammogram example, suppose the doctor says "We really don't know if you are likely to have cancer or not. So we're just going to give 50-50 odds, and see what the test comes back with." That's a very different prior from the known base rate. The results would be, where C means "Cancer" and "R" means "Positive Result": P(C|R) = P(R|C) * P_prior(C) / P(R) = 1.0 * 0.5 / (1.0 * 0.5 + 0.05 * 0.5) = 0.5 / 0.55 = 0.9 A much higher probability. As you can imagine, you can do that in a paper as well: you know the data you have, and you come up with a "plausible" prior to make the data seem important. In my opinion, in any switch to using Bayesian analysis in scientific work, pre-registering priors will be essential.

mike_ivanov 7y ago
Taking 50/50 as a prior indicates that you don't know the true prevalence of C. Note that here 0.9 is not a probability, but rather a degree of belief. In this concrete example it tells you how much you should be worried upon observing R. If the prevalence of C is unknown (even if C is actually rare!), then given a positive R one should be worried a lot, which is exactly what 0.9 says. Also note the language change and its implications for policy making, etc: Instead of saying "there is [not enough] evidence that .." we say "given the observations and these assumptions we should [not] believe/expect that .."
- afthonos 7y ago
  All of this is true. I was writing to mainly illustrate that Bayesian analysis is still open to abuse—though as the sibling commenter says, it's hopefully clearer when it's being abused.
SolarNet 7y ago
It's also nice in that it's a more accessible statistical number than significance testing. It's easier to look at a prior and know the science is suspect (since when is that a 5 in 100,000 chance) than it is a P value test (so this data would fall outside of 8 standard deviations 10% of the time...? I'm not sure how that maps).
pdonis 7y ago
> a big weakness of Bayes rule is that you can look at any data and specify a prior that will make it look good This isn't a weakness in Bayes' rule, it's a weakness in your experimental protocol. You're supposed to pick the prior before doing the experiment and seeing the data. > In my opinion, in any switch to using Bayesian analysis in scientific work, pre-registering priors will be essential. Pre-registering statistical criteria and assumptions should already be essential, whether you're a Bayesian or not. The fact that it isn't is a key factor behind the replication crisis.
- afthonos 7y ago
  > This isn't a weakness in Bayes' rule, it's a weakness in your experimental protocol. You're supposed to pick the prior before doing the experiment and seeing the data. Sure, if the goal is to get to something true. If the goal is to publish or to maintain your position, though, you’ll work differently. You don’t even need to have seen the data. If I set my priors for the earth being flat extreme enough, it’ll take a long time for even good faith updating to converge to reality. As I said in another reply, I was simply pointing out that Bayesian analysis can also be abused, and that proper protocols still need to be followed. A point on which I believe we agree. :-)
  - pdonis 7y ago
    > Bayesian analysis can also be abused, and that proper protocols still need to be followed. A point on which I believe we agree. :-) Yes, indeed. :-)