3 ms·

I see prior probabilities talked about often in relation to scientific studies lately. But is it really appropriate to assign a prior probability to something t

by darawk 11y ago

I see prior probabilities talked about often in relation to scientific studies lately. But is it really appropriate to assign a prior probability to something that is already true or false? The theory's truth or falsehood is a fact in the world that exists independently of any of our models. Does it really have a 'prior probability' of being true or false? And if so is there really any sense in which we could realistically assign it one? Should we look at the history of the field or the researcher?

stdbrouw 11y ago
Think of a prior as what's plausible or not plausible given the current state of our knowledge, rather than as an objective state of the world. Mind you, that doesn't mean priors are necessarily subjective. In medicine, we often have information about the prevalence of diseases, which gives us a prior that we can take into account when looking at the result of a test: if someone tests positive on an incredibly rare disease, chances are the test is wrong because the prior probability of having the disease is so exceedingly low. In psychology, if you read research that claims to have found statistically significant evidence for extrasensory perception, you might not believe that straight away, because it contradicts everything we know about the physical universe and thus is a priori very improbable. In marketing, if you're doing an A/B-test where you change some body copy, in the very best case you might get a 100% increase in conversion, but certainly not a 100,000,000% increase. A prior can be as subtle as "no effect is a tiny bit more likely than an effect of 10^12" and this can be enough to get better estimates when you have very little data. Your interpretation is not unusual, though: in frequentist statistics, parameters are considered fixed values and as such not something you can assign a probability.
- darawk 11y ago
  Ya I suppose my point is that such assignments are inherently subjective, and derived from limited data (except in some limited cases, e.g. disease screening, though not disease testing in general). Consider the theory that there is an unidentified teapot floating around the moon. What prior would you assign that? 1x10^-20? What if I then told you that a teapot was on board the space shuttle during the first moon landing? Maybe then the exponent rises to -10. What if then I further told you that when it came back, that teapot was unaccounted for? Now all of the sudden what seemed wildly improbable actually seems remotely possible. Not because anything changed in the world, mind you, but because your knowledge of the world became more complete. Particularly in the case of a scientific theory, it seems almost completely impossible to adequately estimate the probability of its truth prior to testing. For instance, if I were to propose that conservation of energy is incorrect, most people would assign that an extraordinarily low prior. But would they really be justified in so doing? Is there really evidence that conservation of energy is a fundamental property of the universe, or does there just happen to be a lot of stuff that we've observed that comports with it? I suppose i'm essentially making Plato's cave's argument here.
nonbel 11y ago
You should put priors on the parameter values of a model, not its truth/falsehood. I have no idea where the latter idea comes from. There may be something to it but it sounds nonsensical to me. For example, a model of carcinogenesis could use a prior for mutations rates centered on around 10^-6 somatic mutations/base-pair/cell division rather than 10^-100 or 10^100. The first is too rare and second too common to be consistent with other lines of evidence. The priors don't have to be exact, just in the right ballpark to constrain the model output to realistic scenarios. Of course you could argue about what's a realistic mutation rate, just like you could argue about the choice of model itself (maybe some people believe somatic mutations are a result rather than cause of cancer, so including those values in the model is wrong to begin with).

by darawk 11y ago

stdbrouw 11y ago

Think of a prior as what's plausible or not plausible given the current state of our knowledge, rather than as an objective state of the world. Mind you, that doesn't mean priors are necessarily subjective. In medicine, we often have information about the prevalence of diseases, which gives us a prior that we can take into account when looking at the result of a test: if someone tests positive on an incredibly rare disease, chances are the test is wrong because the prior probability of having the disease is so exceedingly low. In psychology, if you read research that claims to have found statistically significant evidence for extrasensory perception, you might not believe that straight away, because it contradicts everything we know about the physical universe and thus is a priori very improbable. In marketing, if you're doing an A/B-test where you change some body copy, in the very best case you might get a 100% increase in conversion, but certainly not a 100,000,000% increase. A prior can be as subtle as "no effect is a tiny bit more likely than an effect of 10^12" and this can be enough to get better estimates when you have very little data. Your interpretation is not unusual, though: in frequentist statistics, parameters are considered fixed values and as such not something you can assign a probability.

darawk 11y ago
Ya I suppose my point is that such assignments are inherently subjective, and derived from limited data (except in some limited cases, e.g. disease screening, though not disease testing in general). Consider the theory that there is an unidentified teapot floating around the moon. What prior would you assign that? 1x10^-20? What if I then told you that a teapot was on board the space shuttle during the first moon landing? Maybe then the exponent rises to -10. What if then I further told you that when it came back, that teapot was unaccounted for? Now all of the sudden what seemed wildly improbable actually seems remotely possible. Not because anything changed in the world, mind you, but because your knowledge of the world became more complete. Particularly in the case of a scientific theory, it seems almost completely impossible to adequately estimate the probability of its truth prior to testing. For instance, if I were to propose that conservation of energy is incorrect, most people would assign that an extraordinarily low prior. But would they really be justified in so doing? Is there really evidence that conservation of energy is a fundamental property of the universe, or does there just happen to be a lot of stuff that we've observed that comports with it? I suppose i'm essentially making Plato's cave's argument here.

nonbel 11y ago

You should put priors on the parameter values of a model, not its truth/falsehood. I have no idea where the latter idea comes from. There may be something to it but it sounds nonsensical to me. For example, a model of carcinogenesis could use a prior for mutations rates centered on around 10^-6 somatic mutations/base-pair/cell division rather than 10^-100 or 10^100. The first is too rare and second too common to be consistent with other lines of evidence. The priors don't have to be exact, just in the right ballpark to constrain the model output to realistic scenarios. Of course you could argue about what's a realistic mutation rate, just like you could argue about the choice of model itself (maybe some people believe somatic mutations are a result rather than cause of cancer, so including those values in the model is wrong to begin with).