5 ms·

> I agree, under various contrived scenarios, you can show your parameter estimate is unbiased when using a flat prior (yay!) but what happens if you're using t

> I agree, under various contrived scenarios, you can show your parameter estimate is unbiased when using a flat prior (yay!) but what happens if you're using the wrong parameterization of your model? I wouldn't consider estimating, say, the mean length of a population of fish contrived (unbiased estimate: x-bar). Nor would I consider estimating the probability of an event based on observations of the event happening or not happening contrived (unbiased estimate: p-hat = #successes/#trials). These kinds of simple estimation problems and the associated statistical tests account for probably 90% of statistical practice. Dismissing them as contrived is silly. More generally, MLE estimates are always (under regularity conditions) asymptotically unbiased even if not unbiased for a finite sample. This means that the amount of bias decreases to zero as the sample size increases, no matter what the parameterization is. Finally, there is very often a natural parameterization for any given problem. If you're interested in the arithmetic mean of a population, there's no reason to use a log-scale parameterization. Why worry about bias in other parameterizations when you can just use the natural parameterization, where the estimator is unbiased? Again, I don't think such scenarios are contrived: a very large proportion of statistical analyses deal with simple measurements in Euclidian (or very nearly Euclidian--we can typically ignore, for example, relativistic effects) spaces: real world dimensions, time, etc. If you're a Bayesian and very concerned about parameterization effects you can also use a Jeffrey's prior, which is parameterization-invariant. Notably, for the mean of a Normal distribution, the Jeffrey's prior is... the flat prior! > OK so let me clarify because I agree that my wording is wrong: while I agree with you that there are differences between Bayesian and frequentist statistics, they are philosophical; they answer different questions: Yes and no. The Bayesian and frequentist approaches answer different mathematical questions, but they are used by humans to answer the same human questions, such as "do these two populations have the same mean?" > see e.g. [1] for an example where flat priors do actually bias inferences. I don't consider that a good example of flat priors biasing the inference. The posterior with flat prior is diffuse because the data doesn't provide much information; in the lack of much prior or likelihood information, the posterior should be diffuse, so that's a perfectly reasonable result. If you can't stand a diffuse posterior then either collect more data or (carefully!) introduce a more informative prior. The more informative the prior you introduce, the more biased your inference will be; that's fine as long as your prior is chosen carefully. This is not to say that flat priors are never a problem. The setting I'm familiar with where diffuse priors are most dangerous is when doing model comparison--but that issue is specific to Bayesian methods, not frequentist. Indeed this is one of the primary reasons Lindley's paradox arises: the Bayesian model comparison (using marginal likelihoods or Bayes factors) gets tricked by the diffuse prior, while the frequentist model comparison (using null hypothesis testing) does not.

astrophysician 6y ago
> I wouldn't consider estimating, say, the mean length of a population of fish contrived (unbiased estimate: x-bar). Nor would I consider estimating the probability of an event based on observations of the event happening or not happening contrived (unbiased estimate: p-hat = #successes/#trials). Sure, maybe not contrived; my point is that flat priors may work in many "typical" textbook stats problems, but they are one of many choices, and that choice is important to be explicit about and not sweep under the rug. Because if your entire life is measuring sample means, fine, you're never going to need to think about this very much and life will be nice. But when one fine day you decide to do something more complex, these are the land mines that you shouldn't really ignore. > These kinds of simple estimation problems and the associated statistical tests account for probably 90% of statistical practice. Dismissing them as contrived is silly. Whether it's 90% is totally dependent on the types of problems you do. I don't mean to dismiss them, you're right for many problems MLE is just fine. I meant to illustrate that "unbiased" comes with many caveats, and that in many real scenarios flat priors are not ok. > More generally, MLE estimates are always (under regularity conditions) asymptotically unbiased even if not unbiased for a finite sample. This means that the amount of bias decreases to zero as the sample size increases, no matter what the parameterization is. Is this not true of the MAP for most priors? Gaussian/Laplace priors will have this property too, since priors become asymptotically less important the more data you have. If your prior is zero over some of the support, you're out of luck but this doesn't strike me as a good argument for MLE > MAP or for using flat priors everywhere. When we have infinite data, sure, priors are irrelevant, but we live in the real world where data is not infinite. > Finally, there is very often a natural parameterization for any given problem. If you're interested in the arithmetic mean of a population, there's no reason to use a log-scale parameterization. Sure, agree that parametrization isn't a problem a lot of the time, but it is something important to be mindful of, and this points towards, again, not forgetting that you are always using a prior and that you should think about whether or not that prior makes sense. > Why worry about bias in other parameterizations when you can just use the natural parameterization, where the estimator is unbiased? Again, I don't think such scenarios are contrived: a very large proportion of statistical analyses deal with simple measurements in Euclidian (or very nearly Euclidian--we can typically ignore, for example, relativistic effects) spaces: real world dimensions, time, etc. Yea, I mean sure: for easy problems, parametrization is obvious. That's kind of tautological. But sometimes it's not obvious, or sometimes for computational reasons you need to work with a log(theta) instead of theta, etc. If you're a frequentist and you're thinking life is great because you don't need to worry about priors, you're wrong and sooner or later you will get into trouble; be it a parametrization issue or something else, priors are not just something you can completely ignore. It's like saying "I always drive without looking in my rearview mirror" -- ok, great, you will be fine a lot of the time, but eventually one day you will change lanes on the highway at the exact wrong time, and you will really regret your habit of not looking in your mirror. > If you're a Bayesian and very concerned about parameterization effects you can also use a Jeffrey's prior, which is parameterization-invariant. Notably, for the mean of a Normal distribution, the Jeffrey's prior is... the flat prior! Yep, totally agree, I have no problem with Jeffrey's priors (when they make sense), and that's all well and good. Just to clarify: I am not saying "don't use flat priors" -- flat priors are extremely reasonable and a good idea in many cases, my point is flat priors are still priors, and you are still making a statement by using them: "lets assume all possible values of theta are equally likely a priori". Sometimes we don't really believe that but it's useful to see the implications of making this assumption. But sometimes priors are extremely important (e.g. we want a time-dependent measurement of a poisson rate, like conversions per dollar of ad spend, and conversions are relatively rare: priors are your friend here, e.g. a GP prior = Cox process or something else, even if this prior is an operational assumption) > Yes and no. The Bayesian and frequentist approaches answer different mathematical questions, but they are used by humans to answer the same human questions, such as "do these two populations have the same mean?" Yes, agreed. > Indeed this is one of the primary reasons Lindley's paradox arises: the Bayesian model comparison (using marginal likelihoods or Bayes factors) gets tricked by the diffuse prior, while the frequentist model comparison (using null hypothesis testing) does not. Ah lord, but this is a terrible justification for using null hypothesis rejection: we're almost always choosing a very simplistic distribution (e.g. Gaussian) to do this, and reducing the question to "we reject H0 because its very unlikely" is part of the reason why there's a replication crisis in e.g. social sciences, because they're taught this simplistic picture without any of the necessary nuance ('here are the assumptions we make, and under these assumptions + H0, it is a little bit unlikely that we would have observed x'). That's a recipe for disaster. Is it not much better to discuss the full posterior, "degrees of belief" and to be explicit about all of our uncomfortable prior assumptions? I prefer Bayesian model selection over null hypothesis rejection 100% of the time, especially because "Bayesian model selection" is the only logical way to do model selection, the only caveat is that it depends on reasonable prior assumptions and these are the hard part (but again, at least it is explicit!). Also, the Lindley's "paradox" example certainly seems contrived: we believe there's a 50% chance that p = 0.5 exactly?? I just don't understand that type of analysis. Come up with a prior, derive your posterior, decide the answer to your question yourself (what is the chance that p=0.5 exactly? well, it's exactly 0%. How much more likely is it that p=0.5036 vs p=0.5? That's a better question...). By contrived, I mean that it appears designed to exploit the fact that Bayesian stats will automatically prefer simpler models, especially one with 0 degrees of freedom that is relatively close to the right answer, but that's a Good Thing (TM). Both frequentist stats and Bayesian stats are easy to abuse: Bayesian stats gives a false sense of comfort because people don't worry enough about their choice of prior, but at least Bayesian stats is explicit about the prior!. I won't say that hypothesis testing is complete garbage, but it is quite dangerous and frankly dishonest to reduce things to a p value and pretend that's the end of the discussion.

astrophysician 6y ago

> I wouldn't consider estimating, say, the mean length of a population of fish contrived (unbiased estimate: x-bar). Nor would I consider estimating the probability of an event based on observations of the event happening or not happening contrived (unbiased estimate: p-hat = #successes/#trials). Sure, maybe not contrived; my point is that flat priors may work in many "typical" textbook stats problems, but they are one of many choices, and that choice is important to be explicit about and not sweep under the rug. Because if your entire life is measuring sample means, fine, you're never going to need to think about this very much and life will be nice. But when one fine day you decide to do something more complex, these are the land mines that you shouldn't really ignore. > These kinds of simple estimation problems and the associated statistical tests account for probably 90% of statistical practice. Dismissing them as contrived is silly. Whether it's 90% is totally dependent on the types of problems you do. I don't mean to dismiss them, you're right for many problems MLE is just fine. I meant to illustrate that "unbiased" comes with many caveats, and that in many real scenarios flat priors are not ok. > More generally, MLE estimates are always (under regularity conditions) asymptotically unbiased even if not unbiased for a finite sample. This means that the amount of bias decreases to zero as the sample size increases, no matter what the parameterization is. Is this not true of the MAP for most priors? Gaussian/Laplace priors will have this property too, since priors become asymptotically less important the more data you have. If your prior is zero over some of the support, you're out of luck but this doesn't strike me as a good argument for MLE > MAP or for using flat priors everywhere. When we have infinite data, sure, priors are irrelevant, but we live in the real world where data is not infinite. > Finally, there is very often a natural parameterization for any given problem. If you're interested in the arithmetic mean of a population, there's no reason to use a log-scale parameterization. Sure, agree that parametrization isn't a problem a lot of the time, but it is something important to be mindful of, and this points towards, again, not forgetting that you are always using a prior and that you should think about whether or not that prior makes sense. > Why worry about bias in other parameterizations when you can just use the natural parameterization, where the estimator is unbiased? Again, I don't think such scenarios are contrived: a very large proportion of statistical analyses deal with simple measurements in Euclidian (or very nearly Euclidian--we can typically ignore, for example, relativistic effects) spaces: real world dimensions, time, etc. Yea, I mean sure: for easy problems, parametrization is obvious. That's kind of tautological. But sometimes it's not obvious, or sometimes for computational reasons you need to work with a log(theta) instead of theta, etc. If you're a frequentist and you're thinking life is great because you don't need to worry about priors, you're wrong and sooner or later you will get into trouble; be it a parametrization issue or something else, priors are not just something you can completely ignore. It's like saying "I always drive without looking in my rearview mirror" -- ok, great, you will be fine a lot of the time, but eventually one day you will change lanes on the highway at the exact wrong time, and you will really regret your habit of not looking in your mirror. > If you're a Bayesian and very concerned about parameterization effects you can also use a Jeffrey's prior, which is parameterization-invariant. Notably, for the mean of a Normal distribution, the Jeffrey's prior is... the flat prior! Yep, totally agree, I have no problem with Jeffrey's priors (when they make sense), and that's all well and good. Just to clarify: I am not saying "don't use flat priors" -- flat priors are extremely reasonable and a good idea in many cases, my point is flat priors are still priors, and you are still making a statement by using them: "lets assume all possible values of theta are equally likely a priori". Sometimes we don't really believe that but it's useful to see the implications of making this assumption. But sometimes priors are extremely important (e.g. we want a time-dependent measurement of a poisson rate, like conversions per dollar of ad spend, and conversions are relatively rare: priors are your friend here, e.g. a GP prior = Cox process or something else, even if this prior is an operational assumption) > Yes and no. The Bayesian and frequentist approaches answer different mathematical questions, but they are used by humans to answer the same human questions, such as "do these two populations have the same mean?" Yes, agreed. > Indeed this is one of the primary reasons Lindley's paradox arises: the Bayesian model comparison (using marginal likelihoods or Bayes factors) gets tricked by the diffuse prior, while the frequentist model comparison (using null hypothesis testing) does not. Ah lord, but this is a terrible justification for using null hypothesis rejection: we're almost always choosing a very simplistic distribution (e.g. Gaussian) to do this, and reducing the question to "we reject H0 because its very unlikely" is part of the reason why there's a replication crisis in e.g. social sciences, because they're taught this simplistic picture without any of the necessary nuance ('here are the assumptions we make, and under these assumptions + H0, it is a little bit unlikely that we would have observed x'). That's a recipe for disaster. Is it not much better to discuss the full posterior, "degrees of belief" and to be explicit about all of our uncomfortable prior assumptions? I prefer Bayesian model selection over null hypothesis rejection 100% of the time, especially because "Bayesian model selection" is the only logical way to do model selection, the only caveat is that it depends on reasonable prior assumptions and these are the hard part (but again, at least it is explicit!). Also, the Lindley's "paradox" example certainly seems contrived: we believe there's a 50% chance that p = 0.5 exactly?? I just don't understand that type of analysis. Come up with a prior, derive your posterior, decide the answer to your question yourself (what is the chance that p=0.5 exactly? well, it's exactly 0%. How much more likely is it that p=0.5036 vs p=0.5? That's a better question...). By contrived, I mean that it appears designed to exploit the fact that Bayesian stats will automatically prefer simpler models, especially one with 0 degrees of freedom that is relatively close to the right answer, but that's a Good Thing (TM). Both frequentist stats and Bayesian stats are easy to abuse: Bayesian stats gives a false sense of comfort because people don't worry enough about their choice of prior, but at least Bayesian stats is explicit about the prior!. I won't say that hypothesis testing is complete garbage, but it is quite dangerous and frankly dishonest to reduce things to a p value and pretend that's the end of the discussion.