3 ms·

There seems to be a nice duality between Bayesian and Frequentist inference [1]: Assume that both the system state and the observation are drawn from some join

by yccs27 6y ago

There seems to be a nice duality between Bayesian and Frequentist inference [1]: Assume that both the system state and the observation are drawn from some joint probability distribution. There is some function γ of the system state which we seek to estimate. The experimentator applies some decision procedure d to the observation to get their result. A Frequentist will analyze the situation by conditioning on the the model parameter θ. As a result, we get a single target value γ and probability distributions for the observation and decision, depending on θ. If d results in an interval, the Frequentist calculates the confidence level as the probability that the descision procedure d produces an interval containing γ, under worst-case assumptions for θ. Unbiasedness of the decision procedure means that γ is indeed the function it estimates the best, and it is not a better estimator for any other function γ'(θ). A Bayesian, on the other hand, will condition the joint distribution on the observation. Consequently, γ is a random variable, while the observation and decision are known. If d is an interval, its credibility is the probability that γ is within this interval, given the observation. Optimality of the decision procedure means that no other estimator d' produces better results. [1]: S. Noorbaloochi, Unbiasedness and Bayes Estimators, users.stat.umn.edu/~gmeeden/papers/bayunb.pdf

6gvONxR4sf7o 6y ago
> Unbiasedness of the decision procedure means that γ is indeed the function it estimates the best, and it is not a better estimator for any other function γ'(θ). Are you suggesting that unbiased estimators are necessarily better than biased ones? If so, check out Stein’s phenomenon for a counterexample. It’s common for biased estimators to dominate unbiased ones in terms of error rates. That’s where the bias variance trade off in ML comes from. https://en.wikipedia.org/wiki/Stein's_example https://en.wikipedia.org/wiki/Stein's_example
- yccs27 6y ago
  Indeed, the unbiasedness condition for an estimator says nothing about its quality relative to other estimators. Instead, it requires that you cannot change the target function γ without the expected loss becoming worse. The paper I referenced even includes a theorem saying that, as long as the value to be estimated can not be exactly deduced from the observation, no estimator is both Bayes-optimal and unbiased. However for many observations, the Bayes-optimal estimator becomes asymptotically unbiased.

by yccs27 6y ago

6gvONxR4sf7o 6y ago

> Unbiasedness of the decision procedure means that γ is indeed the function it estimates the best, and it is not a better estimator for any other function γ'(θ). Are you suggesting that unbiased estimators are necessarily better than biased ones? If so, check out Stein’s phenomenon for a counterexample. It’s common for biased estimators to dominate unbiased ones in terms of error rates. That’s where the bias variance trade off in ML comes from. https://en.wikipedia.org/wiki/Stein's_example https://en.wikipedia.org/wiki/Stein's_example

yccs27 6y ago
Indeed, the unbiasedness condition for an estimator says nothing about its quality relative to other estimators. Instead, it requires that you cannot change the target function γ without the expected loss becoming worse. The paper I referenced even includes a theorem saying that, as long as the value to be estimated can not be exactly deduced from the observation, no estimator is both Bayes-optimal and unbiased. However for many observations, the Bayes-optimal estimator becomes asymptotically unbiased.