4 ms·

A small but vital correction is that the "biased" estimator will approach a population's std dev as the sample size increases. This is clear since the adjustmen

by tel 16y ago

A small but vital correction is that the "biased" estimator will approach a population's std dev as the sample size increases. This is clear since the adjustment is N/(N-1) which tends to 1 with large N. It's not often a huge deal, but I'm unconvinced that biased estimators deserve such a derogatory name. Using a unbiased estimate does not actually mean you've got a better estimator. I think this is part of why there seems to be no intuitive description of why the N/(N-1) factor removes bias. You instead need to invoke the powerful, abstract ideas of sufficiency and completeness.

lmkg 16y ago
The "intuitive" explanation I've heard is that (N-1) is the number of degrees of freedom of a sample with N data points, so when you divide by (N-1) you're normalizing some valuation of the "spread" of the sample against a measure of the potential for the sample to "spread." I think how "intuitive" this idea is depends on how well you understand thermodynamics. I, clearly, do not.
- tel 16y ago
  I've heard that description before and find it more ad hoc than intuitive. Is there some valid mathematical object K such that often times you will normalize unbiased estimators by N/(N-K)? Is this K sometimes equal to some measure of the number of equations/constraints you have? Does this seem to hold a lot of force in ANOVA especially? Sure, sure, sure. I still don't see why this normalization is required to make a better estimate (excepting when you define "better" as an unbiased estimator, maybe the MVUE).
jules 16y ago
This is the problem with frequentist estimators: it's not clear what part is math and what part is subjective guessing.
- tel 16y ago
  I'm completely certain that the unbiased estimators are a class of estimators that eliminate bias. I'm completely certain that the MVUE is the one of those with the highest Fisher information subject to the Cramer-Rao bound. Those arguments are abstract, but simple. I've just never been too sure why I should prefer unbiasedness to minimizing error (or, more generally, minimizing a cost function of some variety). I've never understood when, theoretically, estimating a 'fixed' parameter is better than pruning it's possible distribution. But, I've also never doubted statements of frequentist statistics made in full rigor. I don't believe they're guessing, I just think they're playing a game that they made up the rules too but got them tragically wrong.
  - jules 16y ago
    Frequentist estimators try to estimate an uncertain quantity, but they don't define what the goal is. Is the goal to get as close to the true value as possible on average (i.e. minimize |t-e| where t is the true value and e is the estimator). Is the goal to minimize (t-e)^2? To minimize some other cost function? No, they don't define the goal all they do is define some desirable properties that don't fully specify the problem. It's like asking a physicist "how do I throw a ball". He can't give a good answer unless you define what your goal is. Is the goal to throw as high as possible? Throw straight up. Is the goal to throw as far as possible? Throw at 45 degrees.
    - tel 16y ago
      Generally, they seem to begin with (t-e)^2 which they rapidly note the orthogonality decomposition between "bias" and variance. The decomposition appears metaphorically in other error functions as well. This is often mentioned as a supportive point, but I'm not so sure it's meaningful. Finding minimization of MSE difficult, frequentists choose to minimize bias first. The method of the MVUE produces a low MSE estimator from the class of unbiased estimators and thus, automatically, ignores low variance biased estimators which may or may not outperform the MVUE. Frankly, though, the bias-variance decomposition seems too widely interpreted, especially as an expectation over all possible data instead of something considering the problem of inference with a known, fixed set of data. The difference here is thus philosophical and hard to mathematically analyze.

by tel 16y ago

lmkg 16y ago

The "intuitive" explanation I've heard is that (N-1) is the number of degrees of freedom of a sample with N data points, so when you divide by (N-1) you're normalizing some valuation of the "spread" of the sample against a measure of the potential for the sample to "spread." I think how "intuitive" this idea is depends on how well you understand thermodynamics. I, clearly, do not.

tel 16y ago
I've heard that description before and find it more ad hoc than intuitive. Is there some valid mathematical object K such that often times you will normalize unbiased estimators by N/(N-K)? Is this K sometimes equal to some measure of the number of equations/constraints you have? Does this seem to hold a lot of force in ANOVA especially? Sure, sure, sure. I still don't see why this normalization is required to make a better estimate (excepting when you define "better" as an unbiased estimator, maybe the MVUE).

jules 16y ago

This is the problem with frequentist estimators: it's not clear what part is math and what part is subjective guessing.

tel 16y ago
I'm completely certain that the unbiased estimators are a class of estimators that eliminate bias. I'm completely certain that the MVUE is the one of those with the highest Fisher information subject to the Cramer-Rao bound. Those arguments are abstract, but simple. I've just never been too sure why I should prefer unbiasedness to minimizing error (or, more generally, minimizing a cost function of some variety). I've never understood when, theoretically, estimating a 'fixed' parameter is better than pruning it's possible distribution. But, I've also never doubted statements of frequentist statistics made in full rigor. I don't believe they're guessing, I just think they're playing a game that they made up the rules too but got them tragically wrong.
- jules 16y ago
  Frequentist estimators try to estimate an uncertain quantity, but they don't define what the goal is. Is the goal to get as close to the true value as possible on average (i.e. minimize |t-e| where t is the true value and e is the estimator). Is the goal to minimize (t-e)^2? To minimize some other cost function? No, they don't define the goal all they do is define some desirable properties that don't fully specify the problem. It's like asking a physicist "how do I throw a ball". He can't give a good answer unless you define what your goal is. Is the goal to throw as high as possible? Throw straight up. Is the goal to throw as far as possible? Throw at 45 degrees.
  - tel 16y ago
    Generally, they seem to begin with (t-e)^2 which they rapidly note the orthogonality decomposition between "bias" and variance. The decomposition appears metaphorically in other error functions as well. This is often mentioned as a supportive point, but I'm not so sure it's meaningful. Finding minimization of MSE difficult, frequentists choose to minimize bias first. The method of the MVUE produces a low MSE estimator from the class of unbiased estimators and thus, automatically, ignores low variance biased estimators which may or may not outperform the MVUE. Frankly, though, the bias-variance decomposition seems too widely interpreted, especially as an expectation over all possible data instead of something considering the problem of inference with a known, fixed set of data. The difference here is thus philosophical and hard to mathematically analyze.