4 ms·

I dont like this article. It mentions, but glosses over, the real reason squared error is used. Assuming a Gaussian error term, squared error yields the maximum

I dont like this article. It mentions, but glosses over, the real reason squared error is used. Assuming a Gaussian error term, squared error yields the maximum likelihood estimator. That is the core reason to use squares - if the error is not normal, it just won't work. This is why, e.g., a small number of outliers can wildly skew a least squares fit - if error was normal, they simply wouldn't exist.

darkmighty 12y ago
Yes, all mathematical niceties don't matter if MMSE estimation is not a good estimation in practice. This is the reason Gauss invented MMSE in the first place along with the eponymous distribution, while working with astronomical observations iirc.
ryan_j_naughton 12y ago
Absolutely true. I wanted add a note on it being BLUE (the best linear unbiased estimator) "The Gauss–Markov theorem states that in a linear regression model in which the errors have expectation zero and are uncorrelated and have equal variances, the best linear unbiased estimator (BLUE) of the coefficients is given by the ordinary least squares (OLS) estimator."[1] [1] http://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem http://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem
- wfunction 12y ago
  This is precisely what I never found convincing: it never explains why we should be looking for a linear unbiased estimator in the first place.
  - ryan_j_naughton 11y ago
    Very good point. We shouldn't necessarily. The benefit of a linear estimator is it is a lot easier (both computationally and algorithmically) to fit/solve a linear model than a non-linear model. Thus, there are advantages to linear models. Further, despite it being called linear, the model can actually be quite complex. It only needs to be linear in the model coefficients, but the regressors/independent variables don't have to be linear. Thus, you can make one variable the square of another variable or add a variable that is the interaction of two other variables. By doing so, you can add non-linearity while still using linear methods to solve the equation. Ok, so then once you see the value of a linear model, then the next question is why we would want the best and most unbiased. I think those are rather obvious.
    - wfunction 11y ago
      > The next question is why we would want the best and most unbiased. I think those are rather obvious. It's not obvious to me why we want an unbiased estimator actually. A Maximum-Likelihood estimate, for example, seems just as reasonable. Do you have an explanation? Not just that, but there's also the question of why we're looking for point estimates in the first place. It isn't the correct thing to do, but no one ever gives an explanation.
masterjack 12y ago
Good point, that really is the key. However, itjust turns it into the almost equivalent problem of: why assume Gaussian error terms, and the rest of the article does a good job at motivating this.
- yummyfajitas 12y ago
  Gaussian is sometimes a good approximation if the error is a sum of independent thin-tailed variables (due to the c.l.t.). That is all. The article does not, in my opinion, motivate this very well.
azakai 12y ago
I'm not sure why you feel that is the "real" reason. It is a strong one, of course, but the fact that the squared error has a derivative - and a nice and simple one - is equally strong in my opinion.
- yummyfajitas 12y ago
  Having a derivative is computationally nice. But if your error term is not normal, that only means you can easily compute the wrong answer. I.e. its great if you dropped your key under the lamppost, but the existence of the lamppost doesn't make it a good place to search.
hessenwolf 12y ago
Nah. Least squares is necessarily optimal under a normal distribution, but that does not imply that it does not work if the distribution is not normal. It might even be optimal for some other distributions, I just cannot remember, and I have to watch the frying pan.
- yummyfajitas 12y ago
  Least squares is disastrously bad for anything with a fat tail (e.g., power law decay). The reason is that in these cases one or two outliers will dominate the sum of the tails. It's not optimal for any other distribution. For a general error distribution g(x), you want to maximize sum(log(g(x-x0))) (or equivalently prod(g(x-x0))) - this only turns to least squares if log(g(x)) = C + D(x-x0)^2.
  - hessenwolf 11y ago
    That sounds neat. Have you got a reference to it handy?
    - yummyfajitas 11y ago
      I don't know of any references, most of this is just stuff that falls out pretty easily once you try and do the math, and that's probably faster than reading a book. I.e., set up maximal likelihood, take logs, and you immediately get least squares for g(x) a gaussian. If g(x)=exp(-|x|), you get l1 minimization. Other distributions give you other things. The general topic to investigate is robust regression: https://en.wikipedia.org/wiki/Robust_regression https://en.wikipedia.org/wiki/Robust_regression

darkmighty 12y ago

Yes, all mathematical niceties don't matter if MMSE estimation is not a good estimation in practice. This is the reason Gauss invented MMSE in the first place along with the eponymous distribution, while working with astronomical observations iirc.

ryan_j_naughton 12y ago

Absolutely true. I wanted add a note on it being BLUE (the best linear unbiased estimator) "The Gauss–Markov theorem states that in a linear regression model in which the errors have expectation zero and are uncorrelated and have equal variances, the best linear unbiased estimator (BLUE) of the coefficients is given by the ordinary least squares (OLS) estimator."[1] [1] http://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem http://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem

wfunction 12y ago
This is precisely what I never found convincing: it never explains why we should be looking for a linear unbiased estimator in the first place.
- ryan_j_naughton 11y ago
  Very good point. We shouldn't necessarily. The benefit of a linear estimator is it is a lot easier (both computationally and algorithmically) to fit/solve a linear model than a non-linear model. Thus, there are advantages to linear models. Further, despite it being called linear, the model can actually be quite complex. It only needs to be linear in the model coefficients, but the regressors/independent variables don't have to be linear. Thus, you can make one variable the square of another variable or add a variable that is the interaction of two other variables. By doing so, you can add non-linearity while still using linear methods to solve the equation. Ok, so then once you see the value of a linear model, then the next question is why we would want the best and most unbiased. I think those are rather obvious.
  - wfunction 11y ago
    > The next question is why we would want the best and most unbiased. I think those are rather obvious. It's not obvious to me why we want an unbiased estimator actually. A Maximum-Likelihood estimate, for example, seems just as reasonable. Do you have an explanation? Not just that, but there's also the question of why we're looking for point estimates in the first place. It isn't the correct thing to do, but no one ever gives an explanation.

masterjack 12y ago

Good point, that really is the key. However, itjust turns it into the almost equivalent problem of: why assume Gaussian error terms, and the rest of the article does a good job at motivating this.

yummyfajitas 12y ago
Gaussian is sometimes a good approximation if the error is a sum of independent thin-tailed variables (due to the c.l.t.). That is all. The article does not, in my opinion, motivate this very well.

azakai 12y ago

I'm not sure why you feel that is the "real" reason. It is a strong one, of course, but the fact that the squared error has a derivative - and a nice and simple one - is equally strong in my opinion.

yummyfajitas 12y ago
Having a derivative is computationally nice. But if your error term is not normal, that only means you can easily compute the wrong answer. I.e. its great if you dropped your key under the lamppost, but the existence of the lamppost doesn't make it a good place to search.

hessenwolf 12y ago

Nah. Least squares is necessarily optimal under a normal distribution, but that does not imply that it does not work if the distribution is not normal. It might even be optimal for some other distributions, I just cannot remember, and I have to watch the frying pan.

yummyfajitas 12y ago
Least squares is disastrously bad for anything with a fat tail (e.g., power law decay). The reason is that in these cases one or two outliers will dominate the sum of the tails. It's not optimal for any other distribution. For a general error distribution g(x), you want to maximize sum(log(g(x-x0))) (or equivalently prod(g(x-x0))) - this only turns to least squares if log(g(x)) = C + D(x-x0)^2.
- hessenwolf 11y ago
  That sounds neat. Have you got a reference to it handy?
  - yummyfajitas 11y ago
    I don't know of any references, most of this is just stuff that falls out pretty easily once you try and do the math, and that's probably faster than reading a book. I.e., set up maximal likelihood, take logs, and you immediately get least squares for g(x) a gaussian. If g(x)=exp(-|x|), you get l1 minimization. Other distributions give you other things. The general topic to investigate is robust regression: https://en.wikipedia.org/wiki/Robust_regression https://en.wikipedia.org/wiki/Robust_regression