3 ms·

The basic concept of least-squares modelling

17 points by gballan 11y ago

dvh 11y ago
When explaining the least-squares, why not drawing the damn squares in the chart then?
- shoo 11y ago
  one of my colleagues was lamenting the fact that he made it all the way through pre-university mathematics education without anyone explaining the geometry link between "x-squared", i.e. "x^2", and the area of a square of side-length x. so this is a pretty reasonable thing to ask for.
- gballan 11y ago
  Author (and Blabr dev) here. I have no answer--other than it seems like a really good idea, now that you've said it. The plotting widget I used is based on flot [0]. It looks like flot can draw simple shapes ([1], see "markings"). I'll look into improving the widget--thanks for the idea. [0] http://www.flotcharts.org/ http://www.flotcharts.org/ [1] https://github.com/flot/flot/blob/master/API.md https://github.com/flot/flot/blob/master/API.md
shoo 11y ago
This is a nice interactive example, but it doesn't explain _why_ one would prefer to use a least-squares estimator over any other arbitrary method. For example, one could measure the error in any other L_p norm - e.g. L_1 or L_{\infty} instead of L_2. What's so special about this one? To answer that, we have the Gauss-Markov theorem [1] -- provided certain assumptions regarding the data are met, least squares is the best linear unbiased estimator. Note, however, that no-one is forcing us to choose an estimator that is either linear or unbiased. What if by dropping one of those two somewhat arbitrary constraints we could get a better estimator? One example of this is the James-Stein estimator [2], which is a biased estimator that has the amusing property of dominating (i.e., always performing no worse than, and performing strictly better than in some case) the bog standard least-squares estimate when one is dealing with data in three or more dimensions (and a few more common assumptions). Crudely speaking, the James-Stein estimator produces estimates that are biased toward the origin, when compared to estimates produced from ordinary least squares. It is also possible to define a James-Stein estimator that biases toward an arbitrary fixed non-origin vector -- these variants also dominate the bog-standard ordinary least squares estimator! Personally I find this result both intriguing and also reasonably offensive (which means my mathematical intuition needs adjustment). [1] - https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem [2] - https://en.wikipedia.org/wiki/James%E2%80%93Stein_estimator https://en.wikipedia.org/wiki/James%E2%80%93Stein_estimator
rmohanx 11y ago
@shoo: Pretty interesting, Stein. Thx. Taleb advocates abs. deviation, mean abs. dev. and the like. OP named reason to pick least squares: "easy to calc" IIRC, the rest of the rationale didn't exist back when it was invented and picked....

17 points by gballan 11y ago

dvh 11y ago

When explaining the least-squares, why not drawing the damn squares in the chart then?

shoo 11y ago
one of my colleagues was lamenting the fact that he made it all the way through pre-university mathematics education without anyone explaining the geometry link between "x-squared", i.e. "x^2", and the area of a square of side-length x. so this is a pretty reasonable thing to ask for.
gballan 11y ago
Author (and Blabr dev) here. I have no answer--other than it seems like a really good idea, now that you've said it. The plotting widget I used is based on flot [0]. It looks like flot can draw simple shapes ([1], see "markings"). I'll look into improving the widget--thanks for the idea. [0] http://www.flotcharts.org/ http://www.flotcharts.org/ [1] https://github.com/flot/flot/blob/master/API.md https://github.com/flot/flot/blob/master/API.md

shoo 11y ago

This is a nice interactive example, but it doesn't explain _why_ one would prefer to use a least-squares estimator over any other arbitrary method. For example, one could measure the error in any other L_p norm - e.g. L_1 or L_{\infty} instead of L_2. What's so special about this one? To answer that, we have the Gauss-Markov theorem [1] -- provided certain assumptions regarding the data are met, least squares is the best linear unbiased estimator. Note, however, that no-one is forcing us to choose an estimator that is either linear or unbiased. What if by dropping one of those two somewhat arbitrary constraints we could get a better estimator? One example of this is the James-Stein estimator [2], which is a biased estimator that has the amusing property of dominating (i.e., always performing no worse than, and performing strictly better than in some case) the bog standard least-squares estimate when one is dealing with data in three or more dimensions (and a few more common assumptions). Crudely speaking, the James-Stein estimator produces estimates that are biased toward the origin, when compared to estimates produced from ordinary least squares. It is also possible to define a James-Stein estimator that biases toward an arbitrary fixed non-origin vector -- these variants also dominate the bog-standard ordinary least squares estimator! Personally I find this result both intriguing and also reasonably offensive (which means my mathematical intuition needs adjustment). [1] - https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem [2] - https://en.wikipedia.org/wiki/James%E2%80%93Stein_estimator https://en.wikipedia.org/wiki/James%E2%80%93Stein_estimator

rmohanx 11y ago

@shoo: Pretty interesting, Stein. Thx. Taleb advocates abs. deviation, mean abs. dev. and the like. OP named reason to pick least squares: "easy to calc" IIRC, the rest of the rationale didn't exist back when it was invented and picked....