4 ms·

Can someone explain to me why this has (so many) upvotes? This is like elementary undergraduate econ stats and kind of trivial? There's very little content eit

Can someone explain to me why this has (so many) upvotes? This is like elementary undergraduate econ stats and kind of trivial? There's very little content either, it's literally a reformulation of the formula, no interesting graphs or geometric interpretation. What I expected from a title like "Linear Regression By Hand" was the minimization of some quadratic error function, by hand (i.e. using pencil and paper).

bluusteel 9y ago
It seems like blog posts on simple statistical methods like this one land on the front page of HNews a lot more than one would expect.
thess24 9y ago
I agree, it seems like no help just to list a formula to memorize. If someone knows enough linear algebra to understand what the formula represents, they can do the derivation. This link [1] is a good one if anyone is interested. [1] https://eli.thegreenplace.net/2014/derivation-of-the-normal-equation-for-linear-regression https://eli.thegreenplace.net/2014/derivation-of-the-normal-...
luckyt 9y ago
Well, unless you're doing original research, everything you write will be trivial to somebody.
theophrastus 9y ago
One issue that I nearly always find missing in intro discussions about linear regression is the near universal assumption of no error in the abcissal/"x" values. And while this is true-ish for time series data, (we know for certain which day we collected the data on - yet the same hour every day?), I'd be rich if I had a nickel for every time I saw standard linear regression done when the "x" had significant (and known) error. In which case you're biasing yourself unless you use some sort of 2d regression, like Deming.[1] [1] https://en.wikipedia.org/wiki/Deming_regression https://en.wikipedia.org/wiki/Deming_regression
- VHRanger 9y ago
  Regression with measurement error is usually treated in much higher level statistics/econometrics classes. If you're interested in this you can read more in Mostly Harmless Econometrics [1] about adressing this with IV methods [1] http://www.development.wne.uw.edu.pl/uploads/Main/recrut_econometrics.pdf http://www.development.wne.uw.edu.pl/uploads/Main/recrut_eco...
  - pacbard 9y ago
    To build on this a little bit more, there are also generalized linear models that allow to specify the reliability (i.e., error level) of a variable. Regarding, 2SLS models, I find them more useful to account for endogeneity in the model rather than measurement error. After all, measurement error is usually unobserved (otherwise you would just take it out). 2SLS “just” reweigh the point estimates by identifying the good variation in the instrumented variable using the instrument (for example, using draft lottery results to account for the endogenous choice to attend college).
VHRanger 9y ago
I think a lot of people here didn't take econometrics courses. If you see all regression problems under the foil of maximum likelihood estimation, you might not know that ordinary least squares regression has a closed form solution
thaumaturgy 9y ago
https://www.xkcd.com/1053/ https://www.xkcd.com/1053/ I almost certainly know something I'd consider "trivial" that you haven't encountered yet. I try to be really excited when that happens.
- baking 9y ago
  I think that perhaps the issue is that machine learning courses skip over the fact that there is a trivial closed form solution to 99.9% of all real-world machine learning problems.
triplesec 9y ago
My suspicion, for the Eeyores round here, is that this has upvotes because people like to bookmark a reminder of references to remind you of the maths behind what most of us do automagically nowadays, for when we might need it. And often it's the comments that provide even better and advance sources, as is the case here. And indeed there are some programmers new to this kind of thing.
abecedarius 9y ago
What I expected "by hand" to mean was something like a handmade analog computer. E.g. print your scatterplot, tape a penny over each data point, push a tack through the origin and let the printout swing around it under gravity until it comes to rest (while keeping it flat -- I guess I should've first stuck the printout onto some cardboard). Is there some generalization of this idea that lets the intercept vary too?
mr_toad 9y ago
There aren’t that many good articles on basic statistics that are freely available on the net. Textbooks are expensive, and most of the free material is of dubious quality. Wikipedia is particularly terrible on Statistics.

bluusteel 9y ago

It seems like blog posts on simple statistical methods like this one land on the front page of HNews a lot more than one would expect.

thess24 9y ago

I agree, it seems like no help just to list a formula to memorize. If someone knows enough linear algebra to understand what the formula represents, they can do the derivation. This link [1] is a good one if anyone is interested. [1] https://eli.thegreenplace.net/2014/derivation-of-the-normal-equation-for-linear-regression https://eli.thegreenplace.net/2014/derivation-of-the-normal-...

luckyt 9y ago

Well, unless you're doing original research, everything you write will be trivial to somebody.

theophrastus 9y ago

One issue that I nearly always find missing in intro discussions about linear regression is the near universal assumption of no error in the abcissal/"x" values. And while this is true-ish for time series data, (we know for certain which day we collected the data on - yet the same hour every day?), I'd be rich if I had a nickel for every time I saw standard linear regression done when the "x" had significant (and known) error. In which case you're biasing yourself unless you use some sort of 2d regression, like Deming.[1] [1] https://en.wikipedia.org/wiki/Deming_regression https://en.wikipedia.org/wiki/Deming_regression

VHRanger 9y ago
Regression with measurement error is usually treated in much higher level statistics/econometrics classes. If you're interested in this you can read more in Mostly Harmless Econometrics [1] about adressing this with IV methods [1] http://www.development.wne.uw.edu.pl/uploads/Main/recrut_econometrics.pdf http://www.development.wne.uw.edu.pl/uploads/Main/recrut_eco...
- pacbard 9y ago
  To build on this a little bit more, there are also generalized linear models that allow to specify the reliability (i.e., error level) of a variable. Regarding, 2SLS models, I find them more useful to account for endogeneity in the model rather than measurement error. After all, measurement error is usually unobserved (otherwise you would just take it out). 2SLS “just” reweigh the point estimates by identifying the good variation in the instrumented variable using the instrument (for example, using draft lottery results to account for the endogenous choice to attend college).

VHRanger 9y ago

I think a lot of people here didn't take econometrics courses. If you see all regression problems under the foil of maximum likelihood estimation, you might not know that ordinary least squares regression has a closed form solution

thaumaturgy 9y ago

https://www.xkcd.com/1053/ https://www.xkcd.com/1053/ I almost certainly know something I'd consider "trivial" that you haven't encountered yet. I try to be really excited when that happens.

baking 9y ago
I think that perhaps the issue is that machine learning courses skip over the fact that there is a trivial closed form solution to 99.9% of all real-world machine learning problems.

triplesec 9y ago

My suspicion, for the Eeyores round here, is that this has upvotes because people like to bookmark a reminder of references to remind you of the maths behind what most of us do automagically nowadays, for when we might need it. And often it's the comments that provide even better and advance sources, as is the case here. And indeed there are some programmers new to this kind of thing.

abecedarius 9y ago

What I expected "by hand" to mean was something like a handmade analog computer. E.g. print your scatterplot, tape a penny over each data point, push a tack through the origin and let the printout swing around it under gravity until it comes to rest (while keeping it flat -- I guess I should've first stuck the printout onto some cardboard). Is there some generalization of this idea that lets the intercept vary too?

mr_toad 9y ago

There aren’t that many good articles on basic statistics that are freely available on the net. Textbooks are expensive, and most of the free material is of dubious quality. Wikipedia is particularly terrible on Statistics.