6 ms·

Two comments here, and I am sorry if they come across as mean: 1) I know this article isn't aimed at me, but I do truly hate the excessive emojis. 2) More sub

Two comments here, and I am sorry if they come across as mean: 1) I know this article isn't aimed at me, but I do truly hate the excessive emojis. 2) More substantively: not once does the word "statistics" enter here. There is not a single illustration of the idea behind linear regression, which is staggeringly simple: it is just finding the best linear fit. Anyone who looks at a 2D scatterplot can do an approximate linear regression with their brains by visualizing a line that sorta fits the data. While the gradient descent stuff is useful for extending a machine learning algorithm, introducing linear regression this way obscures the actual concept - and indeed, obscures what machine learning actually is. In my experience there are a huge number of data scientists who are ignorant of statistics and don't understand their models and algorithms. Although this is unfair, it is my impression that the author himself does not understand linear regression, even if they are able to write formulas using vectors. This article does a disservice to learners.

tluyben2 6y ago
> I do truly hate the excessive emojis. I am more in the camp of actually always finding them excessive. Especially in more serious, and possibly, interesting posts.
- ojnabieoot 6y ago
  Me too, but I am focusing more on the battle than the war :)
- grenoire 6y ago
  I would have loved to believe that they are a fad and will pass away, but I just found myself using them in the same way that I used to use phpBB emoticons. I think they're never going to go away, but at least we can try to eradicate them from professional discourse and education.
  - the_af 6y ago
    The homebanking login page from my bank greets me with "Hello :) How can we help you today?" and it makes me cringe every time. This is a major bank by the way. I do use smileys here on HN, on facebook, and chatting, to signal I'm being friendly and non-confrontational, to preempt and defuse situations which could escalate into anger if someone misreads the tone.
- EForEndeavour 6y ago
  Same here. They're the mental equivalent of speed bumps on a highway.
hansvm 6y ago
Stats are important, but aren't they somewhat excessive for explaining the "idea behind linear regression"? It's neat that this implementation is an unbiased estimator in the presence of gaussian noise, but statistical reasoning seems wholly unnecessary for getting _some_ line of best fit. I bet somebody could write an article much like this and gloss over most (maybe all?) of the math: 1) Here's a 2D scatterplot, and we want a line that kind of follows the dots. 2) If the data were on a line, this function (sum of the y values subtracted from a linear prediction) would be 0. 2a) Maybe introduce squaring here, but it might seem a little magical and unnecessary, so perhaps wait to introduce pathologies or other reasons till after an implementation of some kind. 3) The scatterplot obviously doesn't fall on a line, but let's try to get as close to 0 as we can. Picking random parameters seems good enough for an introduction to linear regression -- certainly not as efficient in the limit as gradient descent, but it requires next to no code or math to explain what we're doing to a new student. Future directions and blog posts) Other optimization algorithms, closed forms, why we care about squaring, other functions that work in addition to squaring, and now that we have a good idea of how linear regression might work let's add some statistical reasoning.
- zwaps 6y ago
  The conditional mean E(y|x), which exists under quite general conditions, is the best predictor in the conventional sense of MSE. The linear projection, assuming finite second moments and a positive definite expectation of the design matrix, is the best linear predictor in the sense of MSE. Its error is by construction (not by assumption) uncorrelated with the regressands. The linear regression comes in when one additionally assumes that the conditional expectation of the error given the explanatory variables is zero. This then gives unbiased estimates. No distributions required. To characterize the distributions and do testing/intervals, one usually employs large sample theory. This also then shows when the probability limit of the estimator converges to the true value (consistency) and at which speed. The idea of requiring Gaussian noise is, except for edge cases, depreciated. Textbooks that build on this should not be used anymore. It is misleading and it leads to the wide spread misconception that normality of the error term is in some sense essential for linear regression. It's fine to come from the CS angle of algorithms and optimizations, as opposed to theoretical distributions and all that, but semi-parametric statistics in quite useful to understand why and when linear regression can do really well.
windsignaling 6y ago
Linear regression isn't exclusive to statistics. One can, for example, find a chapter on linear regression in most machine learning textbooks, without reference to statistics. I've been on both sides of the fence, but it's always the statisticians who take the perspective of, "yeah, well you didn't study model diagnostics, ANOVA, QQ plots, ... so you don't know linear regression". Linear regression as described in this post (which is pretty much what you find in Andrew Ng's famed machine learning course) can be used as an introductory stepping stone to logistic regression, neural networks, etc.

tluyben2 6y ago

> I do truly hate the excessive emojis. I am more in the camp of actually always finding them excessive. Especially in more serious, and possibly, interesting posts.

ojnabieoot 6y ago
Me too, but I am focusing more on the battle than the war :)
grenoire 6y ago
I would have loved to believe that they are a fad and will pass away, but I just found myself using them in the same way that I used to use phpBB emoticons. I think they're never going to go away, but at least we can try to eradicate them from professional discourse and education.
- the_af 6y ago
  The homebanking login page from my bank greets me with "Hello :) How can we help you today?" and it makes me cringe every time. This is a major bank by the way. I do use smileys here on HN, on facebook, and chatting, to signal I'm being friendly and non-confrontational, to preempt and defuse situations which could escalate into anger if someone misreads the tone.
EForEndeavour 6y ago
Same here. They're the mental equivalent of speed bumps on a highway.

hansvm 6y ago

Stats are important, but aren't they somewhat excessive for explaining the "idea behind linear regression"? It's neat that this implementation is an unbiased estimator in the presence of gaussian noise, but statistical reasoning seems wholly unnecessary for getting _some_ line of best fit. I bet somebody could write an article much like this and gloss over most (maybe all?) of the math: 1) Here's a 2D scatterplot, and we want a line that kind of follows the dots. 2) If the data were on a line, this function (sum of the y values subtracted from a linear prediction) would be 0. 2a) Maybe introduce squaring here, but it might seem a little magical and unnecessary, so perhaps wait to introduce pathologies or other reasons till after an implementation of some kind. 3) The scatterplot obviously doesn't fall on a line, but let's try to get as close to 0 as we can. Picking random parameters seems good enough for an introduction to linear regression -- certainly not as efficient in the limit as gradient descent, but it requires next to no code or math to explain what we're doing to a new student. Future directions and blog posts) Other optimization algorithms, closed forms, why we care about squaring, other functions that work in addition to squaring, and now that we have a good idea of how linear regression might work let's add some statistical reasoning.

zwaps 6y ago
The conditional mean E(y|x), which exists under quite general conditions, is the best predictor in the conventional sense of MSE. The linear projection, assuming finite second moments and a positive definite expectation of the design matrix, is the best linear predictor in the sense of MSE. Its error is by construction (not by assumption) uncorrelated with the regressands. The linear regression comes in when one additionally assumes that the conditional expectation of the error given the explanatory variables is zero. This then gives unbiased estimates. No distributions required. To characterize the distributions and do testing/intervals, one usually employs large sample theory. This also then shows when the probability limit of the estimator converges to the true value (consistency) and at which speed. The idea of requiring Gaussian noise is, except for edge cases, depreciated. Textbooks that build on this should not be used anymore. It is misleading and it leads to the wide spread misconception that normality of the error term is in some sense essential for linear regression. It's fine to come from the CS angle of algorithms and optimizations, as opposed to theoretical distributions and all that, but semi-parametric statistics in quite useful to understand why and when linear regression can do really well.

windsignaling 6y ago

Linear regression isn't exclusive to statistics. One can, for example, find a chapter on linear regression in most machine learning textbooks, without reference to statistics. I've been on both sides of the fence, but it's always the statisticians who take the perspective of, "yeah, well you didn't study model diagnostics, ANOVA, QQ plots, ... so you don't know linear regression". Linear regression as described in this post (which is pretty much what you find in Andrew Ng's famed machine learning course) can be used as an introductory stepping stone to logistic regression, neural networks, etc.