3 ms·
Stats are important, but aren't they somewhat excessive for explaining the "idea behind linear regression"? It's neat that this implementation is an unbiased es
by hansvm 6y ago
Stats are important, but aren't they somewhat excessive for explaining the "idea behind linear regression"? It's neat that this implementation is an unbiased estimator in the presence of gaussian noise, but statistical reasoning seems wholly unnecessary for getting _some_ line of best fit.
I bet somebody could write an article much like this and gloss over most (maybe all?) of the math:
1) Here's a 2D scatterplot, and we want a line that kind of follows the dots.
2) If the data were on a line, this function (sum of the y values subtracted from a linear prediction) would be 0.
2a) Maybe introduce squaring here, but it might seem a little magical and unnecessary, so perhaps wait to introduce pathologies or other reasons till after an implementation of some kind.
3) The scatterplot obviously doesn't fall on a line, but let's try to get as close to 0 as we can. Picking random parameters seems good enough for an introduction to linear regression -- certainly not as efficient in the limit as gradient descent, but it requires next to no code or math to explain what we're doing to a new student.
Future directions and blog posts) Other optimization algorithms, closed forms, why we care about squaring, other functions that work in addition to squaring, and now that we have a good idea of how linear regression might work let's add some statistical reasoning.
- zwaps 6y agoThe conditional mean E(y|x), which exists under quite general conditions, is the best predictor in the conventional sense of MSE. The linear projection, assuming finite second moments and a positive definite expectation of the design matrix, is the best linear predictor in the sense of MSE. Its error is by construction (not by assumption) uncorrelated with the regressands. The linear regression comes in when one additionally assumes that the conditional expectation of the error given the explanatory variables is zero. This then gives unbiased estimates. No distributions required. To characterize the distributions and do testing/intervals, one usually employs large sample theory. This also then shows when the probability limit of the estimator converges to the true value (consistency) and at which speed. The idea of requiring Gaussian noise is, except for edge cases, depreciated. Textbooks that build on this should not be used anymore. It is misleading and it leads to the wide spread misconception that normality of the error term is in some sense essential for linear regression. It's fine to come from the CS angle of algorithms and optimizations, as opposed to theoretical distributions and all that, but semi-parametric statistics in quite useful to understand why and when linear regression can do really well.