6 ms·
Linear regression
- thanatropism 10y agoThe single most important fact about linear regression is the Fritsch-Waugh-Lovell theorem. It's the whole reason why we're not scared as shit of using linear models. https://en.wikipedia.org/wiki/Frisch%E2%80%93Waugh%E2%80%93Lovell_theorem https://en.wikipedia.org/wiki/Frisch%E2%80%93Waugh%E2%80%93L... Basically, it means that omitting variables is ok (subject to terms and conditions). Assuming that IQ and shoe size are uncorrelated, but that income is a function of IQ and shoe size, I can run a regression: income = a + b * IQ + residuals and trust my estimate of b. FWL theorem says that running that regression and then residuals = c + d * shoe size is the same as running income = (a+c) + bIQ + dshoe size + eps It's an amazing result and it's all based on linear algebra, no belief in statistics required.
- apathy 10y agoEh, orthogonalizing variables is a necessary precondition for this. When was the last time variables were totally uncorrelated and you didn't know it beforehand? Unfortunately such cases are the exception rather than the rule, and you lose interpretability by projecting into a low-rank orthogonal subspace. Not that interpretability is necessarily the be-all and end-all, but the "terms and conditions" are pretty important here!
- thanatropism 10y ago> When was the last time variables were totally uncorrelated and you didn't know it beforehand Whenever we have an actual theory of the underlying phenomena. It's common in econometrics to impose orthogonality conditions on the theoretical equations so structural parameters can be recovered; or so we can filter partial correlations. For example: a regression of prices and quantities sold will mix the effect of shifts in demand and in supply. Solution: find something that correlated with demand quantities but not price directly (for example, changes in consumers' wealth). A realistic, much cooler example: http://homepage.univie.ac.at/robert.kunst/pan2011_pres_rabas.pdf http://homepage.univie.ac.at/robert.kunst/pan2011_pres_rabas...
- smallnamespace 10y ago> no belief in statistics required Yes, but now you have a statistical assumption: that your structural model holds. To make use of the Fritsch-Waugh-Lovell theorem, you must test that your orthogonality assumption holds using statistics :)
- apathy 10y agoCtrl-F "Gauss" 0 matches Ctrl-F "Markov" 0 matches The fuck? Only the single most beautiful result in the field. Please don't misunderstand me, the presentation is great and it is quite practical, I just wish people would remember that very smart individuals have been coming along for pretty much the entirety of humanity's existence. https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem The proof is hinted at in the writeup, so why not link to it?
- darkhorn 10y agoI don't understand why they down vote you. Probably because they think that they know Statistics. Programmers at Hacker News always think that they are the master mind of Statistics. In fact they know almost nothing about Statistics. For example if you don't test for constand variance and independance of variances then you cannot graduate from Department ofStatistis. But here they down vote you for knowing real Statistics.
- minimaxir 10y ago> But here they down vote you for knowing real Statistics. That is quite literally a No True Scotsman argument. GP was likely downvoted since the Ctrl-F meme adds nothing to the discussion.
- apathy 10y agoPersonally, I felt that the omission was rather important. When the first year's worth of many statistics programs can be condensed into one beautiful theorem, which is centuries old, I feel it's disrespectful to one's forebears to ignore it. However, I get the impression that once the meme sensor goes off, any links remain unvisited and reading comprehension may suffer. So I learned something here, too. I'd edit the original to remove the meme, but since I can't do that at this point, readers will have to try and control their Internet Police tendencies and read a few sentences. Or not. Rather a shame, hopefully the author of the piece will fix it and this will all be superfluous.
- murbard2 10y agoA common gotcha with linear regression. Imagine that your data points look like they roughly fit the shape of a tilted ellipsis. The linear regression will not run along the axis of the ellipsis, the slope will typically be smaller. This is because you're minimizing the square distance along the y axis, not the distance from a point to the line itself.
- thanatropism 10y agoWhat you appear to want is "total least squares": https://en.wikipedia.org/wiki/Total_least_squares https://en.wikipedia.org/wiki/Total_least_squares
- klodolph 10y agoDo you mean "ellipse"? I was a bit confused about the "ellipsis" comment, since an ellipsis looks like a collection of collinear points.
- imurray 10y agoIf you're new to learning about regression and want to use it, I would prioritize the statistical issues. There are lots of computational details you could read about, like in this post. However in simple cases, fitting least-squares linear regression is one line of code: w_fit = X \ yy; % Matlab/Octave, X is NxD, yy is Nx1 w_fit = np.linalg.lstsq(X, yy)[0] # NumPy equivalent Knowing how to construct the "design matrix" X is the important bit. What are your features, how will they be pre-processed, and given what you've done, should you trust w_fit to be meaningful, or just useful for prediction? If you've thrown in lots of features, hoping to get good predictions, you may want to do "regularization". The computation is no harder for simple versions. One way to do "L2 regularization" is to add some extra rows to X and yy before fitting the weights: % Matlab/Octave... the NumPy is nearly as simple. [N, D] = size(X); lambda = 1; % regularization constant X = [X; eye(D)*sqrt(lambda)]; y = [yy; zeros(D,1)] Or you could use an actual stats package like R's libraries that will contain a lot more care. Again, you need to understand the statistics, for example how you should select lambda. Then you'll want to know how to diagnose your model and whether you should trust it. It's fun to work out the least squares maths and implement these things from scratch. But it has been done, and it's not what matters for most applications. EDIT: a discussion with good links on linear models and beyond from the other day https://news.ycombinator.com/item?id=12237998 https://news.ycombinator.com/item?id=12237998
- geezerjay 10y ago> If you're new to learning about regression and want to use it, I would prioritize the statistical issues. There is absolutely no need to take the statistical route in least squares regression. In fact, it needlessly complicates things. Interpreting least squares regression as the minimization of the sum of squares of the residual is a straight-forward way to understand the technique.
- imurray 10y agoI'm not sure what you mean, but I think we're talking past each other. Regression is a statistical term. If it's going to be used for an application involving data and real decisions, not understanding the statistical issues is irresponsible. That's not at odds with viewing it as a procedure to minimize square errors. To clarify: There are probabilistic models (involving Gaussians and stuff) where some interpretations suggest least-squares estimation. That wasn't the distinction I was trying to make. Statisticians who hate probabilistic models will still say that it's important to think about what you put in X and y before you do X\y, and to be careful about what you can conclude from the result.
- theophrastus 10y agoThere is one conceptually simple issue often missing in such nicely presented write-ups (and which appears to be missing here): error in the abscissa ('x-axis') values. Things like time-series tend to dominate such analytics and in such data collections it's typically assumed that the time-stamp data is of suitable accuracy that any error there can be neglected. But there are many other data sources where there is notable error in both 'x' and 'y' data for which commonly employed linear regression doesn't allow, (quick example: my friend the hydrologist collects flow rates in rivers at sample transverse distances which are hard to be sure of as one is dangling above the raging waters). As a respectable starting point for regression which allows for error in both axes i'd recommend Deming regression: https://en.wikipedia.org/wiki/Deming_regression https://en.wikipedia.org/wiki/Deming_regression
- geezerjay 10y agoThose cases are also handled by least squares regression, mainly variants of total least squares regression. https://en.wikipedia.org/wiki/Total_least_squares https://en.wikipedia.org/wiki/Total_least_squares
- p4wnc6 10y agoOne special case of this general effect is non-linear coding error, especially in cases when it ends up being the level of a covariate (i.e. the log of the covariate) that matters for causal inference, or when the covariates are categorical or isotonic. The paper "Let's Put Garbage Can Regressions and Garbage Can Probits Where They Belong" by Achen [0] is a great discussion about some particular properties of this, and the tacit assumptions used to ignore it. In that paper, it's demonstrated that with just a tiny bit of coding error in the covariates, you can end up with a fitted regression coefficient that is statistically significant and has the wrong sign -- even when there is no noise whatsoever in the target variable (i.e. you can set up a toy example in which the target variable is synthetically generated as a true linear function of two covariates with positive coefficients, then perform a slight non-linear distortion on one of the covariates, regress the synthetic target variable on the clean covariate and the distorted covariate, and get wildly incorrect coefficients that appear to be statistically significant). People seem to think these toy example are some kind of alien phenomenon that could never happen with real-world data, but the paper is very explicit in the construction of the example data set. It's not harebrained or contrived, like Anscombe's Quartet or anything -- it's very much a plausible data set. I think it's not hyperbolic at all to say that results like this more or less conclusively show that naive linear regression cannot be trusted. If you're careful with model validation, using randomized hold out data, lots of diagnostic plotting and sanity checking, then regression is a fine tool. But if you do something shocking like take two different univariate models with the same target, fit their regression coefficients, and then select the model with a more favorable t-stat as "the winner" then you are committing an egregious statistical fallacy that often, in real world situations, is giving you not just an inaccurate answer, but an answer pointing totally in the opposite direction of the truth. What's frightening to me is that across many industries, even in places like high finance -- where "real money is on the line" -- it is extremely common to see huge business intelligence systems predicated entirely on this type of fallacious statistical approach with regression. Sadly, it's often because the regression approach was historically more tractable and the fallacies weren't as well known. And so as certain people gained more senior positions and sought to retain political control of the business tools that they oversaw, they grasped for convenient fictions like "interpretability" to justify their political choice to shun modern techniques. [0] < http://www.columbia.edu/~gjw10/achen04.pdf http://www.columbia.edu/~gjw10/achen04.pdf >
- darkhorn 10y agoWhy you are not testing independence of errors? http://people.duke.edu/~rnau/testing.htm http://people.duke.edu/~rnau/testing.htm
- darkhorn 10y agoWhy you are not testing for constant variance? https://onlinecourses.science.psu.edu/stat501/node/367 https://onlinecourses.science.psu.edu/stat501/node/367
- apathy 10y agoBecause, to quote Box, ‘To make the preliminary test on variances is rather like putting to sea in a rowing boat to find out whether conditions are sufficiently calm for an ocean liner to leave port!’ (Box 1953) Plot the residuals instead.
- capkutay 10y agoAnecdotal thought, I find practical guides and tutorials like this far more useful for learning machine learning than any of the public MOOCs like Andrew Ng's machine learning class. There's a lot of dense content shown in video lectures and very little practice. I almost had to google every topic and find a write-up like this to drive each concept home.