3 ms·

The single most important fact about linear regression is the Fritsch-Waugh-Lovell theorem. It's the whole reason why we're not scared as shit of using linear m

The single most important fact about linear regression is the Fritsch-Waugh-Lovell theorem. It's the whole reason why we're not scared as shit of using linear models. https://en.wikipedia.org/wiki/Frisch%E2%80%93Waugh%E2%80%93Lovell_theorem https://en.wikipedia.org/wiki/Frisch%E2%80%93Waugh%E2%80%93L... Basically, it means that omitting variables is ok (subject to terms and conditions). Assuming that IQ and shoe size are uncorrelated, but that income is a function of IQ and shoe size, I can run a regression: income = a + b * IQ + residuals and trust my estimate of b. FWL theorem says that running that regression and then residuals = c + d * shoe size is the same as running income = (a+c) + bIQ + dshoe size + eps It's an amazing result and it's all based on linear algebra, no belief in statistics required.

apathy 10y ago
Eh, orthogonalizing variables is a necessary precondition for this. When was the last time variables were totally uncorrelated and you didn't know it beforehand? Unfortunately such cases are the exception rather than the rule, and you lose interpretability by projecting into a low-rank orthogonal subspace. Not that interpretability is necessarily the be-all and end-all, but the "terms and conditions" are pretty important here!
- thanatropism 10y ago
  > When was the last time variables were totally uncorrelated and you didn't know it beforehand Whenever we have an actual theory of the underlying phenomena. It's common in econometrics to impose orthogonality conditions on the theoretical equations so structural parameters can be recovered; or so we can filter partial correlations. For example: a regression of prices and quantities sold will mix the effect of shifts in demand and in supply. Solution: find something that correlated with demand quantities but not price directly (for example, changes in consumers' wealth). A realistic, much cooler example: http://homepage.univie.ac.at/robert.kunst/pan2011_pres_rabas.pdf http://homepage.univie.ac.at/robert.kunst/pan2011_pres_rabas...
  - smallnamespace 10y ago
    > no belief in statistics required Yes, but now you have a statistical assumption: that your structural model holds. To make use of the Fritsch-Waugh-Lovell theorem, you must test that your orthogonality assumption holds using statistics :)

apathy 10y ago

Eh, orthogonalizing variables is a necessary precondition for this. When was the last time variables were totally uncorrelated and you didn't know it beforehand? Unfortunately such cases are the exception rather than the rule, and you lose interpretability by projecting into a low-rank orthogonal subspace. Not that interpretability is necessarily the be-all and end-all, but the "terms and conditions" are pretty important here!

thanatropism 10y ago
> When was the last time variables were totally uncorrelated and you didn't know it beforehand Whenever we have an actual theory of the underlying phenomena. It's common in econometrics to impose orthogonality conditions on the theoretical equations so structural parameters can be recovered; or so we can filter partial correlations. For example: a regression of prices and quantities sold will mix the effect of shifts in demand and in supply. Solution: find something that correlated with demand quantities but not price directly (for example, changes in consumers' wealth). A realistic, much cooler example: http://homepage.univie.ac.at/robert.kunst/pan2011_pres_rabas.pdf http://homepage.univie.ac.at/robert.kunst/pan2011_pres_rabas...
- smallnamespace 10y ago
  > no belief in statistics required Yes, but now you have a statistical assumption: that your structural model holds. To make use of the Fritsch-Waugh-Lovell theorem, you must test that your orthogonality assumption holds using statistics :)