This is a nice interactive example, but it doesn't explain _why_ one would prefer to use a least-squares estimator over any other arbitrary method. For example,

This is a nice interactive example, but it doesn't explain _why_ one would prefer to use a least-squares estimator over any other arbitrary method. For example, one could measure the error in any other L_p norm - e.g. L_1 or L_{\infty} instead of L_2. What's so special about this one? To answer that, we have the Gauss-Markov theorem [1] -- provided certain assumptions regarding the data are met, least squares is the best linear unbiased estimator. Note, however, that no-one is forcing us to choose an estimator that is either linear or unbiased. What if by dropping one of those two somewhat arbitrary constraints we could get a better estimator? One example of this is the James-Stein estimator [2], which is a biased estimator that has the amusing property of dominating (i.e., always performing no worse than, and performing strictly better than in some case) the bog standard least-squares estimate when one is dealing with data in three or more dimensions (and a few more common assumptions). Crudely speaking, the James-Stein estimator produces estimates that are biased toward the origin, when compared to estimates produced from ordinary least squares. It is also possible to define a James-Stein estimator that biases toward an arbitrary fixed non-origin vector -- these variants also dominate the bog-standard ordinary least squares estimator! Personally I find this result both intriguing and also reasonably offensive (which means my mathematical intuition needs adjustment). [1] - https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem https://en.wikipedia.org/wiki/Gauss%E2%80%93Markov_theorem [2] - https://en.wikipedia.org/wiki/James%E2%80%93Stein_estimator https://en.wikipedia.org/wiki/James%E2%80%93Stein_estimator