Over fitting can, indeed, be "fitting the data too perfectly"; e.g., Lagrangian interpolation where have some X-Y pairs and want a polynomial in x that fits the

Over fitting can, indeed, be "fitting the data too perfectly"; e.g., Lagrangian interpolation where have some X-Y pairs and want a polynomial in x that fits the X-Y pairs. How does that work? You can derive it for yourself: For each of the X-Y pairs, write a polynomial that is zero at all the X points except the one pair you want to fit perfectly and then add all those polynomials. Done. Presto. Bingo. "Look, Ma, fits anything perfectly!" (as long as all the X points are distinct). "I should be able to find a polynomial that fits points from a square wave!" Yup, but graph the polynomial and see what it does away from one of the given X points -- the graph goes wild, that is, shooting off to plus or minus infinity and appears nearly to get there. But in linear fitting, over fitting is simpler: There, of course, do get a linear function which doesn't go wild. And the function can fit the data nicely. The usual problem is that the coefficients in the linear function are not unique.