As a counter-argument Linear regression to ML is "goto statement" to programming. Linear regression looks great on paper since you can derive residuals, slopes

As a counter-argument Linear regression to ML is "goto statement" to programming. Linear regression looks great on paper since you can derive residuals, slopes compare the individual "effects" etc. But thats unnecessary and in some cases wrong when the goal is mere prediction and not explanation. The big difference between ML and statistics is that latter selects a "correct" linear model and then assumes a distribution for "errors" due to pesky reality. The effects are used for explanations (538 Nate Silver style wonk/punditry). Machine Learning on other hand tries to predict as close to observations as possible without imposing a model or caring about an explanation. The simplest introductory Machine Learning approach should not be linear regression but rather a 1- nearest neighbor model. E.g. rather than giving data about house prices and square footage. The question should be "How do you predict price of a house in given location?". "What are relavant features?" (location,location, location,school district,number of rooms,sq ft, etc), "How would you collect labels/data". (Zillow, exclued prices older than 2-3 years). The simplest answer would be that the price is same as that of the neighboring house (closest lat/long) with similar sq foot sold recently. This can then be implemented as a weighted distance metric and tested using Leave one out cross validation (I know not the best metric). But consider how Nearest Neighbors allows us to incorporate location information in a natural manner. That is very important and cannot incorporated in an elegant manner in a linear regression model. A big part of ML is applying different set of methods across several domains. Thus for beginners Teaching ML should not be about teaching limear models or gradient descent but rather how do you start thinking from ML perspective.