I found the opening chapters of "Computational Learning Theory" by Kearns & Vazirani [1] really eye-opening in this regard. It starts using the example of "rec

I found the opening chapters of "Computational Learning Theory" by Kearns & Vazirani [1] really eye-opening in this regard. It starts using the example of "rectangle" learning, where data points are encoded as points in a 2-dimensional space, just like many other learning algorithms start by encoding things as points in N-dimensional space. But then, instead of launching straight into decision trees or k-means or Gaussians or anything like that, the book does something strange: It uses "rectangle" as a machine learning model. The idea is that some points are marked with one label (red) others with another (blue). You want to fit a model to distinguish red from blue. So you just take the min/max of all the x/y corrdinates that you've seen in your training data to get a lower-left and upper-right point defining a rectangle and predict that all points inside the rectangle are red. Then the book continues with all the usual stuff like figuring out error surfaces, computing precision/recall and so forth and then introduces concepts like big-O analysis of data complexity, overfitting problems by having a sense for the kolmogorov complexity in the model space versus amount of data, etc. etc. This really blew my mind, that, at the point where the "magic" was supposed to come in, introducing neural networks or whatever, they just skip over that, sort of saying: That's not important yet, let's just use a simple thing like 'rectangle' as a placeholder for now as it will serve our purpose just as well. So: Any method for selecting a model (e.g. particular rectangle) from a meta-model (e.g. set of all rectangles) can be a valid machine learning method and there really is no "magic" to neural networks. What really matters is whether the model is well-suited to the phenomenon being modelled, given the way the data is encoded. So it makes a lot of sense to obsess over preprocessing of data and to come up with custom models to fit custom modelling needs. It makes very little sense to chase all the newest and greatest breakthroughs in [machine learning flavour of the week]. This has been pretty much how I have been doing machine learning over the course of 10 years, working as a Data Scientist. My advice to people starting out in this field: By walking down this path, you will have predictably successful project outcomes while at the same time flushing your career down the tube. ...your ignorant bosses will never see you as the smart guy in the room, if everyone else is talking about neural networks and you are talking about rectangles. [1] https://a.co/d/5ZEyQKg https://a.co/d/5ZEyQKg