> all of the above NFL says something about it being a wash for arbitrary data. All results are going to be tuned to assumptions we have about our data in part

> all of the above NFL says something about it being a wash for arbitrary data. All results are going to be tuned to assumptions we have about our data in particular (not too many discontinuities, sufficiently well sampled, ...). > neurons in a layer, how many layers, ... Scaling laws are, currently, empirically derived. From those you can pick your goals (e.g., at most $X and maximize accuracy) and work backward to one or more optimal sets of parameters. Except in very restricted domains or with other strong assumptions I haven't seen anything giving you more than that. > activation functions All of the above about how it can't matter for arbitrary data and how parameters need to be empirically derived apply. However: An important inductive bias a lot of practitioners use is that every weight in the model should be roughly equally important. There are other ways you choose activation functions, especially in specialized domains, but when designing a deep network one of the most important things you can do is control the magnitude of information at each level of backpropagation. If your activation function (and surrounding infrastructure) approximately handles that problem then it's probably good enough. > neural network or other techniques For almost every problem you're better off using something other than a neural network (like catboost). I don't have any good intuition for why that's the case. Test them both. That's what the validation dataset is for. > how it connects to the practice For this article in particular, it doesn't connect to a ton of what I personally do. I'm sure it resonates with someone. As soon as pytorch or jax or whatever isn't good enough though and you have to go implement stuff from scratch, you need a deep dive in the theory you're implementing. To a lesser degree, if you're interfacing with big frameworks nontrivially or working around their limitations, you still need a deep understanding of the things you're implementing. Imagine, e.g., that you want all the modern ML tools in a world where dynamic allocation, virtual functions, and all that garbage aren't tractable. You can resoundedly beat every human heuristic for phantom touchpad events in your mouse driver with a tiny neural network, but you can't use pytorch to do it without turning your laptop into a space heater. Embedded devices aren't the only scenario where you might have to venture off the beaten path. Much like the age-old argument of importing a data structure vs writing your own, as soon as you have requirements beyond what the library author provides it's often worth it to do the whole thing on your own, and it takes a firm theoretical foundation to do so swiftly and correctly. > how it connects to practice That's a criticism I have of a lot of educational materials. Connecting the dots is important in writing (competing with all the advantages of brevity). Pick on the Model-Based Learning section as an example. We're asked, to start, to MLE a gaussian. (M)aximum (L)ikelihood (E)stimation is an extremely important concept, and a lot of ML practitioners throw it to the side. Imagine, e.g., a 2-stage process where for each price bracket you have a model reporting the likelihood of conversion and then a second stage where you synthesize those predictions into an optimal strategy. Common failure modes include (a) mishandling variance, (b) assuming that MLE on each of the models allows you to combine the mean/mode/... results into an MLE composite action, (c) really an extension of [b], but if you have the wrong loss function for your model(s) then they aren't meaningfully combinable, .... Something that should be obvious (predict conversion rates, combine those rates to determine what you should do) has tons of pitfalls if you don't holistically reason about the composite process. That's perhaps a failure in the primitives we use to construct those composite processes, but in today's day and age it's still something you have to consider. How does the book connect? I dunno. It looks more like a "kata" (keep your fundamental skills sharp) than anything else. An explicit connection to some real-world problem might make it more tractable.