I kind of agree with the author's major sentiment: that ML research is stuck in a rut with incremental improvement. However, the longer the article goes on, the

I kind of agree with the author's major sentiment: that ML research is stuck in a rut with incremental improvement. However, the longer the article goes on, the less and less I agree with any of their statements. They start of criticizing the incremental improvers. They advocate later that if "stack more layers" beats a method, the method isn't good while completely ignoring anything other than the standard SOTA metrics like data efficiency, computational efficiency, model representation efficiency, etc. And, while I agree there is a "fake rigor" problem in ML research, the particular examples that they bring up aren't extremely good exemplars in my opinion. Instead, they seem to have a problem with the standard operating procedure of mathematics while missing the point that it is what got the "stack more layers" school here in the first place. Simplifying a problem so you can understand it, and then relaxing the assumptions and seeing if you can figure out what implications that has is how advances are made. Plus, they have some hot takes and statements that are just plain wrong. > With Automatic Differentiation, the backward pass is essentially free and is as engaging to compute as 50 digit number long division. Deriving long complicated gradients is fake rigor that was useful before we had computers. What? First of all, AD is not a solved problem and using it is not "essentially free." There's a huge performance overhead when adding AD to a system. Try using a second order method with AD. I hope your Hessian actually finished computing. > Julia on the other hand is a language made for scientific computing where everything is automatically differentiable by default. This is just plain false. I'm a huge proponent of Julia, and there are some great AD packages, but in no way is everything automatically differentiable (even with the nice packages), nor is that a design goal. The work on Flux.jl (a package) is extremely impressive though, and there are particular features of Julia that allow some awesome package interoperability (e.g. the fact that some ODE solvers can be differentiated through with Flux).