> But to those who do, all of this followed a pretty straightforward path, even though certain key steps were only obvious in hindsight. The research still lar

> But to those who do, all of this followed a pretty straightforward path, even though certain key steps were only obvious in hindsight. The research still largely relies on post-hoc justification for these architectural benefits. We know CNNs work, we can open them up and see what they're doing, but we didn't get there from a theoretical foundation that predicted this outcome, nor do we have a real theoretical framework to justify them. The history of pre-science is filled with post-hoc justification that is very similar, allows for practitioners to make progress, but ultimately has turned out to be wildly incorrect. > it is actually pretty obvious. In this entire reply you leave out the theoretical justifications to back up this claim. You show many example of intuitively why these architectures work, but never dive into the rigorous explanation, because such explanations don't exist yet. This comment simply outlines the growing "bag of tricks" we've built up over the years to solve problems, along with the common post-hoc justifications. But at it's current state this is not different than alchemy, which did get some ideas correct, was able to create some useful practices but ultimately failed to provide a theoretical frame work for what was being done. I don't know any serious deep learning researcher who disagrees that at this point the practice far out paces our theoretical understanding.