I find that an often under-appreciated perspective around big networks is the information-theoretical one, or at the risk of oversimplifying: the compression on

I find that an often under-appreciated perspective around big networks is the information-theoretical one, or at the risk of oversimplifying: the compression one. (The deep relationships between learning theory and information theory are of course well-known to researchers and sophisticated practitioners, but IMHO don’t get enough airtime). When you think about training a (not terribly overfit) model as a crop in the Fourier domain, a lot of these results are fairly natural to intuit about, which of course takes nothing away from how cool they are! :)