3 ms·

Can you summarize and explain those things (why deep learning works, why over-parameterization is useful, and why deeper is better than shallower) assuming an u

• stared 7y ago
  Did you try reading these papers? If you want to start from the simpler one, go for the Uber one first. Though, in my understanding, the original paper should be accessible to math/cs undergrads.
• csense 7y ago
  Neural networks are basically graphs with weighted edges. As you increase number of nodes N in a graph, the number of edges grows at most quadratically, but the number of possible subgraphs grows exponentially. So even if you use totally random edge weights, with high probability some of those O(2^N) subgraphs will be pretty good. OP paper is basically providing formal mathematical definitions for these loose notions, then chasing down a bunch of epsilons and deltas to prove that, in fact, good subgraphs actually do always exist (with high probability for large enough N). From a practical standpoint, the main citation's is pretty informative: - "What's Hidden in a Randomly Weighted Neural Network?" by Ramanujan et al, https://arxiv.org/abs/1911.13299 https://arxiv.org/abs/1911.13299 This paper actually gives an algorithm which empirically does a decent job at finding a "good" subgraph.