4 ms·
This is neat, but one important point is that networks NEED to be significantly over parameterized for SGD like methods to find the minima. I know that's slight
by andrewnc 7y ago
This is neat, but one important point is that networks NEED to be significantly over parameterized for SGD like methods to find the minima. I know that's slightly tangential.
Another interesting point is that NNs can be thought about as preforming coordinate transformations on the data manifold, which means these sub nets are potentially approximations to those transformations, potentially up to some scaling factor.
I'm excited to see where this goes
- sjg007 7y ago>networks NEED to be significantly over parameterized for SGD like methods to find the minima Why?
- deleted 7y ago[deleted]
- deleted 7y ago[deleted]
- bigred100 7y agoI’ve only seen a small amount on this but the theoretical analysis I’ve seen used this as an assumption There may be some in here https://youtu.be/zZbHVaw_W9A https://youtu.be/zZbHVaw_W9A