3 ms·
Geoffrey Hinton (now a Nobel Prize winner!) himself did a summary. I think it is the single best summary on this topic. Our labeled datasets were thousands o
by sanxiyn 2y ago
Geoffrey Hinton (now a Nobel Prize winner!) himself did a summary. I think it is the single best summary on this topic.
Our labeled datasets were thousands of times too small.
Our computers were millions of times too slow.
We initialized the weights in a stupid way.
We used the wrong type of non-linearity.
- imjonse 2y agoThat is a pithier formulation of the widely accepted summary of "more data + more compute + algo improvements"
- sanxiyn 2y agoNo, it isn't. It emphasizes importance of Glorot initialization and ReLU.
- helltone 2y agoI'm curious and it's not obvious to me: what changed in terms of weight initialisation?
- HarHarVeryFunny 2y agoAlso: nets too small (not enough layers) gradients not flowing (residual connections) layer outputs not normalized training algorithms and procedures not optimal (Adam, warm-up, etc)