3 ms·
> He seems to have thought someone was going to discover a revolutionary “key” to AI. That didn’t happen; you might say I was right to be skeptical of it. Tran
by Jensson 3y ago
> He seems to have thought someone was going to discover a revolutionary “key” to AI. That didn’t happen; you might say I was right to be skeptical of it.
Transformer architecture happened, that is key for neural networks to build up context dependent reasoning, we wouldn't have gotten to where we are today without it.
Without it scaling neural networks doesn't add much value, since without being able to separate out contexts they fail to develop a large amount of separate skills within the same model.
So yes, a very important key was found, it wasn't just scaling that got us here.
- treprinum 3y agoTransformers are synonymous with scaling as it was the first model that kept improving both by adding more and more layers and throwing more training data at it. All the previous models like CNNs, ResNet, LSTMs etc. hit some depth/training size where no progress was happening anymore as gradient was unreliably flowing down.