3 ms·
I used to agree with you. The paper that made me unsure was "Transformers learn in-context by gradient descent" [1]. Basically, the model learns weights that le
by sebzim4500 4y ago
I used to agree with you. The paper that made me unsure was "Transformers learn in-context by gradient descent" [1]. Basically, the model learns weights that let it run gradient descent at inference time in order to do in context learning. If transformers can learn this, then I think they can learn almost anything given enough compute/parameters/data.
Of course, even if this is true then it's possible that there simply isn't enough high quality data, or that the amount of compute required is beyond our current hardware.
[1] https://arxiv.org/abs/2212.07677 https://arxiv.org/abs/2212.07677
- civilized 4y agoToo theoretical to be pivotal. Years ago we had results about how all sorts of learning was reducible to kernel SVM and that drove a big fad for kernel SVM research at NeurIPS and other hot conferences. Nobody cares about SVM anymore. So we don't have a good enough grasp on the theory of AI to predict what is going to be a breakthrough. Only rigorous empirical results convince. Second, the idea of meta-learning gradient descent, or that some learning algorithms can do it, isn't new or specific to Transformer architectures.
- sebzim4500 4y agoI'm not suggesting that you can use meta-learning to discover gradient descent, that would not be surprising. My claim is that GPT-4 (probably even GPT-2) is almost certainly already using gradient descent internally inside it's own activations. How else can it learn a random small neural network given enough samples in the prompt?