I'm not suggesting that you can use meta-learning to discover gradient descent, that would not be surprising. My claim is that GPT-4 (probably even GPT-2) is al

I'm not suggesting that you can use meta-learning to discover gradient descent, that would not be surprising. My claim is that GPT-4 (probably even GPT-2) is almost certainly already using gradient descent internally inside it's own activations. How else can it learn a random small neural network given enough samples in the prompt?