We know that models like GPT-4 go through phase changes, not in train/test loss, but for loss on specific tasks. This suggests the gradient descent is learning

We know that models like GPT-4 go through phase changes, not in train/test loss, but for loss on specific tasks. This suggests the gradient descent is learning one thing after another, possibly in order of "difficulty".