2 ms·
I really believe phase changes in models are going to tell us a lot about what they’ve learned. Though it has to be said that seeing that sharp improvement in a
by wwarner 3y ago
I really believe phase changes in models are going to tell us a lot about what they’ve learned. Though it has to be said that seeing that sharp improvement in a model during training is the exception rather than the rule — normally you just see gradual improvement until you’ve reached sufficient accuracy and then decide to stop training. So it’s not clear to me that every highly accurate model will contain a phase change.
- iNic 3y agoWe know that models like GPT-4 go through phase changes, not in train/test loss, but for loss on specific tasks. This suggests the gradient descent is learning one thing after another, possibly in order of "difficulty".