This title is somewhat misleading, it’s more that you can use semi supervised learning to improve your fine tuning on a small model that’s already been pretrain

This title is somewhat misleading, it’s more that you can use semi supervised learning to improve your fine tuning on a small model that’s already been pretrained. I think it’s impressive that this technique beats gpt-3 in any setting with such a small model, but for example, what stops you from applying this to gpt-3 itself and getting an equivalent gain? Saying that it exceeds gpt-3 performance sounds like it does better during pretraining, which would be extremely impressive.