The pre-training phase of GPT for next work prediction is essentially that - dump all text you can find and let the model figure out the relationship between to

The pre-training phase of GPT for next work prediction is essentially that - dump all text you can find and let the model figure out the relationship between tokens/words. I agree though - what’s missing in neural architectures is advanced clustering capabilities purely from observing without any labels. Then the labeling phase is cheap and doesn’t require as much data.