3 ms·
That's true for mask-based training (used for embeddedings and BERT and such), but not true for modern autoregressive LLMs as a whole, which are pretrained with
by Philpax 2y ago
That's true for mask-based training (used for embeddedings and BERT and such), but not true for modern autoregressive LLMs as a whole, which are pretrained with next word prediction.
- astrange 2y agoIt's not strictly that though, it's next word prediction with regularization. And the reason LLMs are interesting is that they /fail/ to learn it, but in a good way. If it was a "next word predictor" it wouldn't answer questions but continue them. Also, it's a next token predictor not a word predictor - which is important because the "just a predictor" theory now can't explain how it can form words at all!
- Philpax 2y agoYes, I know; I was clarifying their immediate misunderstanding using the same terminology as them. There's obviously a lot more going on behind the scenes, especially with today's mid- and post-training work!