3 ms·
Yes, true. It's not clear that next word prediction is the best model for such an exercise, but yes, they do seem rather successful.
by Myrmornis 2y ago
Yes, true. It's not clear that next word prediction is the best model for such an exercise, but yes, they do seem rather successful.
- pests 2y agoEveryone always calls it "next word prediction" but that's also a simplification. If you go back to the original Transformer paper, the goal was to translate a document from one language to another. In all prompt systems the model is only given "past" tokens (as it's generating new tokens in real time) but in that original paper the LLM can use backwards and forward context to determine the translation. Just saying the architecture of how a model is trained and how it outputs tokens is less limited than you think.