3 ms·
> What will next-token prediction give us on such a dataset? Haven’t we pretty much figured out these things are doing more than just predicting the next token
by UncleEntity 4y ago
> What will next-token prediction give us on such a dataset?
Haven’t we pretty much figured out these things are doing more than just predicting the next token at this point?
There’s probably a lot to be done with a “prediction machine”, birds aren’t all that smart but can catch bugs in midair.
- skybrian 4y agoPeople keep underestimating what next-token prediction can do, but they’re not wrong that it’s how LLM’s work. It’s actually a good question: what will next-token prediction be able to do on new datasets? The error is thinking you can answer it, even in broad terms.