5 ms·
So instead of next token prediction its next event prediction. At some point this just loops around and we're back to teaching models to predict the next token
by valine 1y ago
So instead of next token prediction its next event prediction. At some point this just loops around and we're back to teaching models to predict the next token in the sequence.
- lumost 1y agoTokens are an awfully convenient way to describe an event.
- phyalow 1y agoTokens are just discretized state representations.
- ww520 1y agoIt’s the next state. So instead of spitting out words, it will spit out a whole movie, or a sequence of world states in a game or simulation.
- deleted 1y ago[deleted]