3 ms·
at that point what it has is not a representation of the input, its a representation of what the next output could be. ie. its a lossy process and you can't ext
by vanjajaja1 3y ago
at that point what it has is not a representation of the input, its a representation of what the next output could be. ie. its a lossy process and you can't extract what came in the past, only the details relevant to next word prediction
(is my understanding)
- rollinDyno 3y agoIf the point was the presentation of only the next token, and predicted tokens were a function of only the preceding token, then the vector of the new token wouldn’t have the information to produce new tokens that kept telling the novel.