3 ms·
The model doesn't "know" how to generate tokens any more than it knows how to stop generating tokens. The sampler simply stops pulling values when it outputs a
by bulder 1mo ago
The model doesn't "know" how to generate tokens any more than it knows how to stop generating tokens. The sampler simply stops pulling values when it outputs a "stop token", which is a token the same way every other token is.
That is to say, it stops when it's statistically the most likely to.
- phatfish 1mo agoOK thanks, so the neural net (that no one can explain fully) generates a "stop" signal at a certain point.