3 ms·
Or know when to shut up. A tangent, but can anyone ELI5 how models "know" when to stop generating tokens? Or what the method to stop them at the right point is
by phatfish 2mo ago
Or know when to shut up.
A tangent, but can anyone ELI5 how models "know" when to stop generating tokens? Or what the method to stop them at the right point is?
- bigfudge 2mo agoI might be out of date but my understanding was that STOP was just another token that gets predicted.
- bulder 2mo agoThe model doesn't "know" how to generate tokens any more than it knows how to stop generating tokens. The sampler simply stops pulling values when it outputs a "stop token", which is a token the same way every other token is. That is to say, it stops when it's statistically the most likely to.
- phatfish 2mo agoOK thanks, so the neural net (that no one can explain fully) generates a "stop" signal at a certain point.