4 ms·
How does this actually work though, since the model could e.g. abruptly end the generation giving something syntactically invalid? Doesn't it need to look at th
by amilios 3y ago
How does this actually work though, since the model could e.g. abruptly end the generation giving something syntactically invalid? Doesn't it need to look at the whole output at some stage?
- brucethemoose2 3y agoAll generated output depends on all the previous output. The model "looks" at (mostly) everything every time. Generation never actually stops, the model just emits a special stop token when stopping is most likely next token. Hence the grammar implementation can prevent this stop token from being emitted prematurely. There was some discussion of models getting "stuck" where there is no syntactically correct token to emit. Some proposals included a "backspace token" IIRC, but I dunno what they actually did. You can look through the discussion in the PR.
- amilios 3y agoOh yeah that's true! Just block the stop token. But yes, my thought is that there are scenarios where it can get "stuck" as you said. I'll look at the PR, thanks!
- uoaei 3y agoAt every generation step you mask the tokens so that only syntactically valid outputs are possible. This includes the <end> token.