4 ms·
> ...and then repeat until the model says stop. How does this actually work? At a high level, the autoregressive token generation process makes sense to me, bu
by xanderlewis 3y ago
> ...and then repeat until the model says stop.
How does this actually work? At a high level, the autoregressive token generation process makes sense to me, but how does it know when a sensible time to stop is rather than just going on forever or abruptly stopping after n tokens? Is it trained on text that has special 'stop' tokens inserted at the end of paragraphs, etc. and when it chucks out one of these the model halts?
- epr 3y agoOne of the tokens represents stopping. If you sample stop from the probability distribution instead of a normal text token, then you stop autoregressive sampling.
- astrange 3y agoOpen source models have an explicit endoftext token, yes. You can set llama.cpp to keep generating after that and it often instantly goes offtopic.
- llm_trw 3y agoNot only off topic but out of sample. The old style LLM hallucinations where it starts spitting out random tokens like: nxrt$$.full #`,,, jump #43one daisy daisy give me your answer ###