3 ms·
The model outputs a probability distribution for the next token, given the sequence of all previous tokens in the context window. It’s just a list of floats in
by anon373839 6mo ago
The model outputs a probability distribution for the next token, given the sequence of all previous tokens in the context window. It’s just a list of floats in the same order as the list of tokens that the tokenizer uses.
After that, a piece of software that is NOT the LLM chooses the next token. This is called the sampler. There are different sampling parameters and strategies available, but if you want repeatable* outputs, just take the token with the highest probability number.
* Perfect determinism in this sense is difficult to achieve because GPU calculations naturally have a minor bit of nondeterminism. But you can get very close.
- 2ndorderthought 6mo agoI'm not so sold the LLM is an LLM without a sampler but it's not worth quibbling over. It's part of the statistical model anyways.
- vrighter 6mo agothe llm is the trained part, the rest is the handwritten part. The sampler is handwritten, not learned.
- 2ndorderthought 6mo agoBelieve it or not in statistics and machine learning the hard coded parts of a model that impact the results are considered part of the model. But I understand that now days we don't care about these things because ai goes brrr.