3 ms·
Consistency has to be a tough problem to solve for a service like this, since randomness in the choice of each token is part of the magic sauce that makes LLMs
by dhc02 4y ago
Consistency has to be a tough problem to solve for a service like this, since randomness in the choice of each token is part of the magic sauce that makes LLMs work.
- rushingcreek 4y agoIt is definitely a hard problem. There are ways to ensure consistency, such as using beam search decoding, which is deterministic. But that comes with other tradeoffs regarding answer quality.