3 ms·
I want to point out that LLMs can be completely deterministic if the final sampler is run with 0 temperature (picking the highest probability token), no top-k,
by mk_stjames 1y ago
I want to point out that LLMs can be completely deterministic if the final sampler is run with 0 temperature (picking the highest probability token), no top-k, fixed seed, etc.
- yencabulator 1y agoHighest probability token can still vary nondeterministically when the computation is essentially racing GPU cores or even separate hosts against each other. Float math evaluation order can change the end result.