4 ms·
If you use a deterministic sampling strategy for the next token (e.g., always output the token with the highest probability) then a traditional LLM should be de
by tibbar 1y ago
If you use a deterministic sampling strategy for the next token (e.g., always output the token with the highest probability) then a traditional LLM should be deterministic on the same hardware/software stack.
- roywiggins 1y agoDeterministic is one thing, but stable to small perturbations in the input is another.
- dragonwriter 1y ago> Deterministic is one thing, but stable to small perturbations in the input is another. Yes, and the one thing that was asked about was "deterministic" not "stable to small perturbations in the input.
- extraduder_ire 1y agoWouldn't seeding the RNG used to pick the next token be more configurable? How would changing the hardware/other software make a difference to what comes out of the model?
- tibbar 1y ago> Wouldn't seeding the RNG used to pick the next token be more configurable? Sure, that would work. > How would changing the hardware/other software make a difference to what comes out of the model? Floating point arithmetic is not entirely consistent between different GPUs/TPUs/operating systems.