2 ms·
> Can't they run an example prompt and verify they get the exact same output token probabilities for all prompts? You don’t even get that with GPUs in general,
by nyrikki 7mo ago
> Can't they run an example prompt and verify they get the exact same output token probabilities for all prompts?
You don’t even get that with GPUs in general, or really floating point in general.
The Art of Computer Programming. Volume 2: Seminumerical Algorithms section 4.2.2 with explain where it loses floating addition associativity property.
Apartness relations are another possible lens.
- ntonozzi 7mo agoYeah you can: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ https://thinkingmachines.ai/blog/defeating-nondeterminism-in....
- nyrikki 7mo ago> However, as the name “batch-invariant” suggests, the technique is currently limited to handling variations related only to the batch dimension, making it robust to continuous batching and other batch-size–related changes, but not to other forms of nondeterminism like changing the TP sizes or GPU types. https://arxiv.org/abs/2506.09501 https://arxiv.org/abs/2506.09501