3 ms·

I don’t understand why people are calling these transformers models non-deterministic? Are you referring to the temperature parameter? I haven’t played with tra

by gnunez 2mo ago

I don’t understand why people are calling these transformers models non-deterministic? Are you referring to the temperature parameter? I haven’t played with transformer internals in a while but my understanding is that if the temperature is fixed at a value where the top logit is always picked, then because they weights are fixed, the exact same input should produce the exact same output. Am I missing something?

LeBit 2mo ago
You could answer your own question really, really quickly.
- gnunez 2mo ago
  I could, but then I would miss the chance to interact with such lovely people as yourself.
gpm 2mo ago
Well, yes and no. By non-deterministic I think people really mean "chaotic" in the chaos theory sense. Small perturbations in the input lead to wild and unpredictable changes in the output. Even with temperature parameters a fixed PRNG seed could mean an LLM was just chaotic and not technically non-deterministic. But more literally while LLMs are in theory deterministic (though perhaps not inference providers implementations if there's anything like a race condition affecting how things are rounded when added together) - we use the LLMs in harnesses that aren't. There are very likely races in the terminal outputs, dates both intentionally put in the context and accidentally leaked to the context, things like that.
- gnunez 2mo ago
  Ok. I see. I guess people are not referring to the raw models themselves when they say non-deterministic, but are also including the harness used in conjunction with the model. Then, in that case, for the exact same input you could get a non-deterministic output. But the model itself and all the mathematical machinery around the model is still very much deterministic. I guess if we really needed to, we could construct a deterministic agent harness. But in most use cases we probably want some chaotic behavior to increase our chances of stumbling on the desired results. Thank you for the clarification
nl 2mo ago
> if the temperature is fixed at a value where the top logit is always picked, then because they weights are fixed, the exact same input should produce the exact same output. Am I missing something? Yes. Your input is part of a batch, and you don't know where in the batch it is. By default batches are not invariant and VLLM only supports invariance at all on some Huwaei Ascend hardware. See https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/batch_invariance.html https://docs.vllm.ai/projects/ascend/en/latest/user_guide/fe...
- gnunez 2mo ago
  I totally missed the memo on batching. That changes everything. Thank you for the info.
sejje 2mo ago
I think people are wrapping that across the English language. In English, these two tasks are exactly the same: "Would you hand me that item?" "Please hand me that item" But when posed to the LLM, they generate different outputs. One character difference in the prompt might be a whole different output. People who aren't programmers mostly don't know that there's any difference. They asked for the same thing, it knows what they want in both cases...but different results.
- esikich 2mo ago
  I'm not sure that's true. Sure, in the end I might hand them the item, but my thoughts about what they said will be different. I think you have to consider my thoughts "output" for this comparison to be valid.
- sebastiennight 2mo ago
  Not to be too pedantic, but these requests would not be exactly the same. There is a bit of indexicality in "Would you hand me that item ?" that might cause it to be interpreted as an actual question rather than a request, and might elicit different responses: - maybe _I_ would not hand this to you (I'm busy right now), but the person next to me whose hands are free would, so I'd nod to them. However, if you had said "Please hand me that item" I'd put down what I was doing to comply. - maybe I would not hand _this_ to you (it's not the right tool IMO), but I'd suggest another option. However, if you had said "Please hand me that item" I'd put my doubts aside to comply. - maybe I would not hand this to _you_ (you're not the one who should be handling it), but I'd do the thing myself or hand it to a more qualified member of the group. However, if you had said "Please hand me that item" I'd trust you enough to comply. I think this distinction is relevant in that I've found people to sometimes have difficulties understanding how similar LLM prompting is to giving instructions to human colleagues. I've had a collaborator who though very highly of his own prompting skills (while his prompts were very ambiguous and of the "make no mistakes, erase everything & correct yourself if you find one" variety) and blamed the models for not being "smart enough", and it was very noticeable that his management style for the juniors on his team was similarly unproductive.
- MarceColl 2mo ago
fimi 2mo ago
https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
neosat 2mo ago
As the above two comments mentioned this is not true in practice due to batch effects (you can read about some interesting work published by Thinking Machines on this), as well as calculation drift that happens across computations esp. now with inference optimization becoming common.

by gnunez 2mo ago

LeBit 2mo ago

You could answer your own question really, really quickly.

gnunez 2mo ago
I could, but then I would miss the chance to interact with such lovely people as yourself.

gpm 2mo ago

Well, yes and no. By non-deterministic I think people really mean "chaotic" in the chaos theory sense. Small perturbations in the input lead to wild and unpredictable changes in the output. Even with temperature parameters a fixed PRNG seed could mean an LLM was just chaotic and not technically non-deterministic. But more literally while LLMs are in theory deterministic (though perhaps not inference providers implementations if there's anything like a race condition affecting how things are rounded when added together) - we use the LLMs in harnesses that aren't. There are very likely races in the terminal outputs, dates both intentionally put in the context and accidentally leaked to the context, things like that.

gnunez 2mo ago
Ok. I see. I guess people are not referring to the raw models themselves when they say non-deterministic, but are also including the harness used in conjunction with the model. Then, in that case, for the exact same input you could get a non-deterministic output. But the model itself and all the mathematical machinery around the model is still very much deterministic. I guess if we really needed to, we could construct a deterministic agent harness. But in most use cases we probably want some chaotic behavior to increase our chances of stumbling on the desired results. Thank you for the clarification

nl 2mo ago

> if the temperature is fixed at a value where the top logit is always picked, then because they weights are fixed, the exact same input should produce the exact same output. Am I missing something? Yes. Your input is part of a batch, and you don't know where in the batch it is. By default batches are not invariant and VLLM only supports invariance at all on some Huwaei Ascend hardware. See https://docs.vllm.ai/projects/ascend/en/latest/user_guide/feature_guide/batch_invariance.html https://docs.vllm.ai/projects/ascend/en/latest/user_guide/fe...

gnunez 2mo ago
I totally missed the memo on batching. That changes everything. Thank you for the info.

sejje 2mo ago

I think people are wrapping that across the English language. In English, these two tasks are exactly the same: "Would you hand me that item?" "Please hand me that item" But when posed to the LLM, they generate different outputs. One character difference in the prompt might be a whole different output. People who aren't programmers mostly don't know that there's any difference. They asked for the same thing, it knows what they want in both cases...but different results.

esikich 2mo ago
I'm not sure that's true. Sure, in the end I might hand them the item, but my thoughts about what they said will be different. I think you have to consider my thoughts "output" for this comparison to be valid.
sebastiennight 2mo ago
Not to be too pedantic, but these requests would not be exactly the same. There is a bit of indexicality in "Would you hand me that item ?" that might cause it to be interpreted as an actual question rather than a request, and might elicit different responses: - maybe _I_ would not hand this to you (I'm busy right now), but the person next to me whose hands are free would, so I'd nod to them. However, if you had said "Please hand me that item" I'd put down what I was doing to comply. - maybe I would not hand _this_ to you (it's not the right tool IMO), but I'd suggest another option. However, if you had said "Please hand me that item" I'd put my doubts aside to comply. - maybe I would not hand this to _you_ (you're not the one who should be handling it), but I'd do the thing myself or hand it to a more qualified member of the group. However, if you had said "Please hand me that item" I'd trust you enough to comply. I think this distinction is relevant in that I've found people to sometimes have difficulties understanding how similar LLM prompting is to giving instructions to human colleagues. I've had a collaborator who though very highly of his own prompting skills (while his prompts were very ambiguous and of the "make no mistakes, erase everything & correct yourself if you find one" variety) and blamed the models for not being "smart enough", and it was very noticeable that his management style for the juniors on his team was similarly unproductive.
MarceColl 2mo ago

fimi 2mo ago

https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/ https://thinkingmachines.ai/blog/defeating-nondeterminism-in...

neosat 2mo ago

As the above two comments mentioned this is not true in practice due to batch effects (you can read about some interesting work published by Thinking Machines on this), as well as calculation drift that happens across computations esp. now with inference optimization becoming common.