6 ms·

> It’s worth noting that LLMs are non-deterministic, This is probably better phrased as "LLMs may not provide consistent answers due to changing data and built

• msgodel 1y ago
  I run my local LLMs with a seed of one. If I re-run my "ai" command (which starts a conversation with its parameters as a prompt) I get exactly the same output every single time.
  • xnx 1y ago
    Yes. This is what I was trying to say. Saying "It’s worth noting that LLMs are non-deterministic" is wrong and should be changed in the blog post.
    • boroboro4 1y ago
      You’re correct in batch size 1 (local is one), but not in production use case when multiple requests get batched together (and that’s how all the providers do this). With batching matrix shapes/request position in them aren’t deterministic and this leads to non deterministic results, regardless of sampling temperature/seed.
      • unsnap_biceps 1y ago
        Isn't that true only if the batches are different? If you run exactly the same batch, you're back to a deterministic result. If I had a black box api, just because you don't know how it's calculated doesn't mean that it's non-deterministic. It's the underlaying algorithm that determines that and a LLM is deterministic.
        • boroboro4 1y ago
          Providers never run same batches because they mix requests between different clients, otherwise GPUs are gonna be severely underutilized. It’s inherently non deterministic because it reflects the reality of having different requests coming to the servers at the same time. And I don’t believe there are any realistic workarounds if you want to keep costs reasonable. Edit: there might be workarounds if matmul algorithms will give stronger guarantees then they are today (invariance on rows/columns swap). Not an expert to say how feasible it is, especially in quantized scenario.
    • TheDong 1y ago
      > Saying "It’s worth noting that LLMs are non-deterministic" is wrong and should be changed in the blog post. Every person in this thread understood that Simon meant "Grok, ChatGPT, and other common LLM interfaces run with a temperature>0 by default, and thus non-deterministically produce different outputs for the same query". Sure, he wrote a shorter version of that, and because of that y'all can split hairs on the details ("yes it's correct for how most people interact with LLMs and for grok, but _technically_ it's not correct"). The point of English blog posts is not to be a long wall of logical prepositions, it's to convey ideas and information. The current wording seems fine to me. The point of what he was saying was to caution readers "you might not get this if you try to repro it", and that is 100% correct.
      • root_axis 1y ago
        Still, the statement that LLMs are non-deterministic is incorrect and could mislead some people who simply aren't familiar with how they work. Better phrasing would be something like "It's worth noting that LLM products are typically operated in a manner that produces non-deterministic output for the user"
        • Veen 1y ago
          Simon would be less engaging if he caveated every generalisation in that way. It’s one of the main reasons academic writing is often tedious to read.
        • antonvs 1y ago
          > It's worth noting that LLM products are typically operated in a manner that produces non-deterministic output for the user Or you could abbreviate this by saying “LLMs are non-deterministic.” Yes, it requires some shared context with the audience to interpret correctly, but so does every text.
      • msgodel 1y ago
        My temperature is set higher than zero as well. That doesn't make them nondeterministic.
    • DemocracyFTW2 1y ago
      "Non-deterministic" in the sense that a dice roll is when you don't know every parameter with ultimate precision. On one hand I find insistence on the wrongness on the phrase a bit too OCD, on the other I must agree that a very simple re-phrasing like "appears {non-deterministic|random|unpredictable} to an outside observer" would've maybe even added value even for less technically-inclined folks, so yeah.
  • lgessler 1y ago
    In my (poor) understanding, this can depend on hardware details. What are you running your models on? I haven't paid close attention to this with LLMs, but I've tried very hard to get non-deterministic behavior out of my training runs for other kinds of transformer models and was never able to on my 2080, 4090, or an A100. PyTorch docs have a note saying that in general it's impossible: https://docs.pytorch.org/docs/stable/notes/randomness.html https://docs.pytorch.org/docs/stable/notes/randomness.html Inference on a generic LLM may not be subject to these non-determinisms even on a GPU though, idk
    • msgodel 1y ago
      Ah. I've typically avoided CUDA except for a couple of really big jobs so I haven't noticed this.
• simonw 1y ago
  I don't think those race conditions are rare. None of the big hosted LLMs provide a temperature=0 plus fixed seed feature which they guarantee won't return different results, despite clear demand for that from developers.
  • xnx 1y ago
    Fair. I dislike "non-deterministic" as a blanket llm descriptor for all llms since it implies some type of magic or quantum effect.
    • basch 1y ago
      I agree its phrased poorly. Better said would be: LLM's are designed to act as if they were non-deterministic.
      • deleted 1y ago
        [deleted]
    • dekhn 1y ago
      I see LLM inference as sampling from a distribution. Multiple details go into that sampling - everything from parameters like temperature to numerical imprecision to batch mixing effects as well as the next-token-selection approach (always pick max, sample from the posterior distribution, etc). But ultimately, if it was truly important to get stable outputs, everything I listed above can be engineered (temp=0, very good numerical control, not batching, and always picking the max probability next token). dekhn from a decade ago cared a lot about stable outputs. dekhn today thinks sampling from a distribution is a far more practical approach for nearly all use cases. I could see it mattering when the false negative rate of a medical diagnostic exceeded a reasonable threshold.
    • tanewishly 1y ago
      Errr... that word implies some type of non-deterministic effect. Like using a randomizer without specifying the seed (ie. sampling from a distribution). I mean, stuff like NFAs (non-deterministic finite automata) isn't magic.
• kcb 1y ago
  FP multiplication is non-commutative.
  • boroboro4 1y ago
    It doesn’t mean it’s non-deterministic though. But it does when coupled with non-deterministic requests batching, which is the case.
  • DemocracyFTW2 1y ago
    That's like you can't deduce the input t from a cryptographic hash h but the same input always gives you the same hash, so t->h is deterministic. h->t is, in practice, not a way that you can or want to walk (because it's so expensive to do) and because there may be / must be collisions (given that a typical hash is much smaller than the typical inputs), so the inverse is not h->t with a single input but h->{t1,t2,...}, a practically open set of possible inputs that is still deterministic.
• troupo 1y ago
  > Barring rare(?) GPU race conditions, LLMs produce the same output given the same inputs. Are these LLMs in the room with us? Not a single LLM available as a SaaS is deterministic. As for other models: I've only run ollama locally, and it, too, provided different answers for the same question five minutes apart Edit/update: not a single LLM available as a SaaS's output is deterministic, especially when used from a UI. Pointing out that you could probably run a tightly controlled model in a tightly controlled environment to achieve deterministic output is very extremely irrelevant when describing output of grok in situations when the user has no control over it
  • fooker 1y ago
    > Not a single LLM available as a SaaS is deterministic. Lower the temperature parameter.
    • troupo 1y ago
      So, how does one do it outside of APIs in the context we're discussing? In the UI or when invoking @grok in X? How do we also turn off all the intermediate layers in between that we don't know about like "always rant about white genocide in South Africa" or "crash when user mentions David Meyer"?
      • marcinzm 1y ago
        Grok is not deterministic would then be the correct statement.
        • troupo 1y ago
          When used through UI, like the author does, Grok isn't. OpenAI isn't. Gemini isn't
    • pydry 1y ago
      It's not enough. Ive done this and still often gotten different results for the same question.
• TOMDM 1y ago
  I think the better statement is likely "LLMs are typically not executed in a deterministic manner", since you're right there are no non deterministic properties interment to the models themselves that I'm aware of
• llm_nerd 1y ago
  That non-deterministic claim, along with the rather ludicrous claim that this is all just some accidental self-awareness of the model or something (rather than Elon clearly and obviously sticking his fat fingers into the machine), make the linked piece technically dubious. A baked LLM is 100% deterministic. It is a straightforward set of matrix algebra with a perfectly deterministic output at a base state. There is no magic quantum mystery machine happening in the model. We add a randomization -- the seed or temperature -- to as a value-add randomize the outputs in the intention of giving creativity. So while it might be true that "in the customer-facing default state an LLM gives non-deterministic output", this is not some base truth about LLMs.
  • simonw 1y ago
    LLMs work using huge amounts of matrix multiplication. Floating point multiplication is non-associative: a = 0.1, b = 0.2, c = 0.3 a * (b * c) = 0.006 (a * b) * c = 0.006000000000000001 Almost all serious LLMs are deployed across multiple GPUs and have operations executed in batches for efficiency. As such, the order in which those multiplications are run depends on all sorts of factors. There are no guarantees of operation order, which means non-associative floating point operations play a role in the final result. This means that, in practice, most deployed LLMs are non-deterministic even with a fixed seed. That's why vendors don't offer seed parameters accompanied by a promise that it will result in deterministic results - because that's a promise they cannot keep. Here's an example: https://cookbook.openai.com/examples/reproducible_outputs_with_the_seed_parameter https://cookbook.openai.com/examples/reproducible_outputs_wi... > Developers can now specify seed parameter in the Chat Completion request to receive (mostly) consistent outputs. [...] There is a small chance that responses differ even when request parameters and system_fingerprint match, due to the inherent non-determinism of our models.
    • deleted 1y ago
      [deleted]
    • llm_nerd 1y ago
      >That's why vendors don't offer seed parameters accompanied by a promise that it will result in deterministic results - because that's a promise they cannot keep. They absolutely can keep such a promise, which anyone who has worked with LLMs could confirm. I can run a sequence of tokens through a large LLMs thousands of times and get identical results every time (and have done precisely this! In fact, in one situation it was a QA test I built). I could run it millions of times and get exactly the same final layer every single time. They don't want to keep such a promise because it limits flexibility and optimizations available when doing things at a very large scale. This is not an LLM thing, and saying "LLMs are non-deterministic" is simply wrong, even if you can find an LLM purveyor who decided to make choices where they no longer have any interest in such an outcome. And FWIW, non-associative floating point arithmetic is usually not the reason. It's like claiming that a chef cannot do something that McDonalds and Burger King don't do, using those purveyors as an example of what is possible when cooking. Nothing works like that.
• spindump8930 1y ago
  The many sources of stochastic/non-deterministic behavior have been mentioned in other replies but I wanted to point out this paper: https://arxiv.org/abs/2506.09501 https://arxiv.org/abs/2506.09501 which analyzes the issues around GPU non determinism (once sampling and batching related effects are removed). One important take-away is that these issues are more likely in longer generations so reasoning models can suffer more.