7 ms·

Defeating Nondeterminism in LLM Inference

345 points by jxmorris12 1y ago

measurablefunc 1y ago
I think this means that the results might also be non-deterministic across hardware revisions b/c I don't think they verified that the kernels will work the same on different GPU & TPU versions b/c how do they know that the compiler will not re-order the operations behind their back?
- TimorousBestie 1y ago
  Ensuring the same floating-point algorithm workload behaves exactly the same on two distinct workstations is a heck of a lot of work that almost no one is willing to pay for.
  - measurablefunc 1y ago
    Not only that but heterogeneous clusters (inevitable at a large enough scale) will also have non-deterministic outputs. So it's great that they wrote kernels to make the forward pass deterministic but getting rid of it entirely at data center scale would mean that they'd also have to do this type of work across cluster nodes as well to maintain "cluster" invariance & not just batch invariance.
- deleted 1y ago
  [deleted]
- reliabilityguy 1y ago
  > will not re-order the operations behind their back? Valid point. Floating point summation is not always commutative.
- AlotOfReading 1y ago
  You can prevent reordering with sufficient amounts of compiler abuse. With revisions, you're trying to ensure a consistent floating point environment where the operations used are deterministic, and used in the same order with the same inputs. The best way to do that is to use operations that adhere to a mostly deterministic standard like IEEE-754.
- saagarjha 1y ago
  Yes, there’s usually no guarantee on how different hardware does operations (for example, even if the hardware is correctly rounding intermediate results, different hardware may use different tile sizes). The reproducibility here is for runs on the same machine. Compilers can also reorder operations but in practice this is rarely an issue because kernels typically synchronize frequently and this limits the ability for compilers to reorder things. This isn’t to say it doesn’t happen, but even if it does happen it’s likely because the compiler changed because the code they generate is generally run-to-run identical.
lrvick 1y ago
Job one is have every bit of software involved also be deterministic, which stagex takes care of. I had no problem getting deterministic LLM outputs when I experimented with this 6 months ago. Run two of these with the same prompts and same seed and you get the same results. Obviously in GPU clusters with different hardware things get more complicated. https://git.distrust.co/public/llmshell https://git.distrust.co/public/llmshell
- saagarjha 1y ago
  What’s stagex?
  - lrvick 1y ago
    supply chain security focused linux distro that does not trust its own maintainers by design.
- spindump8930 1y ago
  That's not what this is about. "I had no problem getting deterministic LLM outputs when I experimented with this 6 months ago" looks like you're using llama-cpp in that repo. This is about vllm serving many requests at once, at long sequence lengths. > As it turns out, our request’s output does depend on the parallel user requests. Not because we’re somehow leaking information across batches — instead, it’s because our forward pass lacks “batch invariance”, causing our request’s output to depend on the batch size of our forward pass. Your situation isn't really comparable.
lsy 1y ago
Fixing "theoretical" nondeterminism for a totally closed individual input-output pair doesn't solve the two "practical" nondeterminism problems, where the exact same input gives different results given different preceding context, and where a slightly transformed input doesn't give a correctly transformed result. Until those are addressed, closed-system nondeterminism doesn't really help except in cases where a lookup table would do just as well. You can't use "correct" unit tests or evaluation sets to prove anything about inputs you haven't tested.
- saagarjha 1y ago
  This is really useful in reproducing bugs.
- kazinator 1y ago
  There is no such thing as "exactly the same input, but with different preceding context". The preceding context is input! If you were to obtain exactly the same output for a given input prompt, regardless of context, then that would mean that the context is being ignored, which is indistinguishable from the session not maintaining any context such that each prompt is in a brand new empty context. Now what some people want is requirements like: - The different wording of a prompt with exactly the same meaning should not change anything in the output; e.g. whether you say "What is the capital of France" or "What is France's capital" the answer should be verbatim identical. - Prior context should not change responses in ways that don't have any interaction with the context. For instance, a prompt is given "what is 2 + 2", then the answer should always be the same, except if the context instructs the LLM that 2 + 2 is to be five. These kinds of requirements betray a misunderstanding of what these LLMs are.
  - skybrian 1y ago
    I wonder if there's a way to use an LLM to rewrite the prompt, standardizing the wording when two prompts mean the same thing?
    - d4mi3n 1y ago
mg 1y ago
I really hope we will get deterministic LLMs in the future. Even if it causes slightly slower response times. Nondeterminism is what currently keeps me from working with other developers. As I wrote in "Prompt Coding" [1], these days I am not looking for good code. I am looking for prompts that create good code. But how do you share prompts among developers when they produce different code every time? You cannot simply state "Here, I found a prompt that makes gpt-5-2025-08-07 output a solution with all the desired attributes". Similar with images. At the moment, for most image models, you cannot outsource the task of writing prompts that create the desired images. Because most image models will not create the same image when given the same prompt and parameters. [1]: https://www.gibney.org/prompt_coding https://www.gibney.org/prompt_coding
- khimaros 1y ago
  i tried to create a makefile driven workflow based on this idea and ended up with https://github.com/khimaros/enc https://github.com/khimaros/enc -- it suffers from the issues you raised i'm hoping that it becomes more useful as models improve and become more reliable at producing working code (though determinism would be great for improving prompts).
- p1necone 1y ago
  Surely if you end up relying on a given prompt to produce the exact same code every time you should instead just check that code into source control the first time you generate it? A deterministic LLM isn't going to behave appreciably differently from a non deterministic one if your input or context varies by even a tiny bit (pun intended) each time.
  - skybrian 1y ago
    If nothing has changed, caching the result would certainly be cheaper. But if you're doing that as part of a test, it's not really running the test and it might defeat the purpose of the test.
- xnx 1y ago
  > most image models will not create the same image when given the same prompt and parameters. Really? If you include the seed as one of the parameters most produce pixel identical output. E.g. "Generate deterministic images" https://cloud.google.com/vertex-ai/generative-ai/docs/image/generate-deterministic-images https://cloud.google.com/vertex-ai/generative-ai/docs/image/...
TNDnow 1y ago
Who needs a working product when you can spend all day designing the most WEWORK looking website and slap some pseud slop on it. It's like crypto "startups" but it's not even fun.
cubefox 1y ago
His solution still relies on greedy (temperature 0) sampling, which is probably not optimal for model performance on various tasks. For example, Gemini 2.5 uses temperature 1 by default. But deterministic inference with temperature >0 can still be achieved by using pseudorandom sampling with a fixed seed.
- mynameismon 1y ago
  The point of the blog is that even at "supposed" deterministic generative sampling, non-determinism creeps in. This in turn has disastrous effects in very real experiments.
  - cubefox 1y ago
    My point is that greedy sampling is not just not sufficient but also not necessary for deterministic inference.
- red2awn 1y ago
  Conceptually setting temperature to be >0 doesn't actually introduce any non-determinism. If your sampler is seeded then it will always choose the same next token. Higher temperature only flattens the logit distribution.
jll29 1y ago
Sometimes, the reason for non-determinism is implementation-specific. For instance, in GPT-2's source code (I haven't checked other model versions), setting the temperature in the GUI does not lead to a value of 0 but "epsilon" (a very small value larger than 0), to avoid a division by zero error in the code, which makes sense. For many applications, non-determinism implies "useless". This has been a long standing issue with LDA topic models. In particular in the legal, financial and regulatory domains, if a method is not deterministic, it may be illegal to use it or it may lead to follow-on requirements that one does not want (e.g. all screens shown to humans must be preserved to be able to go back and reconstruct what exactly happened to a particular user in a particular second).
eldenring 1y ago
Very impressive! I guess this still wouldn't affect their original example > For example, you might observe that asking ChatGPT the same question multiple times provides different results. even with 0.0 temperature due to MOE models routing at a batch level, and you're very unlikely to get a deterministic batch. > Not because we’re somehow leaking information across batches — instead, it’s because our forward pass lacks “batch invariance”, causing our request’s output to depend on the batch size of our forward pass. The router also leaks batch-level information across sequences.
- boroboro4 1y ago
  > even with 0.0 temperature due to MOE models routing at a batch level, and you're very unlikely to get a deterministic batch. I don’t think this is correct - MoE routing happens at per token basis. It can be non deterministic and batch related if you try to balance out your experts load in a batch but that’s performance optimization (just like all of the blogpost) and not the way models are trained to work.
  - eldenring 1y ago
    Ah interesting, good point. So I guess expert-choice routing leaks across the batch. Now I'm not sure.
syntaxing 1y ago
Super interesting. For those unaware, this is the company Mira Murati (OpenAI previous CTO) started
jasonjmcghee 1y ago
I love high quality blog post style research discussion - Anthropic has been leading the charge with this recently and it's great to see it spreading. OpenAI was also doing this during all the RL research days.
nowittyusername 1y ago
I am baffled that I still run against these statement years after LLM's have been around. LLM's are deterministic and always have been. The reason people are having issues with them is because they are basing their assumptions on api based experiments. Like my man, how can you be making these statements when you haven't done the due diligence of running the LLM on your own hardware with all of the variables locked down and accounted for? If you do just that it would become obviously clear that they are deterministic and most of the time the reason you see the non deterministic behavior is because you have not controlled for a variable. Usually prompt caching, batch processing or some other obvious variable. Now this is related to within same system deterministic behavior. You might get different answers when running on a different gpu, but at least for same systems the behavior is 100% identical if you account for all server startup flags and properly account for things like prompt cashing, slot contamination etc...
- tossandthrow 1y ago
  The article literally justifies This in the second paragraph.
  - nowittyusername 1y ago
    I suppose I have issues with the way "determinism" is used in the title of this article. It can mean different things to different people and in my mind stating that "Defeating Nondeterminism in LLM Inference" frames it as an actual issue with LLM inference. But its not, its an issue with LLM inference when you start using large scale inference with more complex parts such as systems which use multi gpu inference systems or batching processes and other mechanisms. It is not an issue when using an LLM without those more complex parts. Stating it this way muddies the signal and gives a false sense that this is a fundamental issue with architecture, where its an issue of the systems at scale...
- golol 1y ago
  Hold on a second. A transformer produces deterministically a probability distribution over the token alphabet from the context. Then one samples from this distribution. This is random and meant to be random.
threeducks 1y ago
It should also be noted that PyTorch has a page about reproducibility: https://docs.pytorch.org/docs/stable/notes/randomness.html https://docs.pytorch.org/docs/stable/notes/randomness.html TL;DR Seed your PRNGs and call torch.use_deterministic_algorithms(True) to get the deterministic kernels. They may be slightly slower, but in practice, you probably will not notice. Note that results will still differ between different drivers and GPUs. It would be great if NVIDIA tried harder in that regard.
- red2awn 1y ago
  The blog post is about LLM non-determinism in the context of serving at scale (variable batch size). The page you link is only about run-to-run determinism implicitly assuming a fixed batch size.
htrp 1y ago
We know what thinking machines does yet?
sudohalt 1y ago
cool project but if this is what you are producing with $2 billion funding, i doubt you will survive. This is the type of article a grad student would write over a weekend.
- lairv 1y ago
  on the contrary this makes me bullish about their team, it shows that people here care about the craft
  - sudohalt 1y ago
    The team is good, and I enjoyed the read. But this is just an engineering blog post. They're promoting this like it's ground breaking research and it's on their front-page. Ultimately this paper is not very meaningful and just a fun debugging session. I've seen this play out dozens of times. So many startups that have come and go in the bay area were composed of extremely talented individuals, but almost all of them failed.
riazrizvi 1y ago
Natural language is ambiguous. It needs to be. I think the approach here of trying to figure out how to make circles into squares, and argue why circles should be squares, is misguided. Discussions of this type are going to eventually morph into better understanding of how to accept ambiguity and randomness in language, and further shape it with other larger sub-patterns beyond the little proto-grammars that the QKV projection matrices extract.
- atoav 1y ago
  Yes, but determinism != ambiguity, because determinism means: for this exact input the same exact output needs to follow. If I ask the same model the same question I should be able to deterministically get the same answer. Now if we phrase the same question slightly differently we would expect to get a slightly different answer.
  - riazrizvi 1y ago
    Me: What’s an example of a dice roll? LLM: 1 “Language ambiguity with determinism”? Sure I can juxtapose the terms but if it’s semantically inconsistent, then what we mean by that is not a deterministic, definitive thing. You’re chasing your tail on this ‘goal’.
    - Nevermark 1y ago
      Ambiguity: The request/prompt leaves a lot of room for interpretation. Many qualitatively different answers may be correct, relative to the prompt. Different or non-deterministic models will return highly variance results. Determinism: If a model is given the exact same request/prompt twice, its two responses will also be identical. Whether or not the consistent response qualifies as correct. The two concepts are very different. (Ambiguous vs. precise prompt) x (Deterministic vs. Non-deterministic model) = 4 different scenarios. A model itself can be non-deterministic without being ambiguous. If you know exactly how it functions, why it is non-deterministic (batch sensitive for instance), that is not an ambiguous model. Its operation is completely characterized. But it is non-deterministic. An ambiguous model would simply be model whose operation was not characterized. A black box model for instance. A black box model can be deterministic and yet ambiguous.
paulbjensen 1y ago
It reminded me of this wonderful talk by the late Joe Armstrong (Erlang's creator): https://www.youtube.com/watch?v=lKXe3HUG2l4 https://www.youtube.com/watch?v=lKXe3HUG2l4 Great post.
quantum_state 1y ago
As the bottom of LLM inference, it is sampling for the next token based on the probability distribution conditioned on the tokens currently in the context window. If the distribution exhibits degeneracy in probability for more than token, outcome of the sampling will naturally, as it should, be nondeterministic. It should be left alone.
bendoy 1y ago
Where this gets really complicated is when you are chaining many LLM calls together (basically any agent). A slight deviation in the call stack can throw off everything else.
bee_rider 1y ago
From their code: A = torch.randn(2048, 2048, device='cuda', dtype=torch.bfloat16) B = torch.randn(2048, 2048, device='cuda', dtype=torch.bfloat16) ref = torch.mm(A, B) for _ in range(1000): assert (torch.mm(A, B) - ref).abs().max().item() == 0 I’m sort of surprised that Torch doesn’t have some kind of lazy evaluation thing to avoid computing anything here. I thought that was one of the nice things about all these fancy frameworks (if I wanted the computer to actually do silly things when I asked it to, I would use BLAS directly, right?).
- nomel 1y ago
  Maybe I'm missing something, but in this case, wouldn't being lazy would be pure overhead? I don't see anything can be lazy here. The reference computed once, nanoseconds before it's needed, and test cases computed at the time of comparison, then tossed away. What would hope to be achieved by making this case lazy? If you wanted these to run in parallel, with a multi-gpu system, you would use the appropriate parallel interface.
  - bee_rider 1y ago
    I mean if you wait long enough, it is asking for .abs().max().item() of something that can be identified as definitionally zero.
    - nomel 1y ago
      I don't understand. Since it's not using the parallel interface, only one operation can happen at a time. This would be, literally, sequential execution with extra overhead, in this case. Again, in this case, what would hope to be achieved from doing things lazily, since the lazy operations would immediately be followed by their evaluation? The parallel interface, which is async, is probably what you're lookin for.
      - bee_rider 1y ago
        The Python commands are encountered sequentially. One could image a library where the Python commands build the computation under the hood. Then, the library would be able to take advantage of situations like this one (or, more practically, reorder multiplications and/or avoid unnecessary temporaries).
gond 1y ago
I am still irritated by the name of the company. What is the reasoning behind these schemes? The hope that bits of the properties of legendary companies will rub off onto the new venture? As if naming the next best venture PARC will inevitably create a breakthrough in networking just by the arrangement of four letters.
- ricardobeat 1y ago
  Are you talking about the “Thinking Machines” company that shut down in 1994? Took me some digging to figure it out, doesn’t seem well-known enough to be the reason - it’s just a nice (and relatively obvious) name.
  - gond 1y ago
    Yes. Danny Hillis’ Thinking Machines Corporation, an AI company which created its own massive parallel processing supercomputer hardware. “We are building a machine that will be proud of us” was their corporate motto. And that was in 1983. One of those Machines is on view at the Computer History Museum in Mountain View. Back then, they could be ordered in “Darth Vader Black”, no kidding here. You can also see a couple of them (the CM-5) as the stereotypical supercomputer in the original Jurassic Park. More here: https://en.m.wikipedia.org/wiki/Thinking_Machines_Corporation https://en.m.wikipedia.org/wiki/Thinking_Machines_Corporatio...
    - kkylin 1y ago
      And in the original Jurassic Park! https://www.google.com/search?q=jurassic+park+cm-5 https://www.google.com/search?q=jurassic+park+cm-5
      - kkylin 1y ago
        [addendum: posted this too quickly & didn't see it in the comment above. duh.]
  - ewoodrich 1y ago
kybernetikos 1y ago
For fun over the last few days, I've built a compressor / decompressor that uses the logits from an LLM, for each token in the input, then takes the ranks and exponential goolomb encodes them. Then you work in reverse to regenerate the original It took me ages to get the prediction for the second token after "hello" to match the same as the prediction for the second token when running the model on the string "hello world", despite the fact that I was using a causal model. I tried all kinds of things before discovering that `quantized: false` was the important setting.
- giveita 1y ago
  What's the Weissman score? Or more seriously :) did it perform well. Sounds like it should. If more and more text is AI slop it should do well. I dont fully understand what you said but I guess higher probability logits are encoded with fewer bits. If your text is the LLM output then you may need a bit or two per token?
  - kybernetikos 1y ago
    I used exponential golomb coding, so the rank 0 logit is encoded with a single bit, ranks 1 and 2 are encoded with three bits, ranks 3-6 are encoded with 5 bits, etc. In terms of performance, I've not done any serious testing, but e.g. the wikipedia article on volcanos compresses to about 20% using GPT2. I've seen other strings compress even further. The big issue is that while encoding is not unreasonable, decoding any significant amount of data is incredibly slow, since I'm doing a model run for every token in the output. It's bad enough that the scheme is probably unworkable as it is. I'm thinking about changing my code so that it streams out the tokens as it decodes them, so you're not just left there waiting for ages.
    - frotaur 1y ago
      I don't know about golomb coding, but with Arithmetic coding you can do stream decoding(AC), if I remember correctly. I supervised a student's project whose goal was exactly that : implement compression with LLMs using AC. Since AC is optimal, if your LLM has an average cross entropy x on some dataset, you can expect that the compression will compress data using x nats per token on average!
daralthus 1y ago
I thought this was pretty well known (at least in the JAX/XLA world). I've hit this many times and got batch variance explained to me before: https://github.com/google-deepmind/penzai/issues/82 https://github.com/google-deepmind/penzai/issues/82 and https://github.com/jax-ml/jax/issues/20047#issuecomment-1975267301 https://github.com/jax-ml/jax/issues/20047#issuecomment-1975...
- Zacharias030 1y ago
  should be the top comment.
nakamoto_damacy 1y ago
"in collaboration with others at Thinking Machines" If you're old enough, you might remember Danny Hillis' Thinking Machines from the late 80s. I wish they had chosen a different name (I say this for nostalgic reasons, having been in front of one of those cubes glowing with red LEDs back in the late 80s at MIT's AI Lab" (renamed to CSAIL at some point). Feynman did some amazing work on that, too: https://longnow.org/ideas/richard-feynman-and-the-connection-machine/ https://longnow.org/ideas/richard-feynman-and-the-connection... In the U.S., the “THINKING MACHINES” trademarks were owned by Thinking Machines Corporation (the company Hillis co-founded), not Hillis personally, and those registrations were cancelled in 1998–1999. USPTO Report +1 The company itself went bankrupt in 1994 and its assets were dispersed (e.g., to Sun Microsystems, later Oracle). There’s a new, pending USPTO application for “THINKING MACHINES” filed in 2025 by Thinking Machines Lab Inc., the company founded by Amira Murati.
- Imnimo 1y ago
  I make this mistake every time I see their name.
- skeezyboy 1y ago
  [flagged]
  - nakamoto_damacy 1y ago
    I had no idea. But I believe the same deal with Einstein being a dick to his wife and never acknowledging his friend who taught him the math he used in his work (I read about that recently from a respectable source.) I guess that makes sense; no one is void of some deep flaw, it's just selectively hidden.
themeiguoren 1y ago
A bit off topic from the technical discussion but does anyone recognize what blog layout or engine this is? I really like the layout with sidenotes and navigation.
- ako 1y ago
  Seems like a Thufte inspired style, something like this: https://clayh53.github.io/tufte-jekyll/articles/20/tufte-style-jekyll-blog https://clayh53.github.io/tufte-jekyll/articles/20/tufte-sty...
unit149 1y ago
[dead]
emharsha1812 1y ago
I think this is an excellent article which addresses the issue that I personally have been thinking about a long time. And no its not just some slop they put but actual an engineering blog(with open source code and reproducible results!) I think the company is off to a good start
reasonableklout 1y ago
Some great discussion on twitter: https://x.com/thinkymachines/status/1965826369721623001 https://x.com/thinkymachines/status/1965826369721623001 Seems a buried lede is that on-policy RL is unlocked by bitwise identical results between training and sampling. I'm not an expert here but my understanding is that this would allow for stronger guarantees about deployment/training alignment for the RL training that the labs already do. I don't fully understand the BigMath example though. They show that off-policy RLVR requires off-policy correction, which avoids divergence, but is suboptimal because it results in noisy rewards. Then they say "we fixed the sampler and trainer numerical mismatch, which allows for on-policy RL, look how much better it is." It's not clear to me whether this is an artificial example that deliberately uses different trainer/sampler setups, or if it's actually impossible to have the same numerics between trainer/sampler without their fixes (even if we use same batch size, no atomics, etc.).
gajjanag 1y ago
As others have pointed out, these phenomena are well known to many folks across companies in the AI infra space. It doesn't really break new ground. This article is a good exposition of the basic strategies though. What I would have loved is a discussion around collectives/multi-node setups. And showing how to get determinism at low performance penalty for multi-node reduction collectives.
orbital-decay 1y ago
By setting the temperature to 0 you get greedy decoding, which does a lot more than just making it predictable, and can degrade outputs. Random sampling exists for a reason! Gemini 2.5 Pro in particular doesn't like temp 0, for example. Focus on correctness, not determinism.
- matusp 1y ago
  Determinism does not require temperature=0. You can have a deterministic behavior even with >0 temperature as long as you fix your random seeds.
dns_snek 1y ago
Why do you care about determinism in a probabilistic system? What difference does it make to the end user if the input "How do I X?" always produces the same deterministic output when semantically equivalent inputs "how do i x?", "how do I x", and "how do I X??" are bound to produce different answers that often won't even be semantically equivalent. What LLMs need is the ability to guarantee semantically-equivalent outputs for all semantically-equivalent inputs, but that's very different from "determinism" as we understand it from other algorithms.
- ZeljkoS 1y ago
  For "bug" reproduction purposes. It is easier to debug a model if the same string always produces the same incorrect or strange LLM output, not every 100th time you run it.
  - ontouchstart 1y ago
    If there is a bug (a behavior defined by whatever criteria), it is just a single path in a very complex systems with high connectivity. This nonlinear and chaotic behavior regardless of implementation details of the black box makes LLM seem to be nondeterministic. But LLM is just a pseudo random number generator with a probability distribution. (As I am writing this on my iPhone with text completion, I can see this nondeterministic behavior)
- Ratelman 1y ago
  Was my thinking exactly - but also semantically equivalent is also only relevant when it needs to be factual, not necessarily for ALL outputs (if we're aiming for LLM's to present as "human" - or for interactions with LLMs to be natural conversational...). This excludes the world where LLMs act as agents - where you would of course always like the LLM to be factual and thus deterministic.
- Taek 1y ago
  You aren't wrong, but that doesn't mean this level of determinism isn't useful. If you don't even have the level of determinism that the exact same input tokens produce the exact same output tokens, then it's very hard to share reproducible results with peers, which can be useful if you are say, red teaming an LLM to produce a very rare / unreliable output.
PeterStuer 1y ago
THANK YOU! Great work and writeup. Hope it finally silences the "concurrency + floating point" crowd and the "LLMs can never be deterministic" zealots.
menaerus 1y ago
> But why aren’t LLM inference engines deterministic? One common hypothesis is that some combination of floating-point non-associativity and concurrent execution leads to nondeterminism based on which concurrent core finishes first. We will call this the “concurrency + floating point” hypothesis for LLM inference nondeterminism. For example, a recent arXiv preprint writes I'm honored to see that Mira and co. appreciated my feedback on the very topic I made 7 months ago here :D > You don't need RNG since the whole transformer is an extremely large floating-point arithmetic unit. A wild guess - how about the source of non-determinism is coming from the fact that, on the HW level, tensor execution order is not guaranteed and therefore (T0 * T1) * T2 can produce slightly different results than T0 * (T1 * T2) due to rounding errors? https://news.ycombinator.com/item?id=42952605#42960047 https://news.ycombinator.com/item?id=42952605#42960047
zacksiri 1y ago
This work is extremely consequential. When building agentic systems determinism will significantly improve the reliability. I hope all the model providers adopt this.
deleted 1y ago
[deleted]
simne 1y ago
This is eternal struggle. - Hardware developers will constantly scale horizontally and make less (time) deterministic hardware, because wall of memory, and scientists could constantly develop new ways to make calculations deterministic. So, even if will be achieved progress just now, I think in predictable future this will be constant dead-end.
Noumenon72 1y ago
Are the results of the matmuls really that far apart in size that you have to lose significant bits when adding them up at FP32?
andy99 1y ago
Deterministic reproducibility is very different from replicability, and imo the latter is more important; even if the details of the reproducibility are interesting I think they're irrelevant. There's a similar situation in other scientific disciplines. People want source code and data so they can reproduce results - that basically tells you someone didn't cheat and they documented everything. But it does not tell you whether a real phenomenon was observed. It's much more interesting to know if roughly the same cause and effect relationships exist so we can predict behavior. Concretely, there are studies that show e.g. randomly capitalizing letters can lead to completely different responses from and LLM. That speaks to a fragility that doesn't have anything to do with deterministic reproduction.
elanax 1y ago
[dead]

345 points by jxmorris12 1y ago

I think this means that the results might also be non-deterministic across hardware revisions b/c I don't think they verified that the kernels will work the same on different GPU & TPU versions b/c how do they know that the compiler will not re-order the operations behind their back?

TimorousBestie 1y ago
Ensuring the same floating-point algorithm workload behaves exactly the same on two distinct workstations is a heck of a lot of work that almost no one is willing to pay for.
- measurablefunc 1y ago
  Not only that but heterogeneous clusters (inevitable at a large enough scale) will also have non-deterministic outputs. So it's great that they wrote kernels to make the forward pass deterministic but getting rid of it entirely at data center scale would mean that they'd also have to do this type of work across cluster nodes as well to maintain "cluster" invariance & not just batch invariance.
deleted 1y ago
[deleted]
reliabilityguy 1y ago
> will not re-order the operations behind their back? Valid point. Floating point summation is not always commutative.
AlotOfReading 1y ago
You can prevent reordering with sufficient amounts of compiler abuse. With revisions, you're trying to ensure a consistent floating point environment where the operations used are deterministic, and used in the same order with the same inputs. The best way to do that is to use operations that adhere to a mostly deterministic standard like IEEE-754.
saagarjha 1y ago
Yes, there’s usually no guarantee on how different hardware does operations (for example, even if the hardware is correctly rounding intermediate results, different hardware may use different tile sizes). The reproducibility here is for runs on the same machine. Compilers can also reorder operations but in practice this is rarely an issue because kernels typically synchronize frequently and this limits the ability for compilers to reorder things. This isn’t to say it doesn’t happen, but even if it does happen it’s likely because the compiler changed because the code they generate is generally run-to-run identical.

lrvick 1y ago

Job one is have every bit of software involved also be deterministic, which stagex takes care of. I had no problem getting deterministic LLM outputs when I experimented with this 6 months ago. Run two of these with the same prompts and same seed and you get the same results. Obviously in GPU clusters with different hardware things get more complicated. https://git.distrust.co/public/llmshell https://git.distrust.co/public/llmshell

saagarjha 1y ago
What’s stagex?
- lrvick 1y ago
  supply chain security focused linux distro that does not trust its own maintainers by design.
spindump8930 1y ago
That's not what this is about. "I had no problem getting deterministic LLM outputs when I experimented with this 6 months ago" looks like you're using llama-cpp in that repo. This is about vllm serving many requests at once, at long sequence lengths. > As it turns out, our request’s output does depend on the parallel user requests. Not because we’re somehow leaking information across batches — instead, it’s because our forward pass lacks “batch invariance”, causing our request’s output to depend on the batch size of our forward pass. Your situation isn't really comparable.

lsy 1y ago

Fixing "theoretical" nondeterminism for a totally closed individual input-output pair doesn't solve the two "practical" nondeterminism problems, where the exact same input gives different results given different preceding context, and where a slightly transformed input doesn't give a correctly transformed result. Until those are addressed, closed-system nondeterminism doesn't really help except in cases where a lookup table would do just as well. You can't use "correct" unit tests or evaluation sets to prove anything about inputs you haven't tested.

saagarjha 1y ago
This is really useful in reproducing bugs.
kazinator 1y ago
There is no such thing as "exactly the same input, but with different preceding context". The preceding context is input! If you were to obtain exactly the same output for a given input prompt, regardless of context, then that would mean that the context is being ignored, which is indistinguishable from the session not maintaining any context such that each prompt is in a brand new empty context. Now what some people want is requirements like: - The different wording of a prompt with exactly the same meaning should not change anything in the output; e.g. whether you say "What is the capital of France" or "What is France's capital" the answer should be verbatim identical. - Prior context should not change responses in ways that don't have any interaction with the context. For instance, a prompt is given "what is 2 + 2", then the answer should always be the same, except if the context instructs the LLM that 2 + 2 is to be five. These kinds of requirements betray a misunderstanding of what these LLMs are.
- skybrian 1y ago
  I wonder if there's a way to use an LLM to rewrite the prompt, standardizing the wording when two prompts mean the same thing?
  - d4mi3n 1y ago

mg 1y ago

I really hope we will get deterministic LLMs in the future. Even if it causes slightly slower response times. Nondeterminism is what currently keeps me from working with other developers. As I wrote in "Prompt Coding" [1], these days I am not looking for good code. I am looking for prompts that create good code. But how do you share prompts among developers when they produce different code every time? You cannot simply state "Here, I found a prompt that makes gpt-5-2025-08-07 output a solution with all the desired attributes". Similar with images. At the moment, for most image models, you cannot outsource the task of writing prompts that create the desired images. Because most image models will not create the same image when given the same prompt and parameters. [1]: https://www.gibney.org/prompt_coding https://www.gibney.org/prompt_coding

khimaros 1y ago
i tried to create a makefile driven workflow based on this idea and ended up with https://github.com/khimaros/enc https://github.com/khimaros/enc -- it suffers from the issues you raised i'm hoping that it becomes more useful as models improve and become more reliable at producing working code (though determinism would be great for improving prompts).
p1necone 1y ago
Surely if you end up relying on a given prompt to produce the exact same code every time you should instead just check that code into source control the first time you generate it? A deterministic LLM isn't going to behave appreciably differently from a non deterministic one if your input or context varies by even a tiny bit (pun intended) each time.
- skybrian 1y ago
  If nothing has changed, caching the result would certainly be cheaper. But if you're doing that as part of a test, it's not really running the test and it might defeat the purpose of the test.
xnx 1y ago
> most image models will not create the same image when given the same prompt and parameters. Really? If you include the seed as one of the parameters most produce pixel identical output. E.g. "Generate deterministic images" https://cloud.google.com/vertex-ai/generative-ai/docs/image/generate-deterministic-images https://cloud.google.com/vertex-ai/generative-ai/docs/image/...

TNDnow 1y ago

Who needs a working product when you can spend all day designing the most WEWORK looking website and slap some pseud slop on it. It's like crypto "startups" but it's not even fun.

cubefox 1y ago

His solution still relies on greedy (temperature 0) sampling, which is probably not optimal for model performance on various tasks. For example, Gemini 2.5 uses temperature 1 by default. But deterministic inference with temperature >0 can still be achieved by using pseudorandom sampling with a fixed seed.

mynameismon 1y ago
The point of the blog is that even at "supposed" deterministic generative sampling, non-determinism creeps in. This in turn has disastrous effects in very real experiments.
- cubefox 1y ago
  My point is that greedy sampling is not just not sufficient but also not necessary for deterministic inference.
red2awn 1y ago
Conceptually setting temperature to be >0 doesn't actually introduce any non-determinism. If your sampler is seeded then it will always choose the same next token. Higher temperature only flattens the logit distribution.

jll29 1y ago

Sometimes, the reason for non-determinism is implementation-specific. For instance, in GPT-2's source code (I haven't checked other model versions), setting the temperature in the GUI does not lead to a value of 0 but "epsilon" (a very small value larger than 0), to avoid a division by zero error in the code, which makes sense. For many applications, non-determinism implies "useless". This has been a long standing issue with LDA topic models. In particular in the legal, financial and regulatory domains, if a method is not deterministic, it may be illegal to use it or it may lead to follow-on requirements that one does not want (e.g. all screens shown to humans must be preserved to be able to go back and reconstruct what exactly happened to a particular user in a particular second).

eldenring 1y ago

Very impressive! I guess this still wouldn't affect their original example > For example, you might observe that asking ChatGPT the same question multiple times provides different results. even with 0.0 temperature due to MOE models routing at a batch level, and you're very unlikely to get a deterministic batch. > Not because we’re somehow leaking information across batches — instead, it’s because our forward pass lacks “batch invariance”, causing our request’s output to depend on the batch size of our forward pass. The router also leaks batch-level information across sequences.

boroboro4 1y ago
> even with 0.0 temperature due to MOE models routing at a batch level, and you're very unlikely to get a deterministic batch. I don’t think this is correct - MoE routing happens at per token basis. It can be non deterministic and batch related if you try to balance out your experts load in a batch but that’s performance optimization (just like all of the blogpost) and not the way models are trained to work.
- eldenring 1y ago
  Ah interesting, good point. So I guess expert-choice routing leaks across the batch. Now I'm not sure.

syntaxing 1y ago

Super interesting. For those unaware, this is the company Mira Murati (OpenAI previous CTO) started

jasonjmcghee 1y ago

I love high quality blog post style research discussion - Anthropic has been leading the charge with this recently and it's great to see it spreading. OpenAI was also doing this during all the RL research days.

nowittyusername 1y ago

I am baffled that I still run against these statement years after LLM's have been around. LLM's are deterministic and always have been. The reason people are having issues with them is because they are basing their assumptions on api based experiments. Like my man, how can you be making these statements when you haven't done the due diligence of running the LLM on your own hardware with all of the variables locked down and accounted for? If you do just that it would become obviously clear that they are deterministic and most of the time the reason you see the non deterministic behavior is because you have not controlled for a variable. Usually prompt caching, batch processing or some other obvious variable. Now this is related to within same system deterministic behavior. You might get different answers when running on a different gpu, but at least for same systems the behavior is 100% identical if you account for all server startup flags and properly account for things like prompt cashing, slot contamination etc...

tossandthrow 1y ago
The article literally justifies This in the second paragraph.
- nowittyusername 1y ago
  I suppose I have issues with the way "determinism" is used in the title of this article. It can mean different things to different people and in my mind stating that "Defeating Nondeterminism in LLM Inference" frames it as an actual issue with LLM inference. But its not, its an issue with LLM inference when you start using large scale inference with more complex parts such as systems which use multi gpu inference systems or batching processes and other mechanisms. It is not an issue when using an LLM without those more complex parts. Stating it this way muddies the signal and gives a false sense that this is a fundamental issue with architecture, where its an issue of the systems at scale...
golol 1y ago
Hold on a second. A transformer produces deterministically a probability distribution over the token alphabet from the context. Then one samples from this distribution. This is random and meant to be random.

threeducks 1y ago

It should also be noted that PyTorch has a page about reproducibility: https://docs.pytorch.org/docs/stable/notes/randomness.html https://docs.pytorch.org/docs/stable/notes/randomness.html TL;DR Seed your PRNGs and call torch.use_deterministic_algorithms(True) to get the deterministic kernels. They may be slightly slower, but in practice, you probably will not notice. Note that results will still differ between different drivers and GPUs. It would be great if NVIDIA tried harder in that regard.

red2awn 1y ago
The blog post is about LLM non-determinism in the context of serving at scale (variable batch size). The page you link is only about run-to-run determinism implicitly assuming a fixed batch size.

htrp 1y ago

We know what thinking machines does yet?

sudohalt 1y ago

cool project but if this is what you are producing with $2 billion funding, i doubt you will survive. This is the type of article a grad student would write over a weekend.

lairv 1y ago
on the contrary this makes me bullish about their team, it shows that people here care about the craft
- sudohalt 1y ago
  The team is good, and I enjoyed the read. But this is just an engineering blog post. They're promoting this like it's ground breaking research and it's on their front-page. Ultimately this paper is not very meaningful and just a fun debugging session. I've seen this play out dozens of times. So many startups that have come and go in the bay area were composed of extremely talented individuals, but almost all of them failed.

riazrizvi 1y ago

Natural language is ambiguous. It needs to be. I think the approach here of trying to figure out how to make circles into squares, and argue why circles should be squares, is misguided. Discussions of this type are going to eventually morph into better understanding of how to accept ambiguity and randomness in language, and further shape it with other larger sub-patterns beyond the little proto-grammars that the QKV projection matrices extract.

atoav 1y ago
Yes, but determinism != ambiguity, because determinism means: for this exact input the same exact output needs to follow. If I ask the same model the same question I should be able to deterministically get the same answer. Now if we phrase the same question slightly differently we would expect to get a slightly different answer.
- riazrizvi 1y ago
  Me: What’s an example of a dice roll? LLM: 1 “Language ambiguity with determinism”? Sure I can juxtapose the terms but if it’s semantically inconsistent, then what we mean by that is not a deterministic, definitive thing. You’re chasing your tail on this ‘goal’.
  - Nevermark 1y ago
    Ambiguity: The request/prompt leaves a lot of room for interpretation. Many qualitatively different answers may be correct, relative to the prompt. Different or non-deterministic models will return highly variance results. Determinism: If a model is given the exact same request/prompt twice, its two responses will also be identical. Whether or not the consistent response qualifies as correct. The two concepts are very different. (Ambiguous vs. precise prompt) x (Deterministic vs. Non-deterministic model) = 4 different scenarios. A model itself can be non-deterministic without being ambiguous. If you know exactly how it functions, why it is non-deterministic (batch sensitive for instance), that is not an ambiguous model. Its operation is completely characterized. But it is non-deterministic. An ambiguous model would simply be model whose operation was not characterized. A black box model for instance. A black box model can be deterministic and yet ambiguous.

paulbjensen 1y ago

It reminded me of this wonderful talk by the late Joe Armstrong (Erlang's creator): https://www.youtube.com/watch?v=lKXe3HUG2l4 https://www.youtube.com/watch?v=lKXe3HUG2l4 Great post.

quantum_state 1y ago

As the bottom of LLM inference, it is sampling for the next token based on the probability distribution conditioned on the tokens currently in the context window. If the distribution exhibits degeneracy in probability for more than token, outcome of the sampling will naturally, as it should, be nondeterministic. It should be left alone.

bendoy 1y ago

Where this gets really complicated is when you are chaining many LLM calls together (basically any agent). A slight deviation in the call stack can throw off everything else.

bee_rider 1y ago

From their code: A = torch.randn(2048, 2048, device='cuda', dtype=torch.bfloat16) B = torch.randn(2048, 2048, device='cuda', dtype=torch.bfloat16) ref = torch.mm(A, B) for _ in range(1000): assert (torch.mm(A, B) - ref).abs().max().item() == 0 I’m sort of surprised that Torch doesn’t have some kind of lazy evaluation thing to avoid computing anything here. I thought that was one of the nice things about all these fancy frameworks (if I wanted the computer to actually do silly things when I asked it to, I would use BLAS directly, right?).

nomel 1y ago
Maybe I'm missing something, but in this case, wouldn't being lazy would be pure overhead? I don't see anything can be lazy here. The reference computed once, nanoseconds before it's needed, and test cases computed at the time of comparison, then tossed away. What would hope to be achieved by making this case lazy? If you wanted these to run in parallel, with a multi-gpu system, you would use the appropriate parallel interface.
- bee_rider 1y ago
  I mean if you wait long enough, it is asking for .abs().max().item() of something that can be identified as definitionally zero.
  - nomel 1y ago
    I don't understand. Since it's not using the parallel interface, only one operation can happen at a time. This would be, literally, sequential execution with extra overhead, in this case. Again, in this case, what would hope to be achieved from doing things lazily, since the lazy operations would immediately be followed by their evaluation? The parallel interface, which is async, is probably what you're lookin for.
    - bee_rider 1y ago
      The Python commands are encountered sequentially. One could image a library where the Python commands build the computation under the hood. Then, the library would be able to take advantage of situations like this one (or, more practically, reorder multiplications and/or avoid unnecessary temporaries).

gond 1y ago

I am still irritated by the name of the company. What is the reasoning behind these schemes? The hope that bits of the properties of legendary companies will rub off onto the new venture? As if naming the next best venture PARC will inevitably create a breakthrough in networking just by the arrangement of four letters.

ricardobeat 1y ago
Are you talking about the “Thinking Machines” company that shut down in 1994? Took me some digging to figure it out, doesn’t seem well-known enough to be the reason - it’s just a nice (and relatively obvious) name.
- gond 1y ago
  Yes. Danny Hillis’ Thinking Machines Corporation, an AI company which created its own massive parallel processing supercomputer hardware. “We are building a machine that will be proud of us” was their corporate motto. And that was in 1983. One of those Machines is on view at the Computer History Museum in Mountain View. Back then, they could be ordered in “Darth Vader Black”, no kidding here. You can also see a couple of them (the CM-5) as the stereotypical supercomputer in the original Jurassic Park. More here: https://en.m.wikipedia.org/wiki/Thinking_Machines_Corporation https://en.m.wikipedia.org/wiki/Thinking_Machines_Corporatio...
  - kkylin 1y ago
    And in the original Jurassic Park! https://www.google.com/search?q=jurassic+park+cm-5 https://www.google.com/search?q=jurassic+park+cm-5
    - kkylin 1y ago
      [addendum: posted this too quickly & didn't see it in the comment above. duh.]
- ewoodrich 1y ago

kybernetikos 1y ago

For fun over the last few days, I've built a compressor / decompressor that uses the logits from an LLM, for each token in the input, then takes the ranks and exponential goolomb encodes them. Then you work in reverse to regenerate the original It took me ages to get the prediction for the second token after "hello" to match the same as the prediction for the second token when running the model on the string "hello world", despite the fact that I was using a causal model. I tried all kinds of things before discovering that `quantized: false` was the important setting.

giveita 1y ago
What's the Weissman score? Or more seriously :) did it perform well. Sounds like it should. If more and more text is AI slop it should do well. I dont fully understand what you said but I guess higher probability logits are encoded with fewer bits. If your text is the LLM output then you may need a bit or two per token?
- kybernetikos 1y ago
  I used exponential golomb coding, so the rank 0 logit is encoded with a single bit, ranks 1 and 2 are encoded with three bits, ranks 3-6 are encoded with 5 bits, etc. In terms of performance, I've not done any serious testing, but e.g. the wikipedia article on volcanos compresses to about 20% using GPT2. I've seen other strings compress even further. The big issue is that while encoding is not unreasonable, decoding any significant amount of data is incredibly slow, since I'm doing a model run for every token in the output. It's bad enough that the scheme is probably unworkable as it is. I'm thinking about changing my code so that it streams out the tokens as it decodes them, so you're not just left there waiting for ages.
  - frotaur 1y ago
    I don't know about golomb coding, but with Arithmetic coding you can do stream decoding(AC), if I remember correctly. I supervised a student's project whose goal was exactly that : implement compression with LLMs using AC. Since AC is optimal, if your LLM has an average cross entropy x on some dataset, you can expect that the compression will compress data using x nats per token on average!

daralthus 1y ago

I thought this was pretty well known (at least in the JAX/XLA world). I've hit this many times and got batch variance explained to me before: https://github.com/google-deepmind/penzai/issues/82 https://github.com/google-deepmind/penzai/issues/82 and https://github.com/jax-ml/jax/issues/20047#issuecomment-1975267301 https://github.com/jax-ml/jax/issues/20047#issuecomment-1975...

Zacharias030 1y ago
should be the top comment.

nakamoto_damacy 1y ago

"in collaboration with others at Thinking Machines" If you're old enough, you might remember Danny Hillis' Thinking Machines from the late 80s. I wish they had chosen a different name (I say this for nostalgic reasons, having been in front of one of those cubes glowing with red LEDs back in the late 80s at MIT's AI Lab" (renamed to CSAIL at some point). Feynman did some amazing work on that, too: https://longnow.org/ideas/richard-feynman-and-the-connection-machine/ https://longnow.org/ideas/richard-feynman-and-the-connection... In the U.S., the “THINKING MACHINES” trademarks were owned by Thinking Machines Corporation (the company Hillis co-founded), not Hillis personally, and those registrations were cancelled in 1998–1999. USPTO Report +1 The company itself went bankrupt in 1994 and its assets were dispersed (e.g., to Sun Microsystems, later Oracle). There’s a new, pending USPTO application for “THINKING MACHINES” filed in 2025 by Thinking Machines Lab Inc., the company founded by Amira Murati.

Imnimo 1y ago
I make this mistake every time I see their name.
skeezyboy 1y ago
[flagged]
- nakamoto_damacy 1y ago
  I had no idea. But I believe the same deal with Einstein being a dick to his wife and never acknowledging his friend who taught him the math he used in his work (I read about that recently from a respectable source.) I guess that makes sense; no one is void of some deep flaw, it's just selectively hidden.

themeiguoren 1y ago

A bit off topic from the technical discussion but does anyone recognize what blog layout or engine this is? I really like the layout with sidenotes and navigation.

ako 1y ago
Seems like a Thufte inspired style, something like this: https://clayh53.github.io/tufte-jekyll/articles/20/tufte-style-jekyll-blog https://clayh53.github.io/tufte-jekyll/articles/20/tufte-sty...

unit149 1y ago

[dead]

emharsha1812 1y ago

I think this is an excellent article which addresses the issue that I personally have been thinking about a long time. And no its not just some slop they put but actual an engineering blog(with open source code and reproducible results!) I think the company is off to a good start

reasonableklout 1y ago

Some great discussion on twitter: https://x.com/thinkymachines/status/1965826369721623001 https://x.com/thinkymachines/status/1965826369721623001 Seems a buried lede is that on-policy RL is unlocked by bitwise identical results between training and sampling. I'm not an expert here but my understanding is that this would allow for stronger guarantees about deployment/training alignment for the RL training that the labs already do. I don't fully understand the BigMath example though. They show that off-policy RLVR requires off-policy correction, which avoids divergence, but is suboptimal because it results in noisy rewards. Then they say "we fixed the sampler and trainer numerical mismatch, which allows for on-policy RL, look how much better it is." It's not clear to me whether this is an artificial example that deliberately uses different trainer/sampler setups, or if it's actually impossible to have the same numerics between trainer/sampler without their fixes (even if we use same batch size, no atomics, etc.).

gajjanag 1y ago

As others have pointed out, these phenomena are well known to many folks across companies in the AI infra space. It doesn't really break new ground. This article is a good exposition of the basic strategies though. What I would have loved is a discussion around collectives/multi-node setups. And showing how to get determinism at low performance penalty for multi-node reduction collectives.

orbital-decay 1y ago

By setting the temperature to 0 you get greedy decoding, which does a lot more than just making it predictable, and can degrade outputs. Random sampling exists for a reason! Gemini 2.5 Pro in particular doesn't like temp 0, for example. Focus on correctness, not determinism.

matusp 1y ago
Determinism does not require temperature=0. You can have a deterministic behavior even with >0 temperature as long as you fix your random seeds.

dns_snek 1y ago

Why do you care about determinism in a probabilistic system? What difference does it make to the end user if the input "How do I X?" always produces the same deterministic output when semantically equivalent inputs "how do i x?", "how do I x", and "how do I X??" are bound to produce different answers that often won't even be semantically equivalent. What LLMs need is the ability to guarantee semantically-equivalent outputs for all semantically-equivalent inputs, but that's very different from "determinism" as we understand it from other algorithms.

ZeljkoS 1y ago
For "bug" reproduction purposes. It is easier to debug a model if the same string always produces the same incorrect or strange LLM output, not every 100th time you run it.
- ontouchstart 1y ago
  If there is a bug (a behavior defined by whatever criteria), it is just a single path in a very complex systems with high connectivity. This nonlinear and chaotic behavior regardless of implementation details of the black box makes LLM seem to be nondeterministic. But LLM is just a pseudo random number generator with a probability distribution. (As I am writing this on my iPhone with text completion, I can see this nondeterministic behavior)
Ratelman 1y ago
Was my thinking exactly - but also semantically equivalent is also only relevant when it needs to be factual, not necessarily for ALL outputs (if we're aiming for LLM's to present as "human" - or for interactions with LLMs to be natural conversational...). This excludes the world where LLMs act as agents - where you would of course always like the LLM to be factual and thus deterministic.
Taek 1y ago
You aren't wrong, but that doesn't mean this level of determinism isn't useful. If you don't even have the level of determinism that the exact same input tokens produce the exact same output tokens, then it's very hard to share reproducible results with peers, which can be useful if you are say, red teaming an LLM to produce a very rare / unreliable output.

PeterStuer 1y ago

THANK YOU! Great work and writeup. Hope it finally silences the "concurrency + floating point" crowd and the "LLMs can never be deterministic" zealots.

menaerus 1y ago

> But why aren’t LLM inference engines deterministic? One common hypothesis is that some combination of floating-point non-associativity and concurrent execution leads to nondeterminism based on which concurrent core finishes first. We will call this the “concurrency + floating point” hypothesis for LLM inference nondeterminism. For example, a recent arXiv preprint writes I'm honored to see that Mira and co. appreciated my feedback on the very topic I made 7 months ago here :D > You don't need RNG since the whole transformer is an extremely large floating-point arithmetic unit. A wild guess - how about the source of non-determinism is coming from the fact that, on the HW level, tensor execution order is not guaranteed and therefore (T0 * T1) * T2 can produce slightly different results than T0 * (T1 * T2) due to rounding errors? https://news.ycombinator.com/item?id=42952605#42960047 https://news.ycombinator.com/item?id=42952605#42960047

zacksiri 1y ago

This work is extremely consequential. When building agentic systems determinism will significantly improve the reliability. I hope all the model providers adopt this.

deleted 1y ago

[deleted]

simne 1y ago

This is eternal struggle. - Hardware developers will constantly scale horizontally and make less (time) deterministic hardware, because wall of memory, and scientists could constantly develop new ways to make calculations deterministic. So, even if will be achieved progress just now, I think in predictable future this will be constant dead-end.

Noumenon72 1y ago

Are the results of the matmuls really that far apart in size that you have to lose significant bits when adding them up at FP32?

andy99 1y ago

Deterministic reproducibility is very different from replicability, and imo the latter is more important; even if the details of the reproducibility are interesting I think they're irrelevant. There's a similar situation in other scientific disciplines. People want source code and data so they can reproduce results - that basically tells you someone didn't cheat and they documented everything. But it does not tell you whether a real phenomenon was observed. It's much more interesting to know if roughly the same cause and effect relationships exist so we can predict behavior. Concretely, there are studies that show e.g. randomly capitalizing letters can lead to completely different responses from and LLM. That speaks to a fragility that doesn't have anything to do with deterministic reproduction.

elanax 1y ago

[dead]