4 ms·

How to wrangle non-deterministic AI outputs into conventional software? (2025)

42 points by druther 9mo ago

ironbound 9mo ago
[flagged]
- dang 9mo ago
  "Don't be snarky." "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
ramity 9mo ago
Let me first start off by saying I and many others have stepped in this pitfall. This is not an attack, but a good faith attempt to share painfully acquired knowledge. I'm actively using AI tooling, and this comment isn't a slight on the tooling but rather how we're all seemingly putting the circle in the square hole and it fits. Querying an LLM to output its confidence in its output is a misguided pattern despite being commonly applied by many. LLMs are not good at classification tasks as the author states. They can "do" it, yes. Perhaps better than random sampling can, but random sampling can "do" it as well. Don't get too tied to that example. The idea here is that if you are okay with something getting the answer wrong every so often, LLMs might be your solve, but this is a post about conforming non-deterministic AI into classical systems. Are you okay if your agentic agent picks the red tool instead of the blue tool 1%, 10%, etc of the time? If so, you're never not going to be wrangling, and that's the reality often left unspoken when integrating these tools. While tangential to this article, I believe its worth stating that when interacting with an LLM in any capacity, remember your own cognitive biases. You often want the response to work, and while generated responses may look good and fit your mental model, it requires increasingly obscene levels of critical evaluation to see through the fluff. For some, there will be inevitable dissonance reading this, but consider that these experiments are local examples. Its lack of robustness will become apparent with large scale testing. The data spaces these models have been trained on are unfathomably large in both quantity and depth, but under/over sampling bias will be ever present (just to name one). Consider the the following thought experiment: You are an applicant for a job submitting your resume with knowledge it will be fed into an LLM. Let's confine your goal into something very simple. Make it say something. Let's oversimplify for the sake of the example and say complete words are tokens. Consider "collocations". [Bated] breath, [batten] down, [diametrically] opposed, [inclement] weather, [hermetically] sealed. Extend this to contexts. [Oligarchy] government, [Chromosome] biology, [Paradigm] technology, [Decimate] to kill. With this in mind, consider how each word of your resume "steers" the model's subsequent response, and consider how the data each model is trained on can subtly influence its response. Now let's bring it home and tie the thought experiment into confidence scoring in responses. Let's say its reasonable to assume that the results of low accuracy/low confidence models are less commonly found on the internet than higher performing ones. If that can be entertained, extend the argument to confidence responses. Maybe the term "JSON" or any other term used in the model input is associated with high confidences. Alright, wrapping it up. The end point here is that the model output provided confidence value is not the likelihood of the answer provided in the response but rather the most likely value following the stream of tokens in the combined input and output. The real sampled confidence values exist closer to code, but they are limited to each token. Not series of tokens.
- arduanika 9mo ago
  "when interacting with an LLM in any capacity, remember your own cognitive biases. You often want the response to work, and while generated responses may look good and fit your mental model, it requires increasingly obscene levels of critical evaluation to see through the fluff." 100% this. Idk about the far-out takes where "AI is an alien lifeform arrived into our present", but the first thing we know about how humans relate to extraterrestrials is: "I want to believe".
an0malous 9mo ago
Aren’t transformers intrinsically deterministic? I thought the randomness was intentional to make chatbots seem more natural, and OpenAI used to have a seed parameter you could set for deterministic output. I don’t know why that feature isn’t more popular, for the reasons this article outlines
- ares623 9mo ago
  Maybe it allowed spitting out copyrighted works verbatim
- jkaptur 9mo ago
  (I'm not an expert. I'd love to be corrected by someone who actually knows.) Floating-point arithmetic is not associative. (A+B)+C does not necessarily equal A+(B+C), but you can get a performance improvement by calculating A, B, and C in parallel, then adding together whichever two finish first. So, in theory, transformers can be deterministic, but in a real system they almost always aren't.
  - 10000truths 9mo ago
    Not an expert either, but my understanding is that large models use quantized weights and tensor inputs for inference. Multiplication and addition of fixed-point values is associative, so unless there's an intermediate "convert to/from IEEE float" step (activation functions, maybe?), you can still build determinism into a performant model.
    - kimixa 9mo ago
      Fixed point arithmetic isn't truly associative unless they have infinite precision. The second you hit a limit or saturate/clamp a value the result very much depends on order of operations.
      - 10000truths 9mo ago
        Ah yes, I forgot about saturating arithmetic. But even for that, you wouldn't need infinite precision for all values, you'd only need "enough" precision for the intermediate values, right? E.g. for an inner product of two N-element vectors containing M-bit integers, an accumulator with at least ceil(log2(N))+2*M bits would guarantee no overflow.
ironbound 9mo ago
Use one of these structured output libraries: https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines https://github.com/jxnl/instructor https://github.com/jxnl/instructor https://github.com/guardrails-ai/guardrails https://github.com/guardrails-ai/guardrails https://www.askmarvin.ai/docs/text/transformation/ https://www.askmarvin.ai/docs/text/transformation/ Some of them allow a JSON schema, others a Pydantic model (which you can transform to/from JSON).
- zby 9mo ago
  Yeah - the author seems oblivious to that option. But to be fair - this post is more about the more basic step of choosing the schema - he argues that it is still a task for a human. In his next post https://www.domainlanguage.com/articles/context-mapping-an-ai-based-component/ https://www.domainlanguage.com/articles/context-mapping-an-a... applying some output schemas would be more useful. By they way you don' need to use library to have output schemas: https://platform.openai.com/docs/guides/structured-outputs https://platform.openai.com/docs/guides/structured-outputs , https://platform.claude.com/docs/en/build-with-claude/structured-outputs https://platform.claude.com/docs/en/build-with-claude/struct...
- iLoveOncall 9mo ago
  All those do is retry the generation if it fails and then give up. Calling them structured output is a lie, it's structured validation. If the LLM persists in generating bad JSON, those are useless.
- johndough 9mo ago
  To add to that, llama.cpp supports GBNF grammars, which allows generation of JSON, or even programming languages: https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md https://github.com/ggml-org/llama.cpp/blob/master/grammars/R...
galaxyLogic 9mo ago
I wonder if the same problem exists with AI based code-development more specifically. To produce an application you should have some good unit-tests. So AI produces some unit-tests for us. Then we ask it again, and the unit-tests will be different. Where does that leave us? Can we be confident that the generated unit-tests are in some sense "correct"? How can they be correct if they are different every time we ask??
gulugawa 9mo ago
This page has some advice: https://p-nand-q.com/programming/languages/java2k/ https://p-nand-q.com/programming/languages/java2k/
Legend2440 9mo ago
The real issue here is that conventional software fundamentally lacks the expressiveness to process the kind of data that LLMs can. That’s why you’re using an LLM in the first place.
- qayxc 9mo ago
  How does this relate to the need for deterministic and consistent output?
noreplydev 9mo ago
this page speaks about the same with some different approach https://selfvm.run/research https://selfvm.run/research

42 points by druther 9mo ago

ironbound 9mo ago

[flagged]

dang 9mo ago
"Don't be snarky." "Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

ramity 9mo ago

Let me first start off by saying I and many others have stepped in this pitfall. This is not an attack, but a good faith attempt to share painfully acquired knowledge. I'm actively using AI tooling, and this comment isn't a slight on the tooling but rather how we're all seemingly putting the circle in the square hole and it fits. Querying an LLM to output its confidence in its output is a misguided pattern despite being commonly applied by many. LLMs are not good at classification tasks as the author states. They can "do" it, yes. Perhaps better than random sampling can, but random sampling can "do" it as well. Don't get too tied to that example. The idea here is that if you are okay with something getting the answer wrong every so often, LLMs might be your solve, but this is a post about conforming non-deterministic AI into classical systems. Are you okay if your agentic agent picks the red tool instead of the blue tool 1%, 10%, etc of the time? If so, you're never not going to be wrangling, and that's the reality often left unspoken when integrating these tools. While tangential to this article, I believe its worth stating that when interacting with an LLM in any capacity, remember your own cognitive biases. You often want the response to work, and while generated responses may look good and fit your mental model, it requires increasingly obscene levels of critical evaluation to see through the fluff. For some, there will be inevitable dissonance reading this, but consider that these experiments are local examples. Its lack of robustness will become apparent with large scale testing. The data spaces these models have been trained on are unfathomably large in both quantity and depth, but under/over sampling bias will be ever present (just to name one). Consider the the following thought experiment: You are an applicant for a job submitting your resume with knowledge it will be fed into an LLM. Let's confine your goal into something very simple. Make it say something. Let's oversimplify for the sake of the example and say complete words are tokens. Consider "collocations". [Bated] breath, [batten] down, [diametrically] opposed, [inclement] weather, [hermetically] sealed. Extend this to contexts. [Oligarchy] government, [Chromosome] biology, [Paradigm] technology, [Decimate] to kill. With this in mind, consider how each word of your resume "steers" the model's subsequent response, and consider how the data each model is trained on can subtly influence its response. Now let's bring it home and tie the thought experiment into confidence scoring in responses. Let's say its reasonable to assume that the results of low accuracy/low confidence models are less commonly found on the internet than higher performing ones. If that can be entertained, extend the argument to confidence responses. Maybe the term "JSON" or any other term used in the model input is associated with high confidences. Alright, wrapping it up. The end point here is that the model output provided confidence value is not the likelihood of the answer provided in the response but rather the most likely value following the stream of tokens in the combined input and output. The real sampled confidence values exist closer to code, but they are limited to each token. Not series of tokens.

arduanika 9mo ago
"when interacting with an LLM in any capacity, remember your own cognitive biases. You often want the response to work, and while generated responses may look good and fit your mental model, it requires increasingly obscene levels of critical evaluation to see through the fluff." 100% this. Idk about the far-out takes where "AI is an alien lifeform arrived into our present", but the first thing we know about how humans relate to extraterrestrials is: "I want to believe".

an0malous 9mo ago

Aren’t transformers intrinsically deterministic? I thought the randomness was intentional to make chatbots seem more natural, and OpenAI used to have a seed parameter you could set for deterministic output. I don’t know why that feature isn’t more popular, for the reasons this article outlines

ares623 9mo ago
Maybe it allowed spitting out copyrighted works verbatim
jkaptur 9mo ago
(I'm not an expert. I'd love to be corrected by someone who actually knows.) Floating-point arithmetic is not associative. (A+B)+C does not necessarily equal A+(B+C), but you can get a performance improvement by calculating A, B, and C in parallel, then adding together whichever two finish first. So, in theory, transformers can be deterministic, but in a real system they almost always aren't.
- 10000truths 9mo ago
  Not an expert either, but my understanding is that large models use quantized weights and tensor inputs for inference. Multiplication and addition of fixed-point values is associative, so unless there's an intermediate "convert to/from IEEE float" step (activation functions, maybe?), you can still build determinism into a performant model.
  - kimixa 9mo ago
    Fixed point arithmetic isn't truly associative unless they have infinite precision. The second you hit a limit or saturate/clamp a value the result very much depends on order of operations.
    - 10000truths 9mo ago
      Ah yes, I forgot about saturating arithmetic. But even for that, you wouldn't need infinite precision for all values, you'd only need "enough" precision for the intermediate values, right? E.g. for an inner product of two N-element vectors containing M-bit integers, an accumulator with at least ceil(log2(N))+2*M bits would guarantee no overflow.

ironbound 9mo ago

Use one of these structured output libraries: https://github.com/outlines-dev/outlines https://github.com/outlines-dev/outlines https://github.com/jxnl/instructor https://github.com/jxnl/instructor https://github.com/guardrails-ai/guardrails https://github.com/guardrails-ai/guardrails https://www.askmarvin.ai/docs/text/transformation/ https://www.askmarvin.ai/docs/text/transformation/ Some of them allow a JSON schema, others a Pydantic model (which you can transform to/from JSON).

zby 9mo ago
Yeah - the author seems oblivious to that option. But to be fair - this post is more about the more basic step of choosing the schema - he argues that it is still a task for a human. In his next post https://www.domainlanguage.com/articles/context-mapping-an-ai-based-component/ https://www.domainlanguage.com/articles/context-mapping-an-a... applying some output schemas would be more useful. By they way you don' need to use library to have output schemas: https://platform.openai.com/docs/guides/structured-outputs https://platform.openai.com/docs/guides/structured-outputs , https://platform.claude.com/docs/en/build-with-claude/structured-outputs https://platform.claude.com/docs/en/build-with-claude/struct...
iLoveOncall 9mo ago
All those do is retry the generation if it fails and then give up. Calling them structured output is a lie, it's structured validation. If the LLM persists in generating bad JSON, those are useless.
johndough 9mo ago
To add to that, llama.cpp supports GBNF grammars, which allows generation of JSON, or even programming languages: https://github.com/ggml-org/llama.cpp/blob/master/grammars/README.md https://github.com/ggml-org/llama.cpp/blob/master/grammars/R...

galaxyLogic 9mo ago

I wonder if the same problem exists with AI based code-development more specifically. To produce an application you should have some good unit-tests. So AI produces some unit-tests for us. Then we ask it again, and the unit-tests will be different. Where does that leave us? Can we be confident that the generated unit-tests are in some sense "correct"? How can they be correct if they are different every time we ask??

gulugawa 9mo ago

This page has some advice: https://p-nand-q.com/programming/languages/java2k/ https://p-nand-q.com/programming/languages/java2k/

Legend2440 9mo ago

The real issue here is that conventional software fundamentally lacks the expressiveness to process the kind of data that LLMs can. That’s why you’re using an LLM in the first place.

qayxc 9mo ago
How does this relate to the need for deterministic and consistent output?

noreplydev 9mo ago

this page speaks about the same with some different approach https://selfvm.run/research https://selfvm.run/research