7 ms·
We are trying to fix probability with more probability. That is a losing game. Thanks for pointing out the elephant in the room with LLMs. The basic design is
by jqpabc123 11mo ago
We are trying to fix probability with more probability. That is a losing game.
Thanks for pointing out the elephant in the room with LLMs.
The basic design is non-deterministic. Trying to extract "facts" or "truth" or "accuracy" is an exercise in futility.
- steer_dev 11mo agoExactly. We treat them like databases, but they are hallucination machines. My thesis isn't that we can stop the hallucinating (non-determinism), but that we can bound it. If we wrap the generation in hard assertions (e.g., assert response.price > 0), we turn 'probability' into 'manageable software engineering.' The generation remains probabilistic, but the acceptance criteria becomes binary and deterministic.
- jqpabc123 11mo agobut the acceptance criteria becomes binary and deterministic. Unfortunately, the use-case for AI is often where the acceptance criteria is not easily defined --- a matter of judgment. For example, "Does this patient have cancer?". In cases where the criteria can be easily and clearly stipulated, AI often isn't really required.
- steer_dev 11mo agoYou're 100% right. For a "judgment" task like "Does this patient have cancer?", the final acceptance criteria must be a human expert. A purely deterministic verifier is impossible. My thesis is that even in those "fuzzy" workflows, the agent's process is full of small, deterministic sub-tasks that can and should be verified. For example, before the AI even attempts to analyze the X-ray for cancer, it must: 1/ Verify it has the correct patient file (PatientIDVerifier). 2/ Verify the image is a chest X-ray and not a brain MRI (ModalityVerifier). 3/ Verify the date of the scan is within the relevant timeframe (DateVerifier). These are "boring," deterministic checks. But a failure on any one of them makes the final "judgment" output completely useless. steer isn't designed to automate the final, high-stakes judgment. It's designed to automate the pre-flight checklist, ensuring the agent has the correct, factually grounded information before it even begins the complex reasoning task. It's about reducing the "unforced errors" so the human expert can focus only on the truly hard part.
- malfist 11mo agoWhy do any of those checks with ai though? All of them you can get a less error prone answer without ai.
- jennyholzer 11mo agoRobo-eugenics is the best answer I can come up with
- multjoy 11mo agoAI doesn’t necessarily mean an LLM, which are the systems making things up.
- scotty79 11mo ago> We treat them like databases, but they are hallucination machines. Which is kind of crazy because we don't even treat people as databases. Or at least we shouldn't. Maybe it's one of those things that will disappear form culture one funeral at a time.
- hrimfaxi 11mo agoHumans demand more reliability from our creations than from each other.
- squidbeak 11mo agoI don't agree that users see them as databases. Sure there are those who expect LLMs to be infallible and punish the technology when it disappoints them, but it seems to me that the overwhelmingly majority quickly learn what AI's shortcomings are, and treat them instead like intelligent entities who will sometimes make mistakes.
- philipallstar 11mo ago> but it seems to me that the overwhelmingly majority The overwhelming majority of what?
- antonvs 11mo agoOf users. It's an implicit subject from the first sentence.
- philipallstar 11mo agoBut how do they know that, if it's of all users?
- antonvs 11mo agoThey didn't claim to know it, they said "it seems to me". Presumably they're extrapolating from their experience, or their expectations of how an average user would behave.
- deleted 11mo ago[deleted]
- deleted 11mo ago[deleted]
- Davidzheng 11mo agolol humans are non-deterministic too
- some_furry 11mo agoHuman minds are more complicated than a language model that behaves like a stochastic echo.
- pixl97 11mo agoBirds are more complicated than jet engines, but jet engines travel a lot faster.
- loloquwowndueo 11mo agoThey also kill a lot more people when they fail.
- pixl97 11mo agoI mean, via bird flu, even conservative estimates show there have been at least 2 million deaths. I know, I know, totally different things, but complex systems have complex side effects.
- loloquwowndueo 11mo agoJet engines run on oil-based fuels. How may deaths can be attributed to problems related to oil ? We can do this all day :) I would suggest we stop, I was really just being snarky.
- exe34 11mo agoIs birdflu the failure mode?
- psychoslave 11mo agoJet engines don't go anywhere without a large industry continuously taking care of all the complexity that even the simplest jet travel imply.
- fzeindl 11mo agoBruce Schneier put it well: "Willison’s insight was that this isn’t just a filtering problem; it’s architectural. There is no privilege separation, and there is no separation between the data and control paths. The very mechanism that makes modern AI powerful - treating all inputs uniformly - is what makes it vulnerable. The security challenges we face today are structural consequences of using AI for everything." - https://www.schneier.com/crypto-gram/archives/2025/1115.html#cg4 https://www.schneier.com/crypto-gram/archives/2025/1115.html...
- CuriouslyC 11mo agoAttributing that to Simon when people have been writing articles about that for the last year and a half doesn't seem fair. Simon gave that view visibility, because he's got a pulpit.
- 6LLvveMx2koXfwn 11mo agoHe referenced Simon's article from September the 12th 2022
- flir 11mo agoLonger, surely? (Though I don't have any evidence I can point to). It's in-band signalling. Same problem DTMF, SS5, etc. had. I would have expected the issue to be intuitvely obvious to anyone who's heard of a blue box? (LLMs are unreliable oracles. They don't need to be fixed, they need their outputs tested against reality. Call it "don't trust, verify").
- zahlman 11mo agoI can still remember when https://en.wikipedia.org/wiki/Fuzzy_electronics https://en.wikipedia.org/wiki/Fuzzy_electronics was the marketing buzz.
- HarHarVeryFunny 11mo agoThe factuality problem with LLMs isn't because they are non-deterministic or statistically based, but simply because they operate at the level of words, not facts. They are language models. You can't blame an LLM for getting the facts wrong, or hallucinating, when by design they don't even attempt to store facts in the first place. All they store are language statistics, boiling down to "with preceding context X, most statistically likely next words are A, B or C". The LLM wasn't designed to know or care that outputting "B" would represent a lie or hallucination, just that it's a statistically plausible potential next word.
- toddmorey 11mo agoYeah, that’s very well put. They don’t store black-and-white they store billions of grays. This is why tool use for research and grounding has been so transformative.
- therealpygon 11mo agoDefinitely, and hence the reason that structuring requests/responses and providing examples for smaller atomic units of work seem to have quite a significant effect on the accuracy of the output (not factuality, but more accurate to the patterns that were emphasized in the preceding prompt). I just wish we could more efficiently ”prime” a pre-defined latent context window instead of hoping for cache hits.
- wisty 11mo agoI think they are much smarter than that. Or will be soon. But they are like a smart student trying to get a good grade (that's how they are trained!). They'll agree with us even if they think we're stupid, because that gets them better grades, and grades are all they care about. Even if they are (or become) smart enough to know better, they don't care about you. They do what they were trained to do. They are becoming like a literal genie that has been told to tell us what we want to hear. And sometimes, we don't need to hear what we want to hear. "What an insightful price of code! Using that API is the perfect way to efficiently process data. You have really highlighted the key point." The problem is that chatbots are trained to do what we want, and most of us would rather have a syncophant who tells us we're right. The real danger with AI isn't that it doesn't get smart, it's that it gets smart enough to find the ultimate weakness in its training function - humanity.
- DoctorOetker 11mo agoDeterminism is not the issue. Synonyms exist, there are multiple ways to express the same message. When numeric models are fit to say scientific measurements, they do quite a good job at modeling the probability distribution. With a corpus of text we are not modeling truths but claims. The corpus contains contradicting claims. Humans have conflicting interests. Source-aware training (which can't be done as an afterthought LoRA tweak, but needs to be done during base model training AKA pretraining) could enable LLM's to express according to which sources what answers apply. It could provide a review of competing interpretations and opinions, and source every belief, instead of having to rely on tool use / search engines. None of the base model providers would do it at scale since it would reveal the corpus and result in attribution. In theory entities like the European Union could mandate that LLM's used for processing government data, or sensitive citizen / corporate data MUST be trained source-aware, which would improve the situation, also making the decisions and reasoning more traceable. This would also ease the discussions and arguments about copyright issues, since it is clear LLM's COULD BE MADE TO ATTRIBUTE THEIR SOURCES. I also think it would be undesirable to eliminate speculative output, it should just mark it explicitly: "ACCORDING to <source(s) A(,B,C,..)> this can be explained by ...., ACCORDING to <other school of thought source(s) D,(E,F,...)> it is better explained by ...., however I SUSPECT that ...., since ...." If it could explicitly separate the schools of thought sourced from the corpus, and also separate its own interpretations and mark them as LLM-speculated-suspicions, then we could still have the traceable references, without losing the potential novel insights LLM's may offer.
- jennyholzer 11mo ago[flagged]
- DoctorOetker 11mo agoLess than 800 words, but more if you follow the link :) https://arxiv.org/abs/2404.01019 https://arxiv.org/abs/2404.01019 "Source-Aware Training Enables Knowledge Attribution in Language Models"
- sweezyjeezy 11mo agoYou could make an LLM deterministic if you really wanted to without a big loss in performance (fix random seeds, make MoE batching deterministic). That would not fix hallucinations. I don't think using deterministic / stochastic as a diagnostic is accurate here - I think that what we're really talking is about some sort of fundamental 'instability' of LLMs a la chaos theory.
- rs186 11mo agoWe talk about "probability" here because the topic is hallucination, not getting different answers each time you ask the same question. Maybe you could make the output deterministic but does not help with the hallucination problem at all.
- sweezyjeezy 11mo agoExactly - 'non-deterministic' is not an accurate diagnosis of the issue.
- ajuc 11mo agoYeah deterministic LLMs just hallucinate the same way every time.
- encyclopedism 11mo agoHallucinations can never be fixed. LLM's 'hallucinate' because that is literally what they can ONLY do, provide some output given some input. The output is measured and judged by a human who then classifies it as 'correct' or 'incorrect'. In the later case it seems to be labelled as a 'hallucination' as if it did something wrong. It did nothing wrong and worked exactly as it was programmed to do.
- pydry 11mo agoI find it amusing that once you try to take LLMs and do productive work with them either this problem trips you up constantly OR the LLM ends up becoming a shallow UI over an existing app (not necessarily better, just different).
- bee_rider 11mo agoThe UI of the Internet (search) has recently gotten quite bad. In this light it is pretty obvious why Google is working heavily on these models. I fully expect local modes to eat up most other LLM applications—there’s no reason for your chat buddy or timer setter to reach out to the internet, but LLMs are pretty good at vibes based search, and that will always require looking at a bunch of websites, so it should slot exactly into the gap left by search engines becoming unusable.
- mrguyorama 11mo agoThe reason search got so bad, even pretending google themselves are some beneficial actors, is because it is a directly adversarial process. It is profitable to be higher in search results than you "naturally" would be, so of course people attack it. Google's entire theory of founding was that you could do better than Yahoo hand picking websites with an algorithm, and pagerank was the demonstration, but IMO that was only possible with a dataset that was non-adversarial because you couldn't "attack" yahoo and friend's processes from the data itself. The moment that changed, the moment pagerank was used in production, the game was up. As long as you try to use content to judge search ranking, content will be changed, modified, abused, cheated to increase your search rank. The very moment it becomes profitable to do the same for LLM "search", it will happen. LLMs are rather vulnerable to "attack", and will run into the exact same adversarial environment that nullified the effectiveness of pagerank. This is orthogonal also to if you believe Google let search be shittier to improve their ad empire. LLM "search" will have exactly this same problem if you believe it exists. If you build a credit card fraud model on a dataset that contains no attacks, you will build a rather bad fraud model. The same is true of pagerank and algorithmic search.
- 11mo ago
- CuriouslyC 11mo agoHard drives and network pipes are non-deterministic too, we use error correction to deal with that problem.
- _yb2s 11mo agoSpecifically, they are capable of inductive logic but not deductive logic. In practice, this may not be a serious limitation, if they get good enough at induction to still almost always get the right answer.
- psychoslave 11mo agoWhat about abduction though?
- _yb2s 11mo agoYou’ll have to wait for the FOOM “Fast Onset of Overwhelming Mastery” for that I’m afraid.
- anal_reactor 11mo agoThis is exactly why I don't like dealing with most people.
- throw4847285 11mo agoEvery thread like this I like to go through and count how many people are making the pro-AI "Argument from Misanthropy." Based on this exercise, I believe that the biggest AI boosters are simply the most disagreeable people in the industry, temperamentally speaking.
- anal_reactor 11mo agoJust because I'm disagreeable it doesn't mean I'm wrong.
- throw4847285 11mo agoIt means you are not representative of humanity as a whole. You are likely in a small minority of people on an extreme of the personality spectrum. Any attempts to glibly dismiss critiques of AI with a phrase equivalent to "well I hate people" should be glibly dismissed in turn.
- anal_reactor 11mo agoMaybe let's try to rectify the discussion. I think that current generation of LLMs displays astounding similarity to human behaviour. I'm not trying to dismiss issues with LLMs, I'm trying to point out the practicality of treating LLMs as awkward humans rather than programs. Yes, I hate people. But usually whenever there's a critique of LLMs, I can find a parallel issue in people. The extension is that "if people can produce economic value despite their flaws, then so do LLMs, because the flaws are very similar at their core". I feel like HackerNews discussions keep circling around "LLMs bad", which gets very tiresome very fast. I wish there was more enthusiasm. Sure, LLMs have a lot of problems, but they also solve a lot of them too. It's the dissonance between endless critique of AI on one hand and evergrowing ubiquity on the other. Feels like talking to my dad who refuses to use a GPS and always takes paper maps, and doesn't see the fact that he always arrives late, and keeps citing that one woman who rode into a lake when following GPS.
- raincole 11mo agoThis very repo is just to "fix probability with more probability." > The next time the agent runs, that rule is injected into its context. It essentially allows me to “Patch” the model’s behavior without rewriting my prompt templates or redeploying code. What a brainrot idea... the whole post being written by LLM is the icing on the cake.
- hbs18 11mo ago> The basic design is non-deterministic Is it? I thought an LLM was deterministic provided you run the exact same query on exact same hardware at a temperature of 0.
- biophysboy 11mo agoMy understanding is that it selects from a probability distribution. Raising the temperature merely flattens that distribution, Boltzmann factor style
- chmod775 11mo agoNot quite then as well, since a lot is typically executed in parallel and the implementation details of most number representations make them sensitive to the order of operations. Given how much number crunching is at the heart of LLMs, these small differences add up.
- coldtea 11mo ago>The basic design is non-deterministic. Trying to extract "facts" or "truth" or "accuracy" is an exercise in futility We ourselves are non-deterministic. We're hardly ever in the same state, can't rollback to prior states, and we hardly ever give the same exact answer when asked the same exact question (and if we include non-verbal communication, never).
- __MatrixMan__ 11mo agoIsn't that true of everything else also? Facts about real things are the result of sampling reality several times and coming up with consistent stores about those things. The accuracy of those stories is always bounded by probabilities related to how complete your sampling strategy is.
- jkubicek 11mo agoThe author's solution feels like adding even more probability to their solution. > The next time the agent runs, that rule is injected into its context. Which the agent may or may not choose to ignore. Any LLM rule must be embedded in an API. Anything else is just asking for bugs or security holes.
- encyclopedism 11mo agoI couldn't agree with you more. I really do find it puzzling so many on HN are convinced LLM's reason or think and continue to entertain this line of reasoning. At the same time also somehow knowing what precisely the brain/mind does and constantly using CS language to provide correspondences where there are none. The simplest example being that LLM's somehow function in a similar fashion to human brains. They categorically do not. I do not have most all of human literary output in my head and yet I can coherently write this sentence. As I'm on the subject LLM's don't hallucinate. They output text and when that text is measured and judged by a human to be 'correct' then it is. LLM's 'hallucinate' because that is literally what they can ONLY do, provide some output given some input. They don't actually understand anything about what they output. It's just text. My paper and pen version of the latest LLM (quite a large bit of paper and certainly a lot of ink I might add) will do the same thing as the latest SOTA LLM. It's just an algorithm. I am surprised so many in the HN community have so quickly taken to assuming as fact that LLM's think or reason. Even anthropomorphising LLM's to this end.
- bsshjdjddjdj 11mo agoPeople believe that because they are financially invested in it. Everyone has known LLMs are bullshit for years now.
- thethirdone 11mo ago> The simplest example being that LLM's somehow function in a similar fashion to human brains. They categorically do not. I do not have most all of human literary output in my head and yet I can coherently write this sentence. The ratio of cognition to knowledge is much higher in humans that LLMs. That is for sure. It is improving in LLMs, particularly small distillations of large models. A lot of where the discussion gets hung up on is just words. I just used "knowledge" to mean ability to recall and recite a wide range of fasts. And "cognition" to mean the ability to generalize, notice novel patterns and execute algorithms. > They don't actually understand anything about what they output. It's just text. In the case of number multiplication, a bunch of papers have shown that the correct algorithm for the first and last digits of the number are embedded into the model weights. I think that counts as "understanding"; most humans I have talked to do not have that understanding of numbers. > It's just an algorithm. > I am surprised so many in the HN community have so quickly taken to assuming as fact that LLM's think or reason. Even anthropomorphising LLM's to this end. I don't think something being an algorithm means it can't reason, know or understand. I can come up with perfectly rigorous definitions of those words that wouldn't be objectionable to almost anyone from 2010, but would be passed by current LLMs. I have found anthropomorphizing LLMs to be a reasonably practical way to leverage the human skill of empathy to predict LLM performance. Treating them solely as text predictors doesn't offer any similar prediction; it is simply too complex to fit into a human mind. Paying a lot of attention to benchmarks, papers, and personal experimentation can give you enough data to make predictions from data, but it is limited to current models, is a lot of work, and isn't much more accurate than anthropomorphization.