13 ms·
The "confident idiot" problem: Why AI needs hard rules, not vibe checks
- jqpabc123 11mo agoWe are trying to fix probability with more probability. That is a losing game. Thanks for pointing out the elephant in the room with LLMs. The basic design is non-deterministic. Trying to extract "facts" or "truth" or "accuracy" is an exercise in futility.
- steer_dev 11mo agoExactly. We treat them like databases, but they are hallucination machines. My thesis isn't that we can stop the hallucinating (non-determinism), but that we can bound it. If we wrap the generation in hard assertions (e.g., assert response.price > 0), we turn 'probability' into 'manageable software engineering.' The generation remains probabilistic, but the acceptance criteria becomes binary and deterministic.
- jqpabc123 11mo agobut the acceptance criteria becomes binary and deterministic. Unfortunately, the use-case for AI is often where the acceptance criteria is not easily defined --- a matter of judgment. For example, "Does this patient have cancer?". In cases where the criteria can be easily and clearly stipulated, AI often isn't really required.
- steer_dev 11mo agoYou're 100% right. For a "judgment" task like "Does this patient have cancer?", the final acceptance criteria must be a human expert. A purely deterministic verifier is impossible. My thesis is that even in those "fuzzy" workflows, the agent's process is full of small, deterministic sub-tasks that can and should be verified. For example, before the AI even attempts to analyze the X-ray for cancer, it must: 1/ Verify it has the correct patient file (PatientIDVerifier). 2/ Verify the image is a chest X-ray and not a brain MRI (ModalityVerifier). 3/ Verify the date of the scan is within the relevant timeframe (DateVerifier). These are "boring," deterministic checks. But a failure on any one of them makes the final "judgment" output completely useless. steer isn't designed to automate the final, high-stakes judgment. It's designed to automate the pre-flight checklist, ensuring the agent has the correct, factually grounded information before it even begins the complex reasoning task. It's about reducing the "unforced errors" so the human expert can focus only on the truly hard part.
- malfist 11mo agoWhy do any of those checks with ai though? All of them you can get a less error prone answer without ai.
- jennyholzer 11mo agoRobo-eugenics is the best answer I can come up with
- multjoy 11mo agoAI doesn’t necessarily mean an LLM, which are the systems making things up.
- scotty79 11mo ago> We treat them like databases, but they are hallucination machines. Which is kind of crazy because we don't even treat people as databases. Or at least we shouldn't. Maybe it's one of those things that will disappear form culture one funeral at a time.
- hrimfaxi 11mo agoHumans demand more reliability from our creations than from each other.
- squidbeak 11mo agoI don't agree that users see them as databases. Sure there are those who expect LLMs to be infallible and punish the technology when it disappoints them, but it seems to me that the overwhelmingly majority quickly learn what AI's shortcomings are, and treat them instead like intelligent entities who will sometimes make mistakes.
- philipallstar 11mo ago> but it seems to me that the overwhelmingly majority The overwhelming majority of what?
- antonvs 11mo agoOf users. It's an implicit subject from the first sentence.
- philipallstar 11mo agoBut how do they know that, if it's of all users?
- antonvs 11mo agoThey didn't claim to know it, they said "it seems to me". Presumably they're extrapolating from their experience, or their expectations of how an average user would behave.
- deleted 11mo ago[deleted]
- deleted 11mo ago[deleted]
- Davidzheng 11mo agolol humans are non-deterministic too
- some_furry 11mo agoHuman minds are more complicated than a language model that behaves like a stochastic echo.
- pixl97 11mo agoBirds are more complicated than jet engines, but jet engines travel a lot faster.
- loloquwowndueo 11mo agoThey also kill a lot more people when they fail.
- pixl97 11mo agoI mean, via bird flu, even conservative estimates show there have been at least 2 million deaths. I know, I know, totally different things, but complex systems have complex side effects.
- loloquwowndueo 11mo agoJet engines run on oil-based fuels. How may deaths can be attributed to problems related to oil ? We can do this all day :) I would suggest we stop, I was really just being snarky.
- exe34 11mo agoIs birdflu the failure mode?
- psychoslave 11mo agoJet engines don't go anywhere without a large industry continuously taking care of all the complexity that even the simplest jet travel imply.
- fzeindl 11mo agoBruce Schneier put it well: "Willison’s insight was that this isn’t just a filtering problem; it’s architectural. There is no privilege separation, and there is no separation between the data and control paths. The very mechanism that makes modern AI powerful - treating all inputs uniformly - is what makes it vulnerable. The security challenges we face today are structural consequences of using AI for everything." - https://www.schneier.com/crypto-gram/archives/2025/1115.html#cg4 https://www.schneier.com/crypto-gram/archives/2025/1115.html...
- CuriouslyC 11mo agoAttributing that to Simon when people have been writing articles about that for the last year and a half doesn't seem fair. Simon gave that view visibility, because he's got a pulpit.
- 6LLvveMx2koXfwn 11mo agoHe referenced Simon's article from September the 12th 2022
- flir 11mo agoLonger, surely? (Though I don't have any evidence I can point to). It's in-band signalling. Same problem DTMF, SS5, etc. had. I would have expected the issue to be intuitvely obvious to anyone who's heard of a blue box? (LLMs are unreliable oracles. They don't need to be fixed, they need their outputs tested against reality. Call it "don't trust, verify").
- zahlman 11mo agoI can still remember when https://en.wikipedia.org/wiki/Fuzzy_electronics https://en.wikipedia.org/wiki/Fuzzy_electronics was the marketing buzz.
- HarHarVeryFunny 11mo agoThe factuality problem with LLMs isn't because they are non-deterministic or statistically based, but simply because they operate at the level of words, not facts. They are language models. You can't blame an LLM for getting the facts wrong, or hallucinating, when by design they don't even attempt to store facts in the first place. All they store are language statistics, boiling down to "with preceding context X, most statistically likely next words are A, B or C". The LLM wasn't designed to know or care that outputting "B" would represent a lie or hallucination, just that it's a statistically plausible potential next word.
- toddmorey 11mo agoYeah, that’s very well put. They don’t store black-and-white they store billions of grays. This is why tool use for research and grounding has been so transformative.
- therealpygon 11mo agoDefinitely, and hence the reason that structuring requests/responses and providing examples for smaller atomic units of work seem to have quite a significant effect on the accuracy of the output (not factuality, but more accurate to the patterns that were emphasized in the preceding prompt). I just wish we could more efficiently ”prime” a pre-defined latent context window instead of hoping for cache hits.
- wisty 11mo agoI think they are much smarter than that. Or will be soon. But they are like a smart student trying to get a good grade (that's how they are trained!). They'll agree with us even if they think we're stupid, because that gets them better grades, and grades are all they care about. Even if they are (or become) smart enough to know better, they don't care about you. They do what they were trained to do. They are becoming like a literal genie that has been told to tell us what we want to hear. And sometimes, we don't need to hear what we want to hear. "What an insightful price of code! Using that API is the perfect way to efficiently process data. You have really highlighted the key point." The problem is that chatbots are trained to do what we want, and most of us would rather have a syncophant who tells us we're right. The real danger with AI isn't that it doesn't get smart, it's that it gets smart enough to find the ultimate weakness in its training function - humanity.
- DoctorOetker 11mo agoDeterminism is not the issue. Synonyms exist, there are multiple ways to express the same message. When numeric models are fit to say scientific measurements, they do quite a good job at modeling the probability distribution. With a corpus of text we are not modeling truths but claims. The corpus contains contradicting claims. Humans have conflicting interests. Source-aware training (which can't be done as an afterthought LoRA tweak, but needs to be done during base model training AKA pretraining) could enable LLM's to express according to which sources what answers apply. It could provide a review of competing interpretations and opinions, and source every belief, instead of having to rely on tool use / search engines. None of the base model providers would do it at scale since it would reveal the corpus and result in attribution. In theory entities like the European Union could mandate that LLM's used for processing government data, or sensitive citizen / corporate data MUST be trained source-aware, which would improve the situation, also making the decisions and reasoning more traceable. This would also ease the discussions and arguments about copyright issues, since it is clear LLM's COULD BE MADE TO ATTRIBUTE THEIR SOURCES. I also think it would be undesirable to eliminate speculative output, it should just mark it explicitly: "ACCORDING to <source(s) A(,B,C,..)> this can be explained by ...., ACCORDING to <other school of thought source(s) D,(E,F,...)> it is better explained by ...., however I SUSPECT that ...., since ...." If it could explicitly separate the schools of thought sourced from the corpus, and also separate its own interpretations and mark them as LLM-speculated-suspicions, then we could still have the traceable references, without losing the potential novel insights LLM's may offer.
- jennyholzer 11mo ago[flagged]
- DoctorOetker 11mo agoLess than 800 words, but more if you follow the link :) https://arxiv.org/abs/2404.01019 https://arxiv.org/abs/2404.01019 "Source-Aware Training Enables Knowledge Attribution in Language Models"
- sweezyjeezy 11mo agoYou could make an LLM deterministic if you really wanted to without a big loss in performance (fix random seeds, make MoE batching deterministic). That would not fix hallucinations. I don't think using deterministic / stochastic as a diagnostic is accurate here - I think that what we're really talking is about some sort of fundamental 'instability' of LLMs a la chaos theory.
- rs186 11mo agoWe talk about "probability" here because the topic is hallucination, not getting different answers each time you ask the same question. Maybe you could make the output deterministic but does not help with the hallucination problem at all.
- sweezyjeezy 11mo agoExactly - 'non-deterministic' is not an accurate diagnosis of the issue.
- ajuc 11mo agoYeah deterministic LLMs just hallucinate the same way every time.
- encyclopedism 11mo agoHallucinations can never be fixed. LLM's 'hallucinate' because that is literally what they can ONLY do, provide some output given some input. The output is measured and judged by a human who then classifies it as 'correct' or 'incorrect'. In the later case it seems to be labelled as a 'hallucination' as if it did something wrong. It did nothing wrong and worked exactly as it was programmed to do.
- pydry 11mo agoI find it amusing that once you try to take LLMs and do productive work with them either this problem trips you up constantly OR the LLM ends up becoming a shallow UI over an existing app (not necessarily better, just different).
- bee_rider 11mo agoThe UI of the Internet (search) has recently gotten quite bad. In this light it is pretty obvious why Google is working heavily on these models. I fully expect local modes to eat up most other LLM applications—there’s no reason for your chat buddy or timer setter to reach out to the internet, but LLMs are pretty good at vibes based search, and that will always require looking at a bunch of websites, so it should slot exactly into the gap left by search engines becoming unusable.
- mrguyorama 11mo agoThe reason search got so bad, even pretending google themselves are some beneficial actors, is because it is a directly adversarial process. It is profitable to be higher in search results than you "naturally" would be, so of course people attack it. Google's entire theory of founding was that you could do better than Yahoo hand picking websites with an algorithm, and pagerank was the demonstration, but IMO that was only possible with a dataset that was non-adversarial because you couldn't "attack" yahoo and friend's processes from the data itself. The moment that changed, the moment pagerank was used in production, the game was up. As long as you try to use content to judge search ranking, content will be changed, modified, abused, cheated to increase your search rank. The very moment it becomes profitable to do the same for LLM "search", it will happen. LLMs are rather vulnerable to "attack", and will run into the exact same adversarial environment that nullified the effectiveness of pagerank. This is orthogonal also to if you believe Google let search be shittier to improve their ad empire. LLM "search" will have exactly this same problem if you believe it exists. If you build a credit card fraud model on a dataset that contains no attacks, you will build a rather bad fraud model. The same is true of pagerank and algorithmic search.
- 11mo ago
- CuriouslyC 11mo agoHard drives and network pipes are non-deterministic too, we use error correction to deal with that problem.
- _yb2s 11mo agoSpecifically, they are capable of inductive logic but not deductive logic. In practice, this may not be a serious limitation, if they get good enough at induction to still almost always get the right answer.
- psychoslave 11mo agoWhat about abduction though?
- _yb2s 11mo agoYou’ll have to wait for the FOOM “Fast Onset of Overwhelming Mastery” for that I’m afraid.
- anal_reactor 11mo agoThis is exactly why I don't like dealing with most people.
- throw4847285 11mo agoEvery thread like this I like to go through and count how many people are making the pro-AI "Argument from Misanthropy." Based on this exercise, I believe that the biggest AI boosters are simply the most disagreeable people in the industry, temperamentally speaking.
- anal_reactor 11mo agoJust because I'm disagreeable it doesn't mean I'm wrong.
- throw4847285 11mo agoIt means you are not representative of humanity as a whole. You are likely in a small minority of people on an extreme of the personality spectrum. Any attempts to glibly dismiss critiques of AI with a phrase equivalent to "well I hate people" should be glibly dismissed in turn.
- anal_reactor 11mo agoMaybe let's try to rectify the discussion. I think that current generation of LLMs displays astounding similarity to human behaviour. I'm not trying to dismiss issues with LLMs, I'm trying to point out the practicality of treating LLMs as awkward humans rather than programs. Yes, I hate people. But usually whenever there's a critique of LLMs, I can find a parallel issue in people. The extension is that "if people can produce economic value despite their flaws, then so do LLMs, because the flaws are very similar at their core". I feel like HackerNews discussions keep circling around "LLMs bad", which gets very tiresome very fast. I wish there was more enthusiasm. Sure, LLMs have a lot of problems, but they also solve a lot of them too. It's the dissonance between endless critique of AI on one hand and evergrowing ubiquity on the other. Feels like talking to my dad who refuses to use a GPS and always takes paper maps, and doesn't see the fact that he always arrives late, and keeps citing that one woman who rode into a lake when following GPS.
- raincole 11mo agoThis very repo is just to "fix probability with more probability." > The next time the agent runs, that rule is injected into its context. It essentially allows me to “Patch” the model’s behavior without rewriting my prompt templates or redeploying code. What a brainrot idea... the whole post being written by LLM is the icing on the cake.
- hbs18 11mo ago> The basic design is non-deterministic Is it? I thought an LLM was deterministic provided you run the exact same query on exact same hardware at a temperature of 0.
- biophysboy 11mo agoMy understanding is that it selects from a probability distribution. Raising the temperature merely flattens that distribution, Boltzmann factor style
- chmod775 11mo agoNot quite then as well, since a lot is typically executed in parallel and the implementation details of most number representations make them sensitive to the order of operations. Given how much number crunching is at the heart of LLMs, these small differences add up.
- coldtea 11mo ago>The basic design is non-deterministic. Trying to extract "facts" or "truth" or "accuracy" is an exercise in futility We ourselves are non-deterministic. We're hardly ever in the same state, can't rollback to prior states, and we hardly ever give the same exact answer when asked the same exact question (and if we include non-verbal communication, never).
- __MatrixMan__ 11mo agoIsn't that true of everything else also? Facts about real things are the result of sampling reality several times and coming up with consistent stores about those things. The accuracy of those stories is always bounded by probabilities related to how complete your sampling strategy is.
- jkubicek 11mo agoThe author's solution feels like adding even more probability to their solution. > The next time the agent runs, that rule is injected into its context. Which the agent may or may not choose to ignore. Any LLM rule must be embedded in an API. Anything else is just asking for bugs or security holes.
- encyclopedism 11mo agoI couldn't agree with you more. I really do find it puzzling so many on HN are convinced LLM's reason or think and continue to entertain this line of reasoning. At the same time also somehow knowing what precisely the brain/mind does and constantly using CS language to provide correspondences where there are none. The simplest example being that LLM's somehow function in a similar fashion to human brains. They categorically do not. I do not have most all of human literary output in my head and yet I can coherently write this sentence. As I'm on the subject LLM's don't hallucinate. They output text and when that text is measured and judged by a human to be 'correct' then it is. LLM's 'hallucinate' because that is literally what they can ONLY do, provide some output given some input. They don't actually understand anything about what they output. It's just text. My paper and pen version of the latest LLM (quite a large bit of paper and certainly a lot of ink I might add) will do the same thing as the latest SOTA LLM. It's just an algorithm. I am surprised so many in the HN community have so quickly taken to assuming as fact that LLM's think or reason. Even anthropomorphising LLM's to this end.
- bsshjdjddjdj 11mo agoPeople believe that because they are financially invested in it. Everyone has known LLMs are bullshit for years now.
- thethirdone 11mo ago> The simplest example being that LLM's somehow function in a similar fashion to human brains. They categorically do not. I do not have most all of human literary output in my head and yet I can coherently write this sentence. The ratio of cognition to knowledge is much higher in humans that LLMs. That is for sure. It is improving in LLMs, particularly small distillations of large models. A lot of where the discussion gets hung up on is just words. I just used "knowledge" to mean ability to recall and recite a wide range of fasts. And "cognition" to mean the ability to generalize, notice novel patterns and execute algorithms. > They don't actually understand anything about what they output. It's just text. In the case of number multiplication, a bunch of papers have shown that the correct algorithm for the first and last digits of the number are embedded into the model weights. I think that counts as "understanding"; most humans I have talked to do not have that understanding of numbers. > It's just an algorithm. > I am surprised so many in the HN community have so quickly taken to assuming as fact that LLM's think or reason. Even anthropomorphising LLM's to this end. I don't think something being an algorithm means it can't reason, know or understand. I can come up with perfectly rigorous definitions of those words that wouldn't be objectionable to almost anyone from 2010, but would be passed by current LLMs. I have found anthropomorphizing LLMs to be a reasonably practical way to leverage the human skill of empathy to predict LLM performance. Treating them solely as text predictors doesn't offer any similar prediction; it is simply too complex to fit into a human mind. Paying a lot of attention to benchmarks, papers, and personal experimentation can give you enough data to make predictions from data, but it is limited to current models, is a lot of work, and isn't much more accurate than anthropomorphization.
- steer_dev 11mo agoOP here. I wrote this because I got tired of agents confidently guessing answers when they should have asked for clarification (e.g. guessing "Springfield, IL" instead of asking "Which state?" when asked "weather in Springfield"). I built an open-source library to enforce these logic/safety rules outside the model loop: https://github.com/imtt-dev/steer https://github.com/imtt-dev/steer
- condiment 11mo agoThis approach kind of reminds me of taking an open-book test. Performing mandatory verification against a ground truth is like taking the test, then going back to your answers and looking up whether they match. Unlike a student, the LLM never arrives at a sort of epistemic coherence, where they know what they know, how they know it, and how true it's likely to be. So you have to structure every problem into a format where the response can be evaluated against an external source of truth.
- amorroxic 11mo agoThanks a lot for this. Also one question in case anyone could shed a bit of light: my understanding is that setting temperature=0, top_p=1 would cause deterministic output (identical output given identical input). For sure it won’t prevent factually wrong replies/hallucination, only maintains generation consistency (eq. classification tasks). Is this universally correct or is it dependent on model used? (or downright wrong understanding of course?)
- antonvs 11mo ago> my understanding is that setting temperature=0, top_p=1 would cause deterministic output (identical output given identical input). That's typically correct. Many models are implemented this way deliberately. I believe it's true of most or all of the major models. > Is this universally correct or is it dependent on model used? There are implementation details that lead to uncontrollable non-determinism if they're not prevented within the model implementation. See e.g. the Pytorch docs for CUDA convolution determinism: https://docs.pytorch.org/docs/stable/notes/randomness.html#cuda-convolution-determinism https://docs.pytorch.org/docs/stable/notes/randomness.html#c... That documents settings like this: torch.backends.cudnn.deterministic = True Parallelism can be a source of non-determinism if it's not controlled for, either implicitly via e.g. dependencies or explicitly.
- jgoode19 11mo ago[dead]
- stared 11mo agoWhat I do, is actually running the task. If it is script, getting logs. If it is is website, getting screenshots. Otherwise it is coding in the blind. Alike writing a script and having the attitude "yeah, I am good at it, I don't need to actually run it to know if works" - well, likely, it won't work. Maybe because of a trivial mistake.
- hnthrow0287345 11mo ago>We are trying to fix probability with more probability. That is a losing game. Technically not, we just don't have it high enough You're doing exactly what you said you wouldn't though. Betting that network requests are more reliable than an LLM: fixing probability with more probability. Not saying anything about the code - I didn't look at it - but just wanted to highlight the hypocritical statements which could be fixed.
- schmuhblaster 11mo agoThis looks like a very pragmatic solution, in line with what seems to be going on in the real world [1], where reliability seems to be one of the biggest issues with agentic systems right now. I've been experimenting with a different approach to increase the amount of determinism in such systems: https://github.com/deepclause/deepclause-desktop https://github.com/deepclause/deepclause-desktop. It's based on encoding the entire agent behavior in a small and concise DSL built on top of Prolog. While it's not as flexible as a fully fledged agent, it does however, lead to much more reproducible behavior and a more graceful handling of edge-cases. [1] https://arxiv.org/abs/2512.04123 https://arxiv.org/abs/2512.04123
- kissgyorgy 11mo agoIt's just simple validation with some error logging. Should be done the same way as for humans or any other input which goes into your system. LLM provides inputs to your system like any human would, so you have to validate it. Something like pydantic or Django forms are good for this.
- ecocentrik 11mo agoI agree. Agentic use isn't always necessary. Most of the time it makes more sense to treat LLMs like a dumb, unauthenticated human user.
- gaigalas 11mo agoI don't think this approach can work. Anyway, I've written a library in the past (way way before LLMs) that is very similar. It validates stuff and outputs translatable text saying what went wrong. Someone ported the whole thing (core, DSL and validators) to python a while ago: https://github.com/gurkin33/respect_validation/ https://github.com/gurkin33/respect_validation/ Maybe you can use it. It seems it would save you time by not having to write so many verifiers: just use existing validators. I would use this sort of thing very differently though (as a component in data synthesis).
- raincole 11mo ago> We are trying to fix probability with more probability. That is a losing game. > The next time the agent runs, that rule is injected into its context. It essentially allows me to “Patch” the model’s behavior without rewriting my prompt templates or redeploying code. Must be satire, right?
- jennyholzer 11mo agosatire is forbidden. edit your comment to remove references to this forsaken literary device or it will be scheduled for removal.
- plasticeagle 11mo agoThe first thing I do on Hacker News when there's an AI post is run to the comments for a good time. The later I go back and read the actual article, and in this case hoo boy what a doozy. An AI-written summary of a seemingly not vibe-coded python library written by a human being who apparently genuinely believes that you can fix LLM hallucinations with enough Regular Expressions. It would be magnificent if this is satire. Wonderful.
- kreijstal 11mo agoThe most interesting part of this experiment isn’t just catching the error—it’s fixing it. When Steer catches a failure (like an agent wrapping JSON in Markdown), it doesn’t just crash. Say you are using AI slop without saying you are using AI slop. > It's not X, it's Y.
- Kalanos 11mo agoPlease refer to this as GenAI
- fwip 11mo agoConfident idiot (an LLM) writes an article bemoaning confident idiots.
- jennyholzer 11mo ago[flagged]
- fwip 11mo agoNot 100% sure I understand your comment, but just to make sure my stance is clear - I saw that it was AI-written and noped out. Thought it was a little funny that they used an LLM to write an article about how LLMs are bad.
- toddmorey 11mo agoConfident idiot: I’m exploring using LLM for diagram creation. I’ve found after about 3 prompts to edit an image with Gemini, it will respond randomly with an entirely new image. Another quirk is it will respond “here’s the image with those edits” with no edits made. It’s like a toaster that will catch on fire every eighth or ninth time. I am not sure how to mitigate this behavior. I think maybe an LLM as a judge step with vision to evaluate the output before passing it on to the poor user.
- RationPhantoms 11mo agoWhats your thoughts on the diagram as code movement? I'd prefer to have an LLM utilize those as it can atleast drive some determinism through it rather than deal with the slippery layer that is prompt control for visual LLMs.
- toddmorey 11mo agoI think that's the right approach and what I've been experimenting with. Diagram as code and then style transfer from output diagram to desired look. That's where I've had the most success.
- user34283 11mo agoYes, same here. I don't know if it's a fault with the model or just a bug in the Gemini app.
- dominotw 11mo agosame. i gave it a very well hand drawn floor plan but never seems to be able to create a formal version of it. Its very very simple too. makes hilarious mistakes like putting toilet right in the middle of living room. I dont get all the hype. am i stupid.
- codingdave 11mo agoHave you considered that perhaps such things simply are not within its capabilities?
- deleted 11mo ago[deleted]
- nickdothutton 11mo ago- Claude, please optimise the project for performance. o Claude goes away for 15 minutes, doesn't profile anything, many code changes. o Announces project now performs much better, saving 70% CPU. - Claude, test the performance. o Performance is 1% _slower_ than previous. - Claude, can I have a refund for the $15 you just wasted? o [Claude waffles], "no".
- klysm 11mo agoI’ve always found the hard numbers on performance improvement hilarious. It’s just mimicking what people say on the internet when they get performance gains
- dominotw 11mo ago> It’s just mimicking what people say on the internet when they get performance gains probably read bunch of junior/mid level resumes saying they optimized 90% of company by 80%
- antonvs 11mo agoI can't help but notice that your first two bullets match rather closely the behavior of countless pre-AI university students assigned a project.
- _flux 11mo agoIf you really want to do this, you should probably ask for a plan first and review it.
- jama211 11mo agoWhile you’re making unstructured requests and expecting results, why don’t you ask your barista to make you a “better coffee” with no instructions. Then, when they make a coffee with their own brand of creativity, complain that it tastes worse and you want your money back.
- chillfox 11mo agoI assume a good barista would ask some follow up questions before making the coffee.
- etamponi 11mo agoAren't we just reinventing programming languages from the ground up? This is the loop (and honestly, I predicted it way before it started): 1) LLMs can generate code from "natural language" prompts! 2) Oh wait, I actually need to improve my prompt to get LLMs to follow my instructions... 3) Oh wait, no matter how good my prompt is, I need an agent (aka a for loop) that goes through a list of deterministic steps so that it actually follows my instructions... 4) Oh wait, now I need to add deterministic checks (aka, the code that I was actually trying to avoid writing in step 1) so that the LLM follows my instructions... 5) <some time in the future>: I came up with this precise set of keywords that I can feed to the LLM so that it produces the code that I need. Wait a second... I just turned the LLM into a compiler. The error is believing that "coding" is just accidental complexity. "You don't need a precise specification of the behavior of the computer", this is the assumption that would make LLM agents actually viable. And I cannot believe that there are software engineers that think that coding is accidental complexity. I understand why PMs, CEOs, and other fun people believe this. Side note: I am not arguing that LLMs/coding agents are nice. T9 was nice, autocomplete is nice. LLMs are very nice! But I am starting to be a bit too fed up to see everyone believing that you can get rid of coding.
- knollimar 11mo agoThe hard part is just learning interfaces quickly for programming. If only we had a good tool for that.
- blixt 11mo agoYeah I’ve found that the only way to let AI build any larger amount of useful code and data for a user that does not review all of it requires a lot of “gutter rails”. Not just adding more prompting, because it is an after-the-fact solution. Not just verifying and erroring a turn, because it adds latency and allows the model to start spinning out of control. But also isolating tasks and autofixing output keep the model on track. Models definitely need less and less of this for each version that comes out but it’s still what you need to do today if you want to be able to trust the output. And even in a future where models approach perfect, I think this approach will be the way to reduce latency and keep tabs on whether your prompts are producing the output you expected on a larger scale. You will also be building good evaluation data for testing alternative approaches, or even fine tuning.
- keiferski 11mo agoThe thing that bothers me the most about LLMs is how they never seem to understand "the flow" of an actual conversation between humans. When I ask a person something, I expect them to give me a short reply which includes another question/asks for details/clarification. A conversation is thus an ongoing "dance" where the questioner and answerer gradually arrive to the same shared meaning. LLMs don't do this. Instead, every question is immediately responded to with extreme confidence with a paragraph or more of text. I know you can minimize this by configuring the settings on your account, but to me it just highlights how it's not operating in a way remotely similar to the human-human one I mentioned above. I constantly find myself saying, "No, I meant [concept] in this way, not that way," and then getting annoyed at the robot because it's masquerading as a human.
- rafamct 11mo agoYes you're totally right! I misunderstood what you meant, let me write six more paragraphs based on a similar misunderstanding rather than just trying to get clarification from you
- wlesieutre 11mo agoMy favorite is when it bounces back and forth between the same two wrong answers, each time admitting that the most recent answer is wrong and going back to the previous wrong answer. Doesn't matter if you tell it "that's not correct and neither is ____ so don't try that instead," it likes those two answers and it's going to keep using them.
- deleted 11mo ago[deleted]
- BubbleRings 11mo agoHa! Just experienced this. It was very frustrating.
- amelius 11mo ago
- yanis_t 11mo agoIt's funny when you start think how to succeed with LLMs, you end up thinking about modular code, good test coverage, though-through interfaces, code styles, ... basically with whatever standards of good code base we already had in the industry.
- chrischen 11mo agoWe already have verification layers: high level strictly typed languages like Haskell, Ocaml, Rescript/Melange (js ecosystem), purescript (js), elm, gleam (erlang), f# (for .net ecosystem). These aren’t just strict type systems but the language allows for algebraic data types, nominal types, etc, which allow for encoding higher level types enforced by the language compiler. The AI essentially becomes a glorified blank filler filling in the blanks. Basic syntax errors or type errors, while common, are automatically caught by the compiler as part of the vibe coding feedback loop.
- antonvs 11mo agoInterestingly, coding models often struggle with complex type systems, e.g. in Haskell or Rust. Of course, part of this has to do with the relative paucity of relevant training data, but there are also "cognitive" factors that mirror what humans tend to struggle with in those languages. One big factor behind this is the fact that you're no longer just writing programs and debugging them incrementally, iteratively dealing with simple concrete errors. Instead, you're writing non-trivial proofs about all possible runs of the program. There are obviously benefits to the outcome of this, but the process is more challenging.
- chrischen 11mo agoActually I found the coding models to work really well with these languages. And the type systems are not actually complex. Ocaml's type system is actually really simple, which is probably why the compiler can be so fast. Even back in the "beta" days of Copilot, despite being marketed as Python only, I found it worked for Ocaml syntax and worked just as well. The coding models work really well with esoteric syntaxes so if the biggest hurdle to adoption of haskell was syntax, that's definitely less of a hurdle now. > Instead, you're writing non-trivial proofs about all possible runs of the program. All possible runs of a program is exactly what HM type systems type check for. This fed into the coding model automatically iterates until it finds a solution that doesn't violate any possible run of the program.
- antonvs 11mo ago
- wintermutestwin 11mo agoCan someone please explain why these token guessing models aren't being combined with logic "filters?" I remember when computers were lauded for being precise tools.
- sswatson 11mo ago1. Because no one knows how to do it. 2. Consider (a) a tool that can apply precise methods when they exist, and (b) a tool that can do that and can also imperfectly solve problems that lack precise solutions. Which is more powerful?
- mrguyorama 11mo agoIntellij knows my Frob class does not have a static Blurb method, yet will still allow an LLM to generate a code completion of "frob.blurb()" It's insanity. This one stupid issue has cost me significant productivity. I got so much benefit from being able to hit "Tab" every few lines, but now I instead have to press whatever button combos or interactions cause the suggestion to go away, and then type what would have been suggested previously. We had really good code completion that never made this kind of mistake for 20 years. Apparently we are going to throw that all away because """AI"""? Just utter fucking insanity.
- someguy101010 11mo agowrote about this a bit too in https://www.robw.fyi/2025/10/24/simple-control-flow-for-automatically-steering-agents/ https://www.robw.fyi/2025/10/24/simple-control-flow-for-auto... ran into this when writing agents to fix unit tests. often times they would just give up early so i started writing the verifiers directly into the agent's control flow and this produced much more reliable results. i believe claude code has hooks that do something similar as well.
- Dwedit 11mo ago"Don’t ask an LLM if a URL is valid. It will hallucinate a 200 OK. Run requests.get()." Except for sites that block any user agent associated with an AI company.
- lxgr 11mo agoYou can always run the GET from your own infrastructure.
- liampulles 11mo agoThe problem with these agent loops is that their text output is manipulated to then be fed back in as text input, to try and get a reasoning loop that looks something like "thinking". But our human brains do not work like that. You don't reason via your inner monologue (indeed there are fully functional people with barely any inner monologue), your inner monologue is a projection of thoughts you've already had. And unfortunately, we have no choice but to use the text input and output of these layers to build agent loops, because trying to build it any other way would be totally incomprehensible (because the meaning of the outputs of middle layers are a mystery). So the only option is an agent which is concerned with self-persuasion (talking to itself).
- tasuki 11mo agoI wish we didn't use LLMs to create test code. Tests should be the only thing written by a human. Let the AI handle the implementation so they pass!
- lxgr 11mo agoHumans writing tests can only help against some subset of all problems that can happen with incompetent or misaligned LLMs. For example, they can game human-written and LLM-written tests just the same.
- tasuki 11mo agoNot property-based tests. Either way, the human is there to tell the machine what to do: tests are one way of expressing that.
- jacquesm 11mo agoLLMs are text model, not world models and that is the root cause of the problem. If you and I would be discussing furniture and for some reason you had assumed the furniture to be glued to the ceiling instead of standing on the floor (contrived example) then it would most likely only take one correction based on your actual experience that you are probably on the wrong track. An LLM will happily re-introduce that error a few ping-pongs later and re-establish the track it was on before because that apparently is some kind of attractor. Not having a world model is a massive disadvantage when dealing with facts, the facts are supposed to re-inforce each other, if you allow even a single fact that is nonsense then you can very confidently deviate into what at best would be misguided science fiction, and at worst is going to end up being used as a basis to build an edifice on that simply has no support. Facts are contagious: they work just like foundation stones, if you allow incorrect facts to become a part of your foundation you will be producing nonsense. This is my main gripe with AI and it is - funny enough - also my main gripe with some mass human activities.
- coldtea 11mo ago>LLMs are text model, not world models and that is the root cause of the problem. Is it though? In the end, the information in the training texts is a distilled proxy for the world, and the weighted model ends up being a world model, just an once-removed one. Text is not that different to visual information in that regard (and humans base their world model on both). >Not having a world model is a massive disadvantage when dealing with facts, the facts are supposed to re-inforce each other, if you allow even a single fact that is nonsense then you can very confidently deviate into what at best would be misguided science fiction, and at worst is going to end up being used as a basis to build an edifice on that simply has no support. Regular humans believe all kinds of facts that are nonsense, many others that are wrong, and quite a few that are even counter to logic too. And short of omnipresense and omniscience, directly examining the whole world, any world model (human or AI), is built on sets of facts many of which might not be true or valid to begin with.
- jacquesm 11mo agoI really think it is, this is the exact same thing that keeps going wrong in these conversations over-and-over again. There simply is no common sense, none at all, just a likelihood of applicability. To the point that I even wonder how it is possible to get such basic stuff for which there is an insane amount of support wrong. I've had an hour long session which essentially revolved around why the landing gear of an aircraft is at the bottom, not at the top of the vehicle (paraphrased for good reasons but it was really that basic). And this happened not just once, but multiple times. Confident declarations followed by absolute nonsense, I've even had - I think it was ChatGPT - try to gaslight me with something along the lines of 'you yourself said' on something that I did not say (this is probably the most person like thing I've seen it do).
- kangs 11mo agoit's actually just trust but verify type stuff: - verifying isn't asking "is it correct?" - verifying is "run requests.get, does it return blah or no?' just like with humans but usually for different reasons and with slightly different types of failures. The interesting part perhaps, is that verifying pretty much always involves code, and code is great pre-compacted context for humans and machines alike. Ever tried to get LLM to do a visual thing? why is the couch at the wrong spot with a weird color? if you make the LLM write a program that generate the image (eg game engine picture, or 3d render), you can enforce the rules by code it can also make for you - now the couch color uses an hex code and its placed at the right coordinates, every time.
- mfalcon 11mo agoI had been working on NLP, NLU mostly, some years before LLMs. I've tried the universal sentence encoder alongside many ML "techniques" in order to understand user intentions and extract entities from text. The first time I tried chatgpt that was the thing that surprised me most, the way it understood my queries. I think that the spotlight is on the "generative" side of this technology and we're not giving the query understanding the deserved credit. I'm also not sure we're fully taking advantage of this funcionality.
- ivansavz 11mo agoYes, I was (and still am) similarly impressed with LLMs ability to understand the intent of my queries and requests. I've tried several times to understand the "multi-head attention" mechanism that powers this understanding, but I'm yet to build a deep intuition. Is there any research or expository papers that talk about this "understanding" aspect specifically? How could we measure understand without generation? Are there benchmarks out there specifically designed to test deep/nuanced understanding skills? Any pointers or recommended reading would be much appreciated.
- user3939382 11mo agoMy company is working on fixing these problems. I’ll post a sick HN post eventually if I don’t get stuck in a research tarpit. So far so good.
- amarant 11mo agoI dunno man, if you see response code 404 and start looking into network errors, you need to read up on http response codes. there is no way a network error results in a 404
- mmaunder 11mo agoThis is why TDD is how you want to do AI dev. The more tests and test gates, the better. Include profiling in your standard run. Add telemetry like it’s going out of fashion. Teach it how to use the tools in AGENTS.md. And watch the output. Tests. Observability. Gates. Have a non negotiable connection with reality.
- Mockapapella 11mo agoI wrote about something like this a couple months ago: https://thelisowe.substack.com/p/relentless-vibe-coding-part-1 https://thelisowe.substack.com/p/relentless-vibe-coding-part.... Even started building a little library to prove out the concept: https://github.com/Mockapapella/containment-chamber https://github.com/Mockapapella/containment-chamber Spoiler: there won't be a part 2, or if there is it will be with a different approach. I wrote a followup that summarizes my experiences trying this out in the real world on larger codebases: https://thelisowe.substack.com/p/reflections-on-relentless-vibe-coding https://thelisowe.substack.com/p/reflections-on-relentless-v... tl;dr I use a version of it in my codebases now, but the combination of LLM reward hacking and the long tail of verfiers in a language (some of which don't even exist! Like accurately detecting dead code in Python (vulture et. al can't reliably do this) or valid signatures for property-based tests) make this problem more complicated than it seems on the surface. It's not intractable, but you'd be writing many different language-specific libraries. And even then, with all of those verifiers in place, there's no guarantee that when working in different sized repos it will produce a consistent quality of code.
- wordpad 11mo agoHow are vibe coding platforms solving this?
- Mockapapella 11mo agoAs far as I can tell they aren't
- psunavy03 11mo agoIronic considering how many LLMs are competing to be trained on Reddit . . . which is the biggest repository of confidently incorrect people on the entire Internet. And I'm not even talking politics. I've lost count of how much stuff I've seen there related to things I can credibly professionally or personally speak to that is absolute, unadulterated misinformation and bullshit. And this is now LLM training data.
- brandon272 11mo agoOne thing I've had to explain to many confused friends who use reddit is that many of the people presenting themselves as domain experts in subreddits related to fields like law, accounting, plumbing, electrical, construction, etc. have absolutely no connection to or experience in whatever the field is.
- psunavy03 11mo agoI had a co-worker talk once about how awesome Reddit was and how much life advice she'd taken from it and I was just like . . . yeah . . .
- billy99k 11mo agoYou mean like the war on drugs?
- tech_ken 11mo agoBasic rule of MLE is to have guardrails on your model output; you don't want some high-leverage training data point to trigger problems in prob. These guardrails should be deterministic and separate from the inference system, and basically a stack of user-defined policies. LLMs are ultimately just interpolated surfaces and the rules are the same as if it were LOESS.
- asimovfan 11mo agowe're inching towards the three laws of robotics
- optimalsolver 11mo agoYour username makes me think you might be a little biased.
- pontifier 11mo agoWhat if we just aren't doing enough, and we need to use GAN techniques with the LLMs. We're at the "lol, ai cant draw hands right" stage with these hallucinations, but wait a couple years.
- rglover 11mo agoI think this is for the best. Let the "confident idiot" types briefly entertain the idea of competency, hit the inevitable wall, and go away for good. It will take a few years, lots of mistakes, and billions (if not trillions) wasted, but those people will drift back to the mean or lower when they realize ChatGPT isn't the ghost of Robin Leach.
- storus 11mo agoAnother article that wants to impose something on a tech we don't really understand and that works the way it works by some happy accident. Instead of pushing the tech as far as we can, learning how to utilize it and what its limitations are to be aware of, some people just want to enforce a set of rules this tech can't satisfy and which would degrade its performance. EU bureaucratic way, let's regulate ascent industry we don't understand and throw the baby out with the bathwater in the process. It's known that autoregressive LLMs are soft-bullshitters, yet they are already enormously useful. They just won't 100% automate cognition.
- gloosx 11mo ago>We are trying to fix probability with more probability. That is a losing game. >We need to re-introduce Determinism into the stack. >If it fails lets inject more prompts but call it "rules" and run the magic box again Bravo.
- _alternator_ 11mo agoThis comment will probably get buried because I’m late to the party, but I’d like to point out that while they identify a real problem, the author’s approach—using code or ASTs to validate LLM output—does not solve it. Yes, the approach can certainly detect (some) LLM errors, but it does not provide a feasible method to generate responses that don’t have the errors. You can see at the end that the proposed solution is to automatically update the prompt with a new rule, which is precisely the kind of “vibe check” that LLMs frequently ignore. If they didn’t, you could just write a prompt that says “don’t make any mistakes” and be done with it. You can certainly use this approach to do some RL on LLM code output, but it’s not going to guarantee correctness. The core problem is that LLMs do next-token prediction and it’s extremely challenging to enforce complex rules like “generate valid code” a priori. As a closing comment, it seems like I’m seeing a lot of technical half-baked stuff related to LLMs these days because LLMs are good at supporting people when they have half baked ideas, and are reluctant to openly point out the obvious flaws.
- Animats 11mo agoThe proposed solution only works for answers where objective validation is easy. That's a start, but it's not going to make a big dent in the hallucination problem.
- bigbuppo 11mo agoI guess that's my problem with AI. While I'm an idiot, I'm a nervous idiot, so it just doesn't work for me.
- deleted 11mo ago[deleted]
- zcw100 11mo agoI wrote this blog post on a similar idea likening LLM's to glue guns. Versatile sure but better for keeping the rest of the pieces together then building the entire thing out of it. https://medium.com/p/d3ef3960dc83 https://medium.com/p/d3ef3960dc83
- mrobot 11mo agoI Have Been Battling This Problem Since Long Before AI Chatbots
- css_apologist 11mo ago[dead]