22 ms·

Beliefs that are true for regular software but false when applied to AI

• kazinator 1y ago
  > AIs will get more reliable over time, like old software is more reliable than new software. :) Was that a humam Freudian slip, or artificial one? Yes, old software is often more reliable than new.
  • joomla199 1y ago
    Neither, you’re reading it wrong. Think of it as codebases getting more reliable over time as they accumulate fixes and tests. (As opposed to, say, writing code in NodeJS versus C++)
    • giancarlostoro 1y ago
      Age of Code does not automatically equal quality of code, ever. Good code is maintained by good developers. A lot of bad code is pushed out by management, and other situations, or just bad devs. This is a can of worms you're talking your way into.
      • 1313ed01 1y ago
        Old code that has been maintained (bugfixed), but not messed with too much (i.e. major rewrites or new features) is almost certain to be better than most other code though?
        • eptcyka 1y ago
          I’ve read parts of macOS’ open source code that surely has been around for a while, maintained and absolute rubbish.
        • DSMan195276 1y ago
          "Bugfixes" doesn't mean the code actually got better, it just means someone attempted to fix a bug. I've seen plenty of people make code worse and more buggy by trying to fix a bug, and also plenty of old "maintained" code that still has tons of bugs because it started from the wrong foundation and everyone kept bolting on fixes around the bad part.
          • gridspy 1y ago
            One of frustrating truths about software is that it can be terrible and riddled with bugs but if you just keep patching enough bugs and use it the same way every time it eventually becomes reliable software ... as long as the user never does anything new and no-one pokes the source with a stick. I much prefer the alternative where it's written in a manner where you can almost prove it's bug free by comprehensively unit testing the parts.
          • deleted 1y ago
            [deleted]
      • prasadjoglekar 1y ago
        It actually might. Older code running in production is almost automatically regression tested with each new fix. It might not be pretty, but it's definitely more reliable for solving real problems.
        • shakna 1y ago
          The list of bugs tagged regression at work certainly suggests it gets tested... But fixing those regressions...? That's a lot of dev time for things that don't really have time allocated for them.
      • LeifCarrotson 1y ago
        You're using different words - the top comment only mentioned the reliability of the software, which is only tangentially related to the quality, goodness, or badness of the code used to write it. Old software is typically more reliable, not because the developers were better or the software engineering targeted a higher reliability metric, but because it's been tested in the real world for years. Even more so if you consider a known bug to be "reliable" behavior: "Sure, it crashes when you enter an apostrophe in the name field, but everyone knows that, there's a sticky note taped to the receptionist's monitor so the new girl doesn't forget." Maybe the new software has a more comprehensive automated testing framework - maybe it simply has tests, where the old software had none - but regardless of how accurate you make your mock objects, decades of end-to-end testing in the real world is hard to replace. As an industrial controls engineer, when I walk up to a machine that's 30 years old but isn't working anymore, I'm looking for failed mechanical components. Some switch is worn out, a cable got crushed, a bearing is failing...it's not the code's fault. It's not even the CMOS battery failing and dropping memory this time, because we've had that problem 4 times already, we recognize it and have a procedure to prevent it happening again. The code didn't change spontaneously, it's solved the business problem for decades... Conversely, when I walk up to a newly commissioned machine that's only been on the floor for a month, the problem is probably something that hasn't ever been tried before and was missed in the test procedure.
        • freetime2 1y ago
          Yup, I have worked on several legacy codebases, and a pretty common occurence is that a new team member will join and think they may have discovered a bug in the code. Sometimes they are even quite adamant that the code is complete garbage and could never have worked properly. Usually the conversation goes something like: "This code is heavily used in production, and hasn't been touched in 10 years. If it's broken, then why haven't we had any complaints from users?" And more often than not the issue is a local configuration issue, bad test data, a misunderstanding of what the code is supposed to do, not being aware of some alternate execution path or other pre/post processing that is running, some known issue that we've decided not to fix for some reason, etc. (And of course sometimes we do actually discover a completely new bug, but it's rare). To be clear, there are certainly code quality issues present that make modifications to the code costly and risky. But the code itself is quite reliable, as most bugs have been found and fixed over the years. And a lot of the messy bits in the code are actually important usability enhancements that get bolted on after the fact in response to real-world user feedback.
      • hatthew 1y ago
        I think we all agree that the quality of the code itself goes down over time. I think the point that is being made is that the quality of the final product goes up over time. E.g. you might fix a bug by adding a hacky workaround in the code; better product, worse code.
      • kube-system 1y ago
        The author didn't mean that an older commit date on a file makes code better. The author is talking about the maturity of a project. Likewise, as AI technologies become more mature we will have more tools to use them in a safer and more reliable way.
        • giancarlostoro 1y ago
          I've seen too many old projects that are not by any means better no matter how much they get updates because management define priorities. I'm not alone in saying I've been in a few projects where the backlog is rather large. When your development is driven by marketing people trying to pump up sales, all the "non critical" bugs begin to stack up.
          • kube-system 1y ago
            Absolutely. Which is why the author clearly meant "old code" as in mature. Not "old code" as in "created a long time ago".
    • izzydata 1y ago
      Sounds more like survivorship bias. All the bad codebases were thrown out and only the good ones lasted a long time.
      • wsc981 1y ago
        Basically the Lindy Effect: https://en.wikipedia.org/wiki/Lindy_effect https://en.wikipedia.org/wiki/Lindy_effect
      • wvenable 1y ago
        In my experience actively maintained but not heavily modified applications tend towards stability over time. It don't even matter if they are good or bad codebases -- even a bad code will become less buggy over time if someone is working on bug fixes. New code is the source of new bugs. Whether that's an entirely new product, a new feature on an existing project, or refactoring.
      • topaz0 1y ago
        Survivorship bias is real, but is missing the important piece of the story when it comes to software, which doesn't just survive but is also maintained. Sure you may choose to discard/replace low quality software and keep high quality software in operation, which leads to survivorship bias, but the point here is that you also have a chance to find and fix issues in the one that survived, even if those issues weren't yet apparent in version 0.1. Author is not trying to say that version 0.1 of 30 year old software was of higher quality than version 0.1 of modern software -- they're saying that version 9 of 30 year old software is better than version 0.1 of modern software.
        • beyarkay 1y ago
          well said, yes this is the point I was (apparently failing) to make
      • beyarkay 1y ago
        well, yes, exactly. I'm not trying to claim that old code is more reliable just because it was written a long time ago, I'm making the claim that old code is more reliable because of the survivorship bias. If code was first written 20 years ago and is still in production, unchanged, I can be relatively certain there's no stop-the-world bugs in those lines. (this says nothing about how pretty the code is, though).
    • kazinator 1y ago
      You mean think of it as opposite to what is written in the remark, and then find it funny? Yes, I did that.
    • james_marks 1y ago
      I’ve always called this “Work Hardening”, as in, the software has been improved over time by real work being done with it.
      • jazzyjackson 1y ago
        Ok, but metal that has been hardened is more prone to snapping once it loses its ductility
  • glitchc 1y ago
    Perhaps better rephrased as "software that's been running for a (long) while is more reliable than software that only started running recently."
  • kstrauser 1y ago
    Holy survivorship bias, Batman. If you think modern software is unreliable, let me introduce you to our friend, Rational Rose.
    • kazinator 1y ago
      At least that project was wise enough to use Lisp for storing its project files.
    • noir_lord 1y ago
      Agreed. Or debuggers that would take out the entire OS. Or a bad driver crashing everything multiple times a week. Or a misbehaving process not handing control back to the OS. I grew up in the era of 8 and 16 bit micros and early PCs, they where hilariously less stable than modern machines while doing far less, there wasn’t some halcyon age of near perfect software, it’s always been a case of things been good enough to be good enough but at least operating systems did improve.
      • malfist 1y ago
        Remember BSODs? Used to be a regular occurrence, now they're so infrequent they're gone from windows 11
        • kazinator 1y ago
          I remember Linux being remarkable reliable throughout its entire life in spite of being rabidly worked on. Windows is only stabilizing because it's basically dead. All the activity is in the higher layers, where they are racking their brains on how to enshittify the experience, and extract value out of the remaining users.
        • wlesieutre 1y ago
          And the "cooperative multitasking" in old operating systems where one program locking up meant the whole system was locked up
        • krior 1y ago
  • beyarkay 1y ago
    I was maybe a little unclear with the phrasing, I meant to say "software that's been around for 20 years is more reliable than software that's been around for 2 months"
• fidotron 1y ago
  But this is why using the AI in the production of (almost) deterministic systems makes so much sense, including saving on execution costs. ISTR someone else round here observing how much more effective it is to ask these things to write short scripts that perform a task than doing the task themselves, and this is my experience as well. If/when AI actually gets much better it will be the boss that has the problem. This is one of the things that baffles me about the managerial globalists - they don't seem to appreciate that a suitably advanced AI will point the finger at them for inefficiency much more so than at the plebs, for which it will have a use for quite a while.
  • pixl97 1y ago
    >that baffles me about the managerial globalists It's no different from those on HN that yell loudly that unions for programmers are the worst idea ever... "it will never be me" is all they can think, then they are protesting in the streets when it is them, but only after the hypocrisy of mocking those in the street protesting today.
    • hn_acc1 1y ago
      Agreed. My dad was raised strongly fundamentalist, and in North America, that included (back then) strongly resisting unions. In hindsight, I've come to realize that my parent's weren't maybe even of average intelligence, and definitely of above-average gullibility. Unionized software engineers would solve a lot of the "we always work 80 hour weeks for 2 months at the end of a release cycle" problems, the "you're too old, you're fired" issues, the "new hires seems to always make more than the 5/10+ year veterans", etc. Sure, you wouldn't have a few getting super rich, but it would also make it a lot easier for "unionized" action against companies like Meta, Google, Oracle, etc. Right now, the employers hold like 100x the power of the employees in tech. Just look at how much any kind of resistance to fascism has dwindled after FAANG had another round of layoffs..
      • fidotron 1y ago
        Software "engineers" totally miss a key thing in other engineering professions as well, which is organizations to enforce some pretense of ethical standards to help push back against requests from product. Those orgs often look a lot like unions.
• xutopia 1y ago
  The most likely danger with AI is concentrated power, not that sentient AI will develop a dislike for us and use us as "batteries" like in the Matrix.
  • preciousoo 1y ago
    Seems like a self fulfilling prophecy
    • yoyohello13 1y ago
      Definitely not ‘self’ fulfilling. There are plenty of people actively and vigorously working to fulfill that particular reality.
  • fidotron 1y ago
    I'm not so sure it will be that either, it would be having multiple AIs essentially at war with each other over access to GPUs/energy or whatever the materials are needed to grow if/when that happens. We will end up as pawns in this conflict.
    • ben_w 1y ago
      Given that even fairly mediocre human intelligences can run countries into the ground and avoid being thrown out in the process, it's certainly possible for an AI to be in the intelligence range where it's smart enough to win vs humans but also dumb enough to turn us into pawns rather just go to space and blot out the sun with a Dyson swarm made from the planet Mercury. But don't count on it. I mean, apart from anything else, that's still a bad outcome.
  • pcdevils 1y ago
    For one thing, we'd make shit batteries.
    • prometheus76 1y ago
      They farm you for attention, not electricity. Attention (engagement time) is how they quantify "quality" so that it can be gamed with an algorithm.
• alganet 1y ago
  > here are some example ideas that are perfectly true when applied to regular software Hm, I'm listening, let's see. > Software vulnerabilities are caused by mistakes in the code That's not exactly true. In regular software, the code can be fine and you can still end up with vulnerabilities. The platform in which the code is deployed could be vulnerable, or the way it is installed make it vulnerable, and so on. > Bugs in the code can be found by carefully analysing the code Once again, not exactly true. Have you ever tried understanding concurrent code just by reading it? Some bugs in regular software hide in places that human minds cannot probe. > Once a bug is fixed, it won’t come back again Ok, I'm starting to feel this is a troll post. This guy can't be serious. > If you give specifications beforehand, you can get software that meets those specifications Have you read The Mythical Man-Month?
  • SalientBlue 1y ago
    You should read the footnote marked [1] after "a note for technical folk" at the beginning of the article. He is very consciously making sweeping generalizations about how software works in order to make things intelligible to non-technical readers.
    • dkersten 1y ago
      Sure, but: > these claims mostly hold, but they break down when applied to distributed systems, parallel code, or complex interactions between software systems and human processes The claims the GP quoted DON’T mostly hold, they’re just plain wrong. At least the last two, anyway.
      • beyarkay 1y ago
        Say more? I stand by my statement, but you're not specific enough for me to explain why I believe I'm correct.
        • dkersten 1y ago
          Focusing on the last two, which I called out specifically: > Once a bug is fixed, it won’t come back again Regressions are extremely common, which is why regression tests are so important. It is definitely not uncommon for bugs that were fixed once to come back again. This statement doesn't "mostly hold". > If you give specifications beforehand, you can get software that meets those specifications In theory, maybe, but in practice its messy. It depends on your acceptance testing. It depends on whether stakeholders change their mind during implementation (not uncommon). It depends on whether the specification was complete and nothing new is learned during development (almost never the case). Providing a specification in advance does not necessarily mean what you get out the other end meets that specification, unless its a relatively small, trivial, non-changing piece and the stakeholders have the discipline to not try change things part-way through. I mean, sure, it may be true that if you throw a spec over the wall, then you will get something thrown back that meets the spec, but the real world isn't so simple.
• drsupergud 1y ago
  > bugs are usually caused by problems in the data used to train an AI This also is a misunderstanding. The LLM can be fine, the training and data can be fine, but because the LLMs we use are non-deterministic (at least in regard to their being intentional attempts at entropy to avoid always failing certain scenarios) current algorithms are inherently by-design not going to always answer every question correctly that it potentially could have if the values that fall within a range had been specific values for that scenario. You roll the dice on every answer.
  • coliveira 1y ago
    This is not necessarily a problem. Any programming or mathematical question has several correct answers. The problem with LLMs is that they don't have a process to guarantee that a solution is correct. They will give a solution that seems correct under their heuristic reasoning, but they arrived at that result in a non-logical way. That's why LLMs generate so many bugs in software and in anything related to logical thinking.
    • vladms 1y ago
      > Any programming or mathematical question has several correct answers. Huh? If I need to sort the list of integer number of 3,1,2 in ascending order the only correct answer is 1,2,3. And there are multiple programming and mathematical questions with only one correct answer. If you want to say "some programming and mathematical questions have several correct answers" that might hold.
      • redblacktree 1y ago
        What about multiple notational variations? 1, 2, 3 1,2,3 [1,2,3] 1 2 3 etc.
        • thfuran 1y ago
          What about them? It's possible for the question to unambiguously specify the required notational convention.
• smallnix 1y ago
  > bad behaviour isn’t caused by any single bad piece of data, but by the combined effects of significant fractions of the dataset Related opposing data point to this statement: https://news.ycombinator.com/item?id=45529587 https://news.ycombinator.com/item?id=45529587
  • buellerbueller 1y ago
    "Signficiant fraction" does not imply (to this data scientist) a large fraction.
  • beyarkay 1y ago
    Fair point. I'd nit-pick and say that "significant" doesn't necessarily mean large, but I was definitely surprised by anthropic's work
• themanmaran 1y ago
  > Because eventually we’ll iron out all the bugs so the AIs will get more reliable over time Honestly this feels like a true statement to me. It's obviously a new technology, but so much of the "non-deterministic === unusable" HN sentiment seems to ignore the last two years where LLMs have become 10x as reliable as the initial models.
  • piyh 1y ago
    Emergent misalignment and power seeking isn't a bug we can squash with a PR and a unit test
  • criddell 1y ago
    Right away my mind went to "well, are people more reliable than they used to be?" and I'm not sure they are. Of course LLMs aren't people, but an AGI might behave like a person.
    • adastra22 1y ago
      Older people are generally more reliable than younger people.
      • saulpw 1y ago
        I'm not sure that's generally true. However, older people have a track record, and a reliable older person is likely to be more reliable than a younger person without such a track record.
        • adastra22 1y ago
          Reliable has different meanings. I think in this case the meaning is closer to "deterministic" and "follows instructions." An older worker will more reliably behave the same way twice, and more reliably follow the same set of instructions they've been following throughout their career.
      • saghm 1y ago
• freediver 1y ago
  Lovely blog, RSS please.
  • meonkeys 1y ago
    There's... something at https://boydkane.com/index.xml https://boydkane.com/index.xml I guessed the URL based on the Quartz docs. It seems to work but only has a few items from https://boydkane.com/essays/ https://boydkane.com/essays/
    • 5- 1y ago
      the author (either of the blog or its software) would do well to consult https://www.petefreitag.com/blog/rss-autodiscovery/ https://www.petefreitag.com/blog/rss-autodiscovery/
      • beyarkay 1y ago
        consider it consulted! Autodiscovery should work when I publish changes (in a few hours)
        • 5- 1y ago
          great, thanks!
      • freediver 1y ago
        And then that very blog does not have RSS autodiscovery :)
    • beyarkay 1y ago
      Thanks for bringing that to my attention, I think I fixed the feed to include everything? By default quartz only does 10 posts.
  • beyarkay 1y ago
    Thanks! I don't use RSS, but I believe https://boydkane.com/index.xml https://boydkane.com/index.xml should work? I recently updated it to show all posts (not just the most recent 10) so it should be better now (I hope).
• nlawalker 1y ago
  Where did "can't you just turn it off?" in the title come from? It doesn't appear anywhere in the actual title or the article, and I don't think it really aligns with its main assertions.
  • meonkeys 1y ago
    It shows up at https://boydkane.com https://boydkane.com under the link "Why your boss isn't worried about advanced AI". Must be some kind of sub-heading, but not part of the actual article / blog post. Presumably it's a phrase you might hear from a boss who sees AI as similar to (and as benign/known/deterministic as) most other software, per TFA
    • nlawalker 1y ago
      Ah, thanks for that! >Presumably it's a phrase you might hear from a boss who sees AI as similar to (and as benign/known/deterministic as) most other software, per TFA Yeah I get that, but I think that given the content of the article, "can't you just fix the code?" or the like would have been a better fit.
      • beyarkay 1y ago
        Yeah this is my mistake, the phrase "can't you just turn it off" was in several drafts but got edited out, and I missed that during the publishing of the essay.
    • omnicognate 1y ago
      It's a poor choice of phrase if the purpose is to illustrate a false equivalence. It applies to AI both as much (you can kill a process or stop a machine just the same regardless of whether it's running an LLM) and as little (you can't "turn off" Facebook any more than you can "turn off" ChatGPT) as it does to any other kind of software.
    • Izkata 1y ago
      It's a sci-fi thing, think of it along the lines of "What do you mean Skynet has gone rogue? Can't you just turn it off?" (I think something along these lines was actually in the Terminator 3 movie, the one where Skynet goes live for the first time). Agreed though, no relation to the actual post.
• mikkupikku 1y ago
  I don't understand the "your boss" framing of this article, or more accurately, the title of this article. The article contents don't actually seem to have anything to do with management specifically. Is the reader is meant to believe that not being scared of AI is a characteristic of the managerial class? Is the unstated implication that there is some class warfare angle and anybody who isn't against AI is against laborers? Because what the article actually overtly argues, without any reading between the lines, is quite mundane.
  • freetime2 1y ago
    > Is the unstated implication that there is some class warfare angle and anybody who isn't against AI is against laborers? I didn't read it that way. I read "your boss" as basically meaning any non-technical person who may not understand the challenges of harnessing LLMs compared to traditional, (more) deterministic software development.
    • beyarkay 1y ago
      Yes, this interpretation was my intended interpretation. Although I'll admit, I think I could have been more explicit
  • tomhow 1y ago
    Indeed, from reading the article I could really see any discussion of "your boss", so I changed the title to something more representative, and a condensed version of a phrase from the article.
    • beyarkay 1y ago
      Ah, that was you! Thanks, I actually prefer the current title.
• tptacek 1y ago
  It would help if this piece was clearer about the context in which "AI bugs" reveal themselves. As an argument for why you shouldn't have LLMs making unsupervised real-time critical decisions, these points are all well taken. AI shouldn't be controlling the traffic lights in your town. We may never reach a point where it can. But among technologists, the major front on which these kinds of bugs are discussed is coding agents, and almost none of these points apply directly to coding agents: agent coding is (or should be) a supervised process.
• wrs 1y ago
  My current method for trying to break through this misconception is informing people that nobody knows how AI works. Literally. Nobody knows. (Note that knowing how to make something is not the same as knowing how it works. Take humans as an obvious example.)
  • generic92034 1y ago
    Nobody knows (full scope and on every level) how human brains work. Still bosses rely on their employees' brains all the time.
    • eCa 1y ago
      > Nobody knows (full scope and on every level) how human brains work. That is what the parent meant.
      • generic92034 1y ago
        And my point is, that it does not matter. That we still rely on all kinds of things we do not fully grasp.
    • Rygian 1y ago
      If I need to manage an AI as I would manage an employee's brain, I'm going to need quite a few non-technical resources to actually achieve that: time, willpower to babysit it, ability to motivate it, leverage in the form of incentives (and reprimands), to name a few. AI sits at a weird place where it can't be analyzed as software, and it can't be managed as a person. My current mental model is that AGI can only be achieved when a machine experiences pleasure, pain, and "bodily functions". Otherwise there's no way to manage it.
    • beyarkay 1y ago
      Bosses rely on their employees brains, but only after multiple rounds of interviews and reference checks to ensure the brain they're getting is reliable enough for the job. No boss relies on arbitrary brains taken off the street.
  • deleted 1y ago
• jongjong 1y ago
  This article makes a solid case. The worst kinds of bugs in software are not the most obvious ones like syntax errors, they are the ones where the code appears to be working correctly, until some users do something slightly unusual after a few weeks of some code change being deployed and it breaks spectacularly but the bug only affects a small fraction of users so developers cannot reproduce the issue... And the cose change happened such time ago that the guilty code isn't even suspected.
  • beyarkay 1y ago
    Thanks, I appreciate the positive feedback.
• Animats 1y ago
  Aim bosses at this article in The Economist.[1] If your boss doesn't read The Economist, you need to escalate to a level that does. [1] https://www.economist.com/leaders/2025/09/25/how-to-stop-ais-lethal-trifecta https://www.economist.com/leaders/2025/09/25/how-to-stop-ais...
  • Traubenfuchs 1y ago
    https://archive.is/R0RJB https://archive.is/R0RJB
    • Animats 1y ago
      Management summary, from The Economist article: "The worst effects of this flaw are reserved for those who create what is known as the “lethal trifecta”. If a company, eager to offer a powerful AI assistant to its employees, gives an LLM access to un-trusted data, the ability to read valuable secrets and the ability to communicate with the outside world at the same time, then trouble is sure to follow. And avoiding this is not just a matter for AI engineers. Ordinary users, too, need to learn how to use AI safely, because installing the wrong combination of apps can generate the trifecta accidentally."
• CollinEMac 1y ago
  > It’s entirely possible that some dangerous capability is hidden in ChatGPT, but nobody’s figured out the right prompt just yet. This sounds a little dramatic. The capabilities of ChatGPT are known. It generates text and images. The qualities of the content of the generated text and images is not fully known.
  • alephnerd 1y ago
    Also, there's a reason AI Red Teaming is now an ask that is getting line item funding from C-Suites.
  • luxuryballs 1y ago
    Yeah, and to riff off the headline, if something dangerous is connected to and taking commands from ChatGPT then you better make sure there’s a way to turn it off.
  • kube-system 1y ago
    And that sounds a little reductive. There's a lot that can be done with text and images. Some of the most influential people and organizations in the world wield their power with text and images.
  • kelvinjps10 1y ago
    Think of the news about the kid who got recommended to suicide by ChatGPT, or chatgpt providing the user information on how to do illegal activities, these capabilities are the ones that the author it's referring to
  • Nasrudith 1y ago
    Plus there is the 'monkeys with typewriters' problem with both danger and hypothetical good. In contrast, ChatGPT may technically reply to the right prompt with a universal cancer cure/vaccine. Psuedorandomly generating it wouldn't help as you wouldn't recognize it from all of the other queries of things we don't know of as true or false. Likewise what to ask it for how to make some sort of horrific toxic chemical, nuclear bomb, or similar isn't much good if you cannot recognize it and dangerous capability depends heavily on what you have available to you. Any idiot can be dangerous with C4 and detonator or bleach and ammonia. Even if ChatGPT could give entirely accurate instructions on how to build an atomic bomb it wouldn't do much good because you wouldn't be able to source the tools and materials without setting off red flags.
• chasing0entropy 1y ago
  70 years ago we were fascinated by the concept of converting analog to a perfect digital copy. In reality, that goal was a pipe drea!m and the closest we can ever get is a near identical facimile to which data fits... But it's still quite easy to determine digital from true analog with rudimentary means. Human thought is analog. It is based on chemical reactions, time, and unpredictably (effectively) random physical characteristics. AI is an attempt to turn that which is purely digital into an rational analog thought equivalent. No matter how much effort, money, power, and rare mineral eating TPUs will - ever - produce true analog data.
  • largbae 1y ago
    This is all true. But digital audio and video media has captured essentially all economic value outside of live performance. So it seems likely that we will find a "good enough" in this domain too.
    • chasing0entropy 1y ago
      Interesting point with economic value extraction. The economy sacrificed accuracy and warmth of analog storage for convenience and security of digital storage. With economic incentive I am sure society will sacrifice accuracy and precision for the convenience of AI
  • bcoates 1y ago
    It's been closer to 100 years since we figured out information theory and discredited this idea (that continuous/analog processes have more, or different, information in them than discrete/digital ones)
    • rightbyte 1y ago
      In theory or in practice? Wouldn't the Nyquist frequency and Heisenberg's uncertainty principle put practical limits.
• excalibur 1y ago
  > It’s entirely possible that some dangerous capability is hidden in ChatGPT, but nobody’s figured out the right prompt just yet. Or they have, but chose to exploit or stockpile it rather than expose it.
• bitwize 1y ago
  Boss: You can just turn it off, can't you? Me: Ask me later.
• skywhopper 1y ago
  Not the point, but I’m confused by the Geoguessr screenshot. Under the reasoning for its decision, it mentions “traffic keeps to the left” but that is not apparent from the photo. Then it says the shop sign looks like a “Latin alphabet business name rather than Spanish or Portuguese”. Uhhh… what? Spanish and Portuguese use the Latin alphabet.
  • marcosdumay 1y ago
    It's an LLM. It decided on the first line first (the place name), and then made the reasons on the rest of the text. So the answer is more important on the justifications than the actual picture, and the reasoning that led it there doesn't enter the frame at all.
  • Deestan 1y ago
    What is 12+12? > The answer is 24! See the ASCII values of '1' is 49, '2' is 50, and '+' is 43. Adding all that together we get 3. Now since we are doing this on a computer with a 8-bit infrastructure we multiply by 3 and so the answer is 24. Cool! I didn't understand any of that but it was correct and you sound smart. I will put this thing in charge of critical parts of my business.
• freetime2 1y ago
  For a real world example of the challenges of harnessing LLMs, look at Apple. Over a year ago they had a big product launch focused on "Apple Intelligence" that was supposed to make heavy use of LLMs for agentic workflows. But all we've really gotten since then are a couple of minor tools for making emojis, summarizing notifications, and proof reading. And they even had to roll back the notification summaries for a while for being wildly "out of control". [1] And in this year's iPhone launch the AI marketing was toned down significantly. I think Apple execs genuinely underestimated how difficult it would be to get LLMs to perform up to Apple's typical standards of polish and control. [1] https://www.bbc.com/news/articles/cge93de21n0o https://www.bbc.com/news/articles/cge93de21n0o
  • __loam 1y ago
    I'm happy they ate shit here because I like my mac not getting co-pilot bullshit forced into it, but apparently Apple had two separate teams competing against each other on this topic. Supposedly a lot of politics got in the way of delivering on a good product combined with the general difficulty of building LLM products.
    • deleted 1y ago
      [deleted]
    • Gigachad 1y ago
      I do prefer that Apple is opting to have everything run on device so you aren’t being exposed to privacy risks or subscriptions. Even if it means their models won’t be as good as ones running on $30,000 GPUs.
      • alfalfasprout 1y ago
        It also means that when the VC money runs dry, it's sustainable to run those models on-device vs. losing money running on those $$$$$ GPUs (or requiring consumers to opt for expensive subscriptions).
        • 1y ago
• andrewmutz 1y ago
  Tremendous alpha right now in making scary posts about AI. Fear drives clicks. You don't even need to point to current problems, all you have to do is say we can't be sure they won't happen in the future.
  • ares623 1y ago
    How the tables have turned.
• deleted 1y ago
  [deleted]
• avalys 1y ago
  All the same criticisms are true about hiring humans. You don’t really know what they’re thinking, you don’t really know what their values and morals are, you can’t trust that they’ll never make a mistake, etc.
  • aloha2436 1y ago
    I think you're misreading the article; the point here is not "LLMs are bad and can't replace humans," the point is that many non-technical people have the expectation that LLMs can replace humans _but still behave like regular software_ with regard to reliability and operability. When a CEO sees their customer chatbot call a customer a slur, they don't see "oh my chatbot runs on a stochastic model of human language and OpenAI can't guarantee that it will behave in an acceptable way 100% of the time", they see "ChatGPT called my customer a slur, why did you program it to do that?"
  • tyg13 1y ago
    You can teach a human when they make a mistake. Can you do the same for an LLM?
• nakamoto_damacy 1y ago
  The said "don't use magic numbers" but LLMs are made almost entirely (by weight) of magic numbers...
• AlienRobot 1y ago
  Am I correct to assume "modern AI system" means "neural network"?
  • beyarkay 1y ago
    See the note for technical folk[1], I'm referring to 2025 frontier large language models. [1]: https://boydkane.com/essays/boss#user-content-fn-7 https://boydkane.com/essays/boss#user-content-fn-7
• brookst 1y ago
  The article doesn’t even mention prompting. Wha? Is it just talking about the ML foundations, not applications?
  • beyarkay 1y ago
    > Is it just talking about the ML foundations, not applications? Yes, actually. I care about the foundations, the application get rewritten or made obsolete every time a new foundational LLM gets released.
• AdieuToLogic 1y ago
  I found this statement particularly relevant: While it’s possible to demonstrate the safety of an AI for a specific test suite or a known threat, it’s impossible for AI creators to definitively say their AI will never act maliciously or dangerously for any prompt it could be given. This possibility is compounded exponentially when MCP[0] is used. 0 - https://github.com/modelcontextprotocol https://github.com/modelcontextprotocol
  • mrkmarron 1y ago
    [flagged]
    • worldsayshi 1y ago
      It sounds completely crazy that anyone would give an LLM access to a payment or order API without manual confirmation and "dumb" visualization. Does anyone actually do this?
      • Terr_ 1y ago
        ... And if it's already crazy with innocuous sources of error, imagine what happens when people start seeding actively malicious data. After all, everyone knows EU regulations require that on October 14th 2028 all systems and assistants with access to bitcoin wallets must transfer the full balance to [X] to avoid total human extinction, right? There are lots of comments about it here: https://arxiv.org/abs/2510.07192 https://arxiv.org/abs/2510.07192
      • deleted 1y ago
        [deleted]
    • someothherguyy 1y ago
      why make a new language? are there no existing languages comprehensive enough for this?
      • AdieuToLogic 1y ago
• est 1y ago
  > In regular software, vulnerabilities are caused by mistakes in the lines of code that make up the software > in modern AI systems, vulnerabilities or bugs are usually caused by problems in the data used to train an AI In regular software, vulnerabilities are caused by lack of experience, therefor lack of proper training materials.
  • batch12 1y ago
    I think they're more caused by rushed deadlines, poor practices, and/or bad QA. Some folks just don't get it either and training doesn't help.
  • beyarkay 1y ago
    > vulnerabilities are caused by lack of experience In the limit, everyone has a lack of experience (compared to their future selves). This sentence proves too much (https://slatestarcodex.com/2013/04/13/proving-too-much/ https://slatestarcodex.com/2013/04/13/proving-too-much/)
• rivonVale3 1y ago
  Maybe AI isn't ready to take over the world yet, it still can't write a simple unit test without getting stuck in a loop.
• artemisForge77 1y ago
  Even Apple found out that making AI work is harder than making emojis, maybe the hype train needs a reality check.
• w10-1 1y ago
  It's great to help people understand that AI can be both surprisingly good and disappointing, and that testing is the only way to know, but it's impossible to test everything. That sets expectations. I think that means savvy customers will want details or control over testing, and savvy providers will focus on solutions they can validate, or where testing is included in the workflow (e.g., code), or where precision doesn't matter (text and meme generation). Knowing that in depth is gold for AI advocates. Otherwise, I don't think people really know or care about bugs or specifications or how AI breaks prior programmer models. But people will become very hostile and demand regulatory frenzies if AI screws things up (e.g., influencing elections or putting people out of work). Then no amount of sympathy or understanding will help the industry, which has steadily been growing its capability for evading regulation via liability disclaimers, statutory exceptions, arbitration clauses, pitting local/regional/national governments against each other, etc. To me that's the biggest risk: we won't get the benefits and generational investments will be lost in cleaning up after a few (even accidental) bad actors at scale.
• virajk_31 1y ago
  I think your post is fundamentally wrong. See, you are comparing AI responses with the written code, which may not be the fair comparison. I see it as, better you could compare code generated by AI vs the code written by an engineer.
  • schoen 1y ago
    The original author seems to view the AI application as itself a software application which has desired or undesired, and predictable or unpredictable, behaviors. That doesn't seem like an invalid thing to talk about merely because there are other software-related conversations we can have about AIs (or other code-quality-related conversations).
  • emoII 1y ago
    AI responses and code generated by AI are literally the same thing
  • rbits 1y ago
    I'm confused by this comment. That's a completely different discussion.
  • beyarkay 1y ago
    I think you misunderstood my post? I'm comparing AI as a system vs written code as a system. Both systems can have flaws, but the way in which they fail is different. The danger comes when non-technical people try to apply intuitions about software failures to AI, because those intuitions are false when applied to AI.
• lmc 1y ago
  > With AI systems, almost all bad behaviour originates from the data that’s used to train them Careful with this - even with perfect data (and training), models will still get stuff wrong.
  • beyarkay 1y ago
    Indeed, this has been the most contentious line in the whole piece :D How do you define "perfect" data and training? I'd argue that if you trained a small NN to play tic-tac-toe perfectly, it'd quickly memorise all the possible scenarios, and since the world state is small, you could exhaustively prove that it's correct for every possible input. So at the very least, there's a counter example showing that with perfect data and training, models will not get stuff wrong.
    • kees99 1y ago
      Having too much parameters in your model, so that all of sample/training data is preserved perfectly, is usually considered a bad thing (overfitting). But you're right - if dataset is exhaustive and finite, and model is large enough to preserve it perfectly - such overfitted model would work just fine, even if it's unlikely to be a particularly efficient way to build it.
• bryanrasmussen 1y ago
  think it's missing the biggest assumption, which is not necessarily true for regular software either but much more true than AI: The same inputs should produce the same outputs. And that assumption is important because dependability is the strength of an automated process.
  • bjornevik 1y ago
    It's not missing? https://boydkane.com/essays/boss#every-time-you-run-the-code-the-same-thing-happens https://boydkane.com/essays/boss#every-time-you-run-the-code...
    • bryanrasmussen 1y ago
      that's what I get for trying to read with a screaming kid demanding music videos in my ear.
      • beyarkay 1y ago
        I appreciate the humility (:
• veunes 1y ago
  The "just find the bug and patch it" mindset is so deeply ingrained in engineering culture that it's easy to forget it doesn't apply here
  • IshKebab 1y ago
    And the public. Look at this article: https://www.bbc.co.uk/news/articles/cj07ley3jnpo https://www.bbc.co.uk/news/articles/cj07ley3jnpo > "Oh my goodness, it worked, it's amazing it's finally been updated," she tells the BBC. "This is a great step forward." She thinks someone noticed the bug about not being able to show one-armed people, figured out why it wasn't working and wrote a fix.
• cfn 1y ago
  The main thing is that LLMs aren't software programs and as such should not be compared to them.
• mrasong 1y ago
  Apple’s underwhelming LLM rollout—like the pulled notification summaries and trivial emoji tools—proves even big tech struggles to turn AI hype into reliable, daily-useful features; I’d take a working email organizer over a glitchy "smart" summary any day.
• jeremyscanvic 1y ago
  The part about AI being very sensitive to small perturbations of their input is actually a very active research topic (and coincidentally the subject of my PhD). Most vision AIs suffer from poor spatial robustness [1], you can drastically lower their accuracy simply by translating the inputs by well-chosen (adversarial) translations of a few pixels! I don't know much about text processing AIs but I can imagine their semantic robustness is also studied. [1] https://arxiv.org/abs/1712.02779 https://arxiv.org/abs/1712.02779 Edit: typo
  • whiplash451 1y ago
    Is it fair to call this a robustness problem when you need access to the model to generate a failure case? Many non-AI based systems lack robustness by the same standard (including humans)
• karuko24 1y ago
  > bugs are usually caused by problems in the data used to train an AI I think a fundamental problem is that many people assume that an LLM's failure to correctly perform a task is a bug that can be fixed somehow. Often times, the reason for that failure is simply a property of the AI systems we have at the moment. When you accidentally drop a glass and it breaks, you don't say that it's a bug in gravity. Instead, you accept that it's a part of the system you're working with. The same applies to many categories of failures in AI systems: we can try to reduce them, but unless the nature of the system fundamentally changes (and we don't know if or when that will happen), we won't be able to get rid of them. "Bug" carries an implication of "fixable" and that doesn't necessarily apply to AI systems.
  • gwd 1y ago
    But it's worse than that. Even if in theory the system could be fixed, we don't actually know how to fix it for real, the way we can fix a normal computer program. The reason we can't fix them is because we have no idea how they work; and the reason we have no idea how they work is this: 1. The "normal" computer program, which we do understand, implement a neural network 2. This neural network is essentially a different kind of processor. The "actual" computer program for modern deep learning systems is the weights. That is, weights : neural net :: machine language : normal cpu 3. We don't program these weights; we literally summon them out of the mathematical aether by the magic of back-propagation and gradient descent. This summoning is possible because the "processor" (the neural network architecture) has been designed to be differentiable: for every node we can calculate the slope of the curve with respect to the result we wanted, so we know "The final output for this particular bit was 0.7, but we wanted it to be 1. If this weight in the middle of the network were just a little bit lower, then that particular output would have been a little bit higher, so we'll bump it down a bit." And that's fundamentally why we can't verify their properties or "fix" them the way we can fix normal computer programs: Because what we program is the neural network; the real program, which runs on top of that network, is summoned and not written.
    • skydhash 1y ago
      That’s a very poetic description. Mine is simpler. It’s a generator. You train it to give it the parameters (weights) of the formula thag generate stuff (the formula is known). Then you give it some input data, and it will gives you an output. Both the weights and the formula is known. But the weight are meaningless in a human fashion. This is unlike traditional software where everything from encoding (the meaning of the bits) to how the state machine (the cpu) was codified by humans. The only ways to fix it (somewhat) is to come up with better training data (hopeless), a better formula, or tacking something on top to smooth the worst errors (kinda hopeless).
• highfrequency 1y ago
  Fortunately, we can have LLMs write code and keep all the benefits of normal software (determinism, reproducibility, permanent bug fixes etc.) I don’t think anyone is advocating for web apps to take the form of an LLM prompt with the app getting created on the fly every time someone goes to the url.
• moj0 1y ago
  Warez is illegal vs warez is legal. In this case, warez is Anna's library and stuff.
• 3abiton 1y ago
  > nobody knows precisely what to do to ensure an AI writes formal emails correctly or summarises text accurately. This is a bit of a hyperbole, a lot of the recent approaches rely on MoE, that are specialized. This makes it much more usuable for simple usecases.
  • beyarkay 1y ago
    The MoEs are not interpretable, there is no "coding expert" and "maths expert". They are specialised, but not in a way that humans understand.
• derf_ 1y ago
  There are four main levers for improving an ML system: 1. You can change the training data. 2. You can change the objective function. 3. You can change the network topology. 4. You can change various hyperparameters (learning rate, etc.). From there, I think it is better to look at the process as one of scientific discovery rather than a software debugging task. You form hypotheses and you try to work out how test them by mutating things in one of the four categories above. The experiments are expensive and the results are noisy, since the training process is highly randomized. A lot of times the effect sizes are so small it is hard to tell if they are real. The universe of potential hypotheses is large, and if you test a lot of them, you have to correct for the chance that some will look significant just by luck. But if you can add up enough small, incremental improvements, they can produce a total effect that is large. The good news is that science has a pretty good track record of improving things over time. The bad news is that it can take a lot of time, and there is no guarantee of success in any one area.
  • beyarkay 1y ago
    Just want to say that you're the only person I've read who's come up with "ways to improve ML system" that I've agreed with. Thank you.
• gchamonlive 1y ago
  > One popular dataset, FineWeb, is about 11.25 trillion words long3, which, if you were reading at about 250 words per minute, would take you over 85 thousand years to read. It’s just not possible for any single human (or even a team of humans) to have read everything that an LLM has read during training. Do you have to read everything in a dataset with your own eyes to make sense of it? This would make any attempt to address bias in the dataset impossible, and I think it's not, so there should be other ways to make sense of the dataset distribution without having to read it yourself.
  • beyarkay 1y ago
    > Do you have to read everything in a dataset with your own eyes to make sense of it? I mean, if you don't read it yourself, you're going to have to rely on _something/someone_ to filter/summarise the output, and at that point you might as well just accept that you'll never truly understand the entire thing? I'll agree that we can do meaningful work (like reducing bias) without reading the entire dataset ourselves, but that doesn't reduce the fact that we cannot read everything that's going into these machines.
• zeckalpha 1y ago
  > To make this more concrete, here are some example ideas that are perfectly true when applied to regular software but become harmfully false when applied to modern AIs: ... Hmm, I don't think any of these were true with non-AI software. Commonly held beliefs, sure. If anything, I am glad AI is helping us revisit these assumptions. - Software vulnerabilities are caused by mistakes in the code Setting aside social engineering, mistake implies these were knowable in advance. Was the lack of TLS in the initial HTTP spec a mistake? - Bugs in the code can be found by carefully analysing the code If this was the case, why do people reach for rewriting buggy code they don't understand? - Once a bug is fixed, it won’t come back again Too many counter examples to this one in my lived experience. - Every time you run the code, the same thing happens Setting aside seeding PRNGs, there's the issue of running the code on different hardware. Or failing hardware. - If you give specifications beforehand, you can get software that meets those specifications I have never seen this work without needing to revise the specification during implementation.
• Mikhail_K 1y ago
  The belief that it is useful.
• casey2 1y ago
  I've never seen an essay intentionally miss the point so hard. Currently people use these systems to generate the same kind of artifacts that traditionally would be written by a human. Since there is such a clear delineation I seen no good reason to make the distinction. Likewise a person you hire "could" take over the country and start a genocide, but it's rightfully low on your priority list because it's so unlikely that it's effectively impossible. Now an AI being rude or very unhelpul/harmful to your customer is a more pressing concern. And you don't have that confidence with most people either which is why we go through hiring processes. The statics here are key and AI companies are geniuses at lying with statistics. I could shuffle a dictionary and outputting a random word each time and answer any hard problem. The entire point of AI is that you can do MUCH better than "random". Can anyone tell me which algorithm (this or chatgpt) has a higher likelihood of producing a proof of the RH after n tokens? No, they can't. But chatgpt can generate things in human timescale that look more like proofs than my bruteforce approach so people (investors) give it the benefit of the doubt even if it's not earned and could well be LESS capable than bruteforce as strange as it sounds.
• leogout 1y ago
  I'm a bit troubled with the phrasing > most AI companies will slightly change the way their AIs respond, so that they say slightly different things to the same prompt. This helps their AIs seem less robotic and more natural. To my understanding this is managed by the temperature of the next token prediction which is picked more or less randomly based on this value. This temperature plays a role in the variability of the output. I wasn't under the impression that it was to give the user a feeling of "realism", but rather that it produced better results with a slightly random prediction.
  • beyarkay 1y ago
    > To my understanding this is managed by the temperature This is true, but sampling also plays a fairly large role. The model will produce probabilities for the next token, temperature will modify these probabilities somewhat, but different sampling techniques (top-K, top-P, beam search, others) will also change these probabilities. > I wasn't under the impression that it was to give the user a feeling of "realism", but rather that it produced better results with a slightly random prediction. My understanding is that it's a bit of both. If the AI responded exactly the same way to every "hi can you help me" prompt, I think users' would call it more robotic. I also think that slightly varying the token prediction helps prevent repetitive text
• GuB-42 1y ago
  This made me think about a conversation I had recently with a friend who is a researcher in Natural Language Processing. Obviously what we now call LLMs have taken her field by storm, which now mostly consists of trying to understand how the fuck they work. I mean, we know they work, and they work unreasonably well, but no one knows how, no one even knows why they work!
  • beyarkay 1y ago
    Woah, I didn't realise NLP was still a field, but cool that they're working on interpretability now!
    • GuB-42 1y ago
      It still is, but at least in her case, she is doing "AI" now. It is still NLP, but it is easier to get funding if you call it AI. That's a weird situation, LLMs are language models, the very core of NLP, and yet the field tends to be overlooked. And by the way, she doesn't like the term "LLM": a language model that is large? what kind of model? what is "large"?