7 ms·

This highlights a thing I've seen with LLM's generally: they make different mistakes than humans. This makes catching the errors much more difficult. What I me

This highlights a thing I've seen with LLM's generally: they make different mistakes than humans. This makes catching the errors much more difficult. What I mean by this is that we have thousands of years of experience catching human mistakes. As such, we're really good at designing systems that catch (or work around) human mistakes and biases. LLM's, while impressive and sometimes less mistake-prone than humans, make errors in a fundamentally different manner. We just don't have the intuition and understanding of the way that LLM's "think" (in a broad sense of the word). As such, we have a hard time designing systems that account for this and catch the errors.

tharkun__ 2y ago
I don't think that's universally true. We have different humans with different levels of ability to catch errors. I see that with my teams. Some people can debug. Some can't. Some people can write tests. Some can't. Some people can catch stuff in reviews. Some can't. I asked Sonnet 3.7 in Cursor to fix a failing test. While it made the necessary fix, it also updated a hard-coded expected constant to instead be computed using the same algorithm as the original file, instead of preserving the constant as the test was originally written. Guess what? Guess the number of times I had to correct this from humans doing it in their tests over my career! And guess where the models learned the bad behavior from.
- __MatrixMan__ 2y ago
  I agree. I've been been struck by how remarkably understandable the errors are. It's quite often something that I'd have done myself if I wasn't paying attention to the right thing.
- fn-mote 2y ago
  > Some people can debug. Some can't. Some people can write tests. Some can't. Wait… really? No way do I want to work with someone who can’t debug or write tests. I thought those were entry stakes to the profession. People whose skills you use in other ways because they are more productive? Maybe. But still. Clean up after yourself. It’s something that should be learned in the apprentice phase.
  - hobs 2y ago
    Keyword want - most people don't control who their peers are, and complaining to your boss doesn't get you that far, especially when said useless boss is fostering said useless person.
  - tharkun__ 2y ago
    Like my sibling says, you can't always choose. That's one side of that coin. The other is: Some people are naturally good at writing "green field" (or re-writing everything) and do produce actual good software. But these same people, which you do want to keep around if that's the best you can get, are next to useless when you throw a customer reported bug at them. Takes them ages to figure anything out and they go down endless rabbit holes chasing the wrong path for hours. You also have people that are super awesome at debugging. They have knack for seeing some brokenness and having the right idea or an idea of the right direction to investigate in right away, can apply the scientific method to test their theories and have the bug fixed in the time it take one of these other people to go down even a single of the rabbit holes they will go down. But these same people in some cases are next to useless if you ask them to properly structure a new green field feature or rewrite parts of something to use some new library coz the old one is no longer maintained or something and digging through said new library and how it works. Both of these types of people are not bad in and of themselves. Especially if you can't get the unicorns that can do all of these things well (or well enough), e.g. because your company can't or won't pay for it or only for a few of them, which they might call "Staff level". And you'd be amazed how easy it is to get quite a few review comments in for even Staff level people if you basically ignore their actual code and just jump right into the tests. It's a pet peeve of mine. I start with the tests and go from there when reviewing :) What you really don't want is if someone is not good at any of these of course.
  - groby_b 2y ago
    > No way do I want to work with someone who can’t debug or write tests. I thought those were entry stakes to the profession. Those are almost entry stakes at tier-one companies. (There are still people who can't, it's just much less common) In your average CRUD/enterprise automation/one-off shellscript factory, the state of skills is... not fun. There's a reason there's the old saw of "some people have twenty years experience, some have the same year 20 times over". People learn & grow when they are challenged to, and will mostly settle at acquiring the minimum skill level that lets them do their particular work. And since we as an industry decided to pretend we're a "science", not skills based, we don't have a decent apprenticeship system that would force a minimum bar. And whenever we discuss LLMs and how they might replace software engineering, I keep remembering that they'll be prompted by the people who set that hiring bar and thought they did well.
- skerit 2y ago
  Claude Sonnet 3.7 really, really loves to rewrite tests so they'll pass. I've had it happen many times in a claude-code session, I had to add this to each request (though it did not fix it 100%) - Never disable, skip, or comment out failing unit tests. If a unit test fails, fix the root cause of the exception. - Never change the unit test in such a way that it avoids testing the failing feature (e.g., by removing assertions, adding empty try/catch blocks, or making tests trivial). - Do not mark tests with @Ignore or equivalent annotations. - Do not introduce conditional logic that skips test cases under certain conditions. - Always ensure the unit test continues to properly validate the intended functionality.
  - _ea1k 2y ago
    I'm guessing this is a side effect of mistakes in the reinforcement learning face. It'd be really easy to build a reward model that favors passing tests, without properly measuring the quality of those tests.
sorokod 2y ago
You may find this interesting: "AI Mistakes Are Very Different from Human Mistakes" https://www.schneier.com/blog/archives/2025/01/ai-mistakes-are-very-different-from-human-mistakes.html https://www.schneier.com/blog/archives/2025/01/ai-mistakes-a...
woopwoop 2y ago
Agree, but I would point out that the errors that I make are selected on the fact that I don't notice I'm making them, which tips the scale toward LLM errors being not as bad.
- worldsayshi 2y ago
  Yeah it's the reason pair programming is nice. Now the bugs need to pass two filters instead of one. Although I suppose LLM's aren't that good at catching my bugs without me pointing them out.
  - diggan 2y ago
    I've found both various ChatGPT and Claude to be pretty good at finding unknown bugs, but you need a somewhat hefty prompt. Personally I use a prompt that goes something like this (shortened here): "Go through all the code below and analyze everything it's doing step-by-step. Then try to explain the overall purpose of the code based on your analysis. Then think through all the edge-cases and tradeoffs based on the purpose, and finally go through the code again and see if you can spot anything weird" Basically, I tried to think of what I do when I try to spot bugs in code, then I just wrote a reusable prompt that basically repeats my own process.
    - worldsayshi 2y ago
      It's so interesting that you can tell it to think about a thing and then it does that. Sounds like a nice prompt to run automatically on PRs.
MostlyStable 2y ago
This is, I think, a better way to think about LLM mistakes compared to the usual "hallucinations". I think of them as similar to human optical illusions. There are things about the human visual cortex (and also other sensory systems, see the McGurk Effect [0]), that, when presented with certain kinds of inputs, will consistently produce wrong interpretations/outputs. Even when we are 100% ware of the issue, we can't prevent our brains from generating the incorrect interpretation. LLMs seem to have similar issues along dramatically different axes, axes that humans are not used to seeing these kinds of mistakes; where nearly no human would make this kind of mistake and so we interpret it (in my opinion incorrectly) as lack of ability or intelligence. Because these are engineered systems, we may figure out ways to solve these problems (although I personally think the best we will ever do is decrease their prevalence), but more important is probably learning to recognize the places that LLMs are likely to make these errors, and, as your comment suggests, design work flows and systems that can deal with them. [0] https://youtu.be/2k8fHR9jKVM https://youtu.be/2k8fHR9jKVM
- admiralrohan 2y ago
  Hallucinating is fine but overconfidence is the problem. But I heard it's not an easy problem to solve.
  - jacksnipe 2y ago
    Unfortunately, in the system most of us work in today, I think overconfidence is an intelligent behavior.
    - onemoresoop 2y ago
      I find it extremely dumb to see overconfident people that really have nothing special about them or are even incompetent. These people are not contributing positively to the system, quite on the contrary.
      - xwolfi 2y ago
        But we don't work for the system, fundamentally, we work for ourselves, and the system incentivizes us to work for it by aligning our constraints: if you work that direction, you'll get that reward. Overconfident people ofc do not contribute positively to the system, but they skew the system reward's calculation towards them: I swear I've done that work in that direction, where's my reward ? In a sense, they are extremely successful: they managed to do very low effort, get very high reward, help themselves like all of us but at a much better profit margin, by sacrificing a system that, let's be honest, none of us care about really. Your problem maybe, is that you swallowed the little BS the system fed you while incentivizing you: that the system matters more than yourself, at least at a greater extent than healthy ? And you see the same thing with AI: these things convince people so deeply of their intelligence that it blew to such proportion that NVidia is now worth trillions. I had a colleague mumbling yesterday that his wife now speaks more with ChatGPT than him. Overconfidence is a positive attribute... for oneself.
vanschelven 2y ago
Nevermind designing _systems_ that account for this, even just debugging such errors is much harder than ones you create yourself: https://www.bugsink.com/blog/copilot-induced-crash/ https://www.bugsink.com/blog/copilot-induced-crash/
- fragmede 2y ago
  For that case, it sounds more like having your tools commit for you after each change, as is the default for Aider, is the real winner. "git log -p" would have exposed that crazy import in minutes instead of hours. commit early, commit often.
  - zahlman 2y ago
    FTA: > Note that it took me about two hours to debug this, despite the problem being freshly introduced. (Because I hadn’t committed yet, and had established that the previous commit was fine, I could have just run git diff to see what had changed). > In fact, I did run git diff and git diff --staged multiple times. But who would think to look at the import statements? The import statement is the last place you’d expect a bug to be introduced.
    - fragmede 2y ago
      git diff != git log. To expand on that, the problem with only having git diff is there's no way to go backwards halfway. You can't step backwards in time until you get to the bad commit just before the good commit, and then do a precise diff between the two. (aka git bisect) Reviewing 300 lines out of git diff and trying to find the bug somewhere in there is harder than when there are only 10.
  - danenania 2y ago
    I’m working an AI coding agent[1], and all changes accumulate in a sandbox by default that is isolated from the project. Auto-commit is also enabled (by default) when you do apply the changes to your project, but I think keeping them separated until you review is better for higher stakes work and goes a long way to protect you from stray edits getting left behind. 1 - https://github.com/plandex-ai/plandex https://github.com/plandex-ai/plandex
dpacmittal 2y ago
I just prompted cursor to remove a string from a svelte app. It created a boolean variable showString, set it as false and then proceeded to use that to hide the string

tharkun__ 2y ago

I don't think that's universally true. We have different humans with different levels of ability to catch errors. I see that with my teams. Some people can debug. Some can't. Some people can write tests. Some can't. Some people can catch stuff in reviews. Some can't. I asked Sonnet 3.7 in Cursor to fix a failing test. While it made the necessary fix, it also updated a hard-coded expected constant to instead be computed using the same algorithm as the original file, instead of preserving the constant as the test was originally written. Guess what? Guess the number of times I had to correct this from humans doing it in their tests over my career! And guess where the models learned the bad behavior from.

__MatrixMan__ 2y ago
I agree. I've been been struck by how remarkably understandable the errors are. It's quite often something that I'd have done myself if I wasn't paying attention to the right thing.
fn-mote 2y ago
> Some people can debug. Some can't. Some people can write tests. Some can't. Wait… really? No way do I want to work with someone who can’t debug or write tests. I thought those were entry stakes to the profession. People whose skills you use in other ways because they are more productive? Maybe. But still. Clean up after yourself. It’s something that should be learned in the apprentice phase.
- hobs 2y ago
  Keyword want - most people don't control who their peers are, and complaining to your boss doesn't get you that far, especially when said useless boss is fostering said useless person.
- tharkun__ 2y ago
  Like my sibling says, you can't always choose. That's one side of that coin. The other is: Some people are naturally good at writing "green field" (or re-writing everything) and do produce actual good software. But these same people, which you do want to keep around if that's the best you can get, are next to useless when you throw a customer reported bug at them. Takes them ages to figure anything out and they go down endless rabbit holes chasing the wrong path for hours. You also have people that are super awesome at debugging. They have knack for seeing some brokenness and having the right idea or an idea of the right direction to investigate in right away, can apply the scientific method to test their theories and have the bug fixed in the time it take one of these other people to go down even a single of the rabbit holes they will go down. But these same people in some cases are next to useless if you ask them to properly structure a new green field feature or rewrite parts of something to use some new library coz the old one is no longer maintained or something and digging through said new library and how it works. Both of these types of people are not bad in and of themselves. Especially if you can't get the unicorns that can do all of these things well (or well enough), e.g. because your company can't or won't pay for it or only for a few of them, which they might call "Staff level". And you'd be amazed how easy it is to get quite a few review comments in for even Staff level people if you basically ignore their actual code and just jump right into the tests. It's a pet peeve of mine. I start with the tests and go from there when reviewing :) What you really don't want is if someone is not good at any of these of course.
- groby_b 2y ago
  > No way do I want to work with someone who can’t debug or write tests. I thought those were entry stakes to the profession. Those are almost entry stakes at tier-one companies. (There are still people who can't, it's just much less common) In your average CRUD/enterprise automation/one-off shellscript factory, the state of skills is... not fun. There's a reason there's the old saw of "some people have twenty years experience, some have the same year 20 times over". People learn & grow when they are challenged to, and will mostly settle at acquiring the minimum skill level that lets them do their particular work. And since we as an industry decided to pretend we're a "science", not skills based, we don't have a decent apprenticeship system that would force a minimum bar. And whenever we discuss LLMs and how they might replace software engineering, I keep remembering that they'll be prompted by the people who set that hiring bar and thought they did well.
skerit 2y ago
Claude Sonnet 3.7 really, really loves to rewrite tests so they'll pass. I've had it happen many times in a claude-code session, I had to add this to each request (though it did not fix it 100%) - Never disable, skip, or comment out failing unit tests. If a unit test fails, fix the root cause of the exception. - Never change the unit test in such a way that it avoids testing the failing feature (e.g., by removing assertions, adding empty try/catch blocks, or making tests trivial). - Do not mark tests with @Ignore or equivalent annotations. - Do not introduce conditional logic that skips test cases under certain conditions. - Always ensure the unit test continues to properly validate the intended functionality.
- _ea1k 2y ago
  I'm guessing this is a side effect of mistakes in the reinforcement learning face. It'd be really easy to build a reward model that favors passing tests, without properly measuring the quality of those tests.

sorokod 2y ago

You may find this interesting: "AI Mistakes Are Very Different from Human Mistakes" https://www.schneier.com/blog/archives/2025/01/ai-mistakes-are-very-different-from-human-mistakes.html https://www.schneier.com/blog/archives/2025/01/ai-mistakes-a...

woopwoop 2y ago

Agree, but I would point out that the errors that I make are selected on the fact that I don't notice I'm making them, which tips the scale toward LLM errors being not as bad.

worldsayshi 2y ago
Yeah it's the reason pair programming is nice. Now the bugs need to pass two filters instead of one. Although I suppose LLM's aren't that good at catching my bugs without me pointing them out.
- diggan 2y ago
  I've found both various ChatGPT and Claude to be pretty good at finding unknown bugs, but you need a somewhat hefty prompt. Personally I use a prompt that goes something like this (shortened here): "Go through all the code below and analyze everything it's doing step-by-step. Then try to explain the overall purpose of the code based on your analysis. Then think through all the edge-cases and tradeoffs based on the purpose, and finally go through the code again and see if you can spot anything weird" Basically, I tried to think of what I do when I try to spot bugs in code, then I just wrote a reusable prompt that basically repeats my own process.
  - worldsayshi 2y ago
    It's so interesting that you can tell it to think about a thing and then it does that. Sounds like a nice prompt to run automatically on PRs.

MostlyStable 2y ago

This is, I think, a better way to think about LLM mistakes compared to the usual "hallucinations". I think of them as similar to human optical illusions. There are things about the human visual cortex (and also other sensory systems, see the McGurk Effect [0]), that, when presented with certain kinds of inputs, will consistently produce wrong interpretations/outputs. Even when we are 100% ware of the issue, we can't prevent our brains from generating the incorrect interpretation. LLMs seem to have similar issues along dramatically different axes, axes that humans are not used to seeing these kinds of mistakes; where nearly no human would make this kind of mistake and so we interpret it (in my opinion incorrectly) as lack of ability or intelligence. Because these are engineered systems, we may figure out ways to solve these problems (although I personally think the best we will ever do is decrease their prevalence), but more important is probably learning to recognize the places that LLMs are likely to make these errors, and, as your comment suggests, design work flows and systems that can deal with them. [0] https://youtu.be/2k8fHR9jKVM https://youtu.be/2k8fHR9jKVM

admiralrohan 2y ago
Hallucinating is fine but overconfidence is the problem. But I heard it's not an easy problem to solve.
- jacksnipe 2y ago
  Unfortunately, in the system most of us work in today, I think overconfidence is an intelligent behavior.
  - onemoresoop 2y ago
    I find it extremely dumb to see overconfident people that really have nothing special about them or are even incompetent. These people are not contributing positively to the system, quite on the contrary.
    - xwolfi 2y ago
      But we don't work for the system, fundamentally, we work for ourselves, and the system incentivizes us to work for it by aligning our constraints: if you work that direction, you'll get that reward. Overconfident people ofc do not contribute positively to the system, but they skew the system reward's calculation towards them: I swear I've done that work in that direction, where's my reward ? In a sense, they are extremely successful: they managed to do very low effort, get very high reward, help themselves like all of us but at a much better profit margin, by sacrificing a system that, let's be honest, none of us care about really. Your problem maybe, is that you swallowed the little BS the system fed you while incentivizing you: that the system matters more than yourself, at least at a greater extent than healthy ? And you see the same thing with AI: these things convince people so deeply of their intelligence that it blew to such proportion that NVidia is now worth trillions. I had a colleague mumbling yesterday that his wife now speaks more with ChatGPT than him. Overconfidence is a positive attribute... for oneself.

vanschelven 2y ago

Nevermind designing _systems_ that account for this, even just debugging such errors is much harder than ones you create yourself: https://www.bugsink.com/blog/copilot-induced-crash/ https://www.bugsink.com/blog/copilot-induced-crash/

fragmede 2y ago
For that case, it sounds more like having your tools commit for you after each change, as is the default for Aider, is the real winner. "git log -p" would have exposed that crazy import in minutes instead of hours. commit early, commit often.
- zahlman 2y ago
  FTA: > Note that it took me about two hours to debug this, despite the problem being freshly introduced. (Because I hadn’t committed yet, and had established that the previous commit was fine, I could have just run git diff to see what had changed). > In fact, I did run git diff and git diff --staged multiple times. But who would think to look at the import statements? The import statement is the last place you’d expect a bug to be introduced.
  - fragmede 2y ago
    git diff != git log. To expand on that, the problem with only having git diff is there's no way to go backwards halfway. You can't step backwards in time until you get to the bad commit just before the good commit, and then do a precise diff between the two. (aka git bisect) Reviewing 300 lines out of git diff and trying to find the bug somewhere in there is harder than when there are only 10.
- danenania 2y ago
  I’m working an AI coding agent[1], and all changes accumulate in a sandbox by default that is isolated from the project. Auto-commit is also enabled (by default) when you do apply the changes to your project, but I think keeping them separated until you review is better for higher stakes work and goes a long way to protect you from stray edits getting left behind. 1 - https://github.com/plandex-ai/plandex https://github.com/plandex-ai/plandex

dpacmittal 2y ago

I just prompted cursor to remove a string from a svelte app. It created a boolean variable showString, set it as false and then proceeded to use that to hide the string