10 ms·

There is a certain amount of irony that people try really hard to say that hallucinations are not a big problem anymore and then a company that would benefit fr

There is a certain amount of irony that people try really hard to say that hallucinations are not a big problem anymore and then a company that would benefit from that narrative gets directly hurt by it. Which of course they are going to try to brush it all away. Better than admitting that this problem very much still exists and isn’t going away anytime soon.

ModernMech 1y ago
It's a huge problem. I just can't get past it and I get burned by it every time I try one of these products. Cursor in particular was one of the worst; the very first time I allowed it to look at my codebase, it hallucinated a missing brace (my code parsed fine), "helpfully" inserted it, and then proceeded to break everything. How am I supposed to trust and work with such a tool? To me, it seems like the equivalent of lobbing a live hand grenade into your codebase. Don't get me wrong, I use AI every day, but it's mostly as a localized code complete or to help me debug tricky issues. Meaning I've written and understand the code myself, and the AI is there to augment my abilities. AI works great if it's used as a deductive tool. Where it runs into issues is when it's used inductively, to create things that aren't there. When it does this, I feel the hallucinations can be off the charts -- inventing APIs, function names, entire libraries, and even entire programming languages on occasion. The AI is more than happy to deliver any kind of information you want, no matter how wrong it is. AI is not a tool, it's a tiny Kafkaesque bureaucracy inside of your codebase. Does it work today? Yes! Why does it work? Who can say! Will it work tomorrow? Fingers crossed!
- mediaman 1y ago
  I'd add that the deductive abilities translate to well-defined spec. I've found it does well when I know what APIs I want it to use, and what general algorithmic approaches I want (which are still sometimes brainstormed separately with an AI, but not within the codebase). I provide it a numbered outline of the desired requirements and approach to take, and it usually does a good job. It does poorly without heavy instruction, though, especially with anything more than toy projects. Still a valuable tool, but far from the dreamy autonomous geniuses that they often get described as.
- simeonww 1y ago
  [flagged]
- Mountain_Skies 1y ago
  Versioning in source control for even personal projects just got far more important.
  - AdrianEGraphene 1y ago
    It's wild how people write without version control... Maybe I'm missing something.
    - chrisweekly 1y ago
      yeah, "git init" (if you haven't botherered to create a template repo) is not exactly cumbersome.
  - o11c 1y ago
    Thankfully modern source control doesn't reuse user-supplied filenames for its internals. In the dark ages, I destroyed more than one checkout using commands of the form: find -name '*somepattern*' -exec clobbering command ...
- yodsanklai 1y ago
  You're not supposed to trust the tool, you're supposed to review and rework the code before submitting for external review. I use AI for rather complex tasks. It's impressive. It can make a bunch of non-trivial changes to several files, and have the code compile without warnings. But I need to iterate a few times so that the code looks like what I want. That being said, I also lose time pretty regularly. There's a learning curve, and the tool would be much more useful if it was faster. It takes a few minutes to make changes, and there may be several iterations.
  - deleted 1y ago
    [deleted]
  - ryandrake 1y ago
    > You're not supposed to trust the tool, you're supposed to review and rework the code before submitting for external review. It sounds like the guys in this article should not have trusted AI to go fully open loop on their customer support system. That should be well understood by all "customers" of AI. You can't trust it to do anything correctly without human feedback/review and human quality control.
  - schmichael 1y ago
    > You're not supposed to trust the tool This is just an incredible statement. I can't think of another development tool we'd say this about. I'm not saying you're wrong, or that it's wrong to have tools we can't just, just... wow... what a sea change.
    - theonething 1y ago
      > I can't think of another development tool we'd say this about. Because no other dev tool actually generates unique code like AI does. So you treat it like the other components of your team that generates code, the other developers. Do you trust other developers to write good code without mistakes without getting it reviewed by others. Of course not.
      - 1y ago
- theonething 1y ago
  > it hallucinated a missing brace (my code parsed fine), "helpfully" inserted it, and then proceeded to break everything. Your tone is rather hyperbolic here, making it sound like an extra brace resulted in a disaster. It didn't. It was easy to detect and easy to fix. Not a big deal.
  - ModernMech 1y ago
    It's not a big deal in the sense that it's easily reversed, but it is a big deal in that it means the tool is unpredictably unhelpful. Of the properties that good tools in my workflow possess, "unpredictably unhelpful" does not make the top 100. When a tool starts confidently inserting random wrong code into my 100% correct code, there's not much more I need to see to know it's not a tool for me. That's less like a tool and more like a vandal. That's not something I need in my toolbox, and I'm certainly not going to replace my other tools with it.
  - nothrabannosir 1y ago
    https://dwheeler.com/essays/apple-goto-fail.html https://dwheeler.com/essays/apple-goto-fail.html
- skissane 1y ago
  > the very first time I allowed it to look at my codebase, it hallucinated a missing brace (my code parsed fine), "helpfully" inserted it, and then proceeded to break everything. This is not an inherent flaw of LLMs, rather it is a flaw of a particular implementation-if you use guided sampling, so during sampling you only consider tokens allowed by the programming language grammar at that position, it becomes impossible for the LLM to generate ungrammatical output > When it does this, I feel the hallucinations can be off the charts -- inventing APIs, function names, entire libraries, They can use guided sampling for this too - if you know the set of function names which exist in the codebase and its dependencies, you can reject tokens that correspond to non-existent function names during sampling Another approach, instead of or as well as guided sampling, is to use an agent with function calling - so the LLM can try compiling the modified code itself, and then attempt to recover from any errors which occur.
cryptoegorophy 1y ago
I think that’s why Apple is very slow at rolling out AI if it ever actually will. Downside is way too big than the upside.
- sillyfluke 1y ago
  They already rolled out an "AI" product. Got humiliated pretty bad, and rolled it back. [0] [0] https://www.bbc.com/news/articles/cq5ggew08eyo https://www.bbc.com/news/articles/cq5ggew08eyo
  - furyofantares 1y ago
    They also have text thread and email summaries. I still think it counts as a slow rollout.
  - VenturingVole 1y ago
    They had an opportunity to actually adapt, to embrace getting rapid feedback/iterating: But they are not equipped for it culturally. Major lost opportunity as it could have been a driver of internal change. I'm certain they'll get it right soon enough though. People were writing off Google in terms of AI until this year.. and oh how attitudes have changed.
    - tonyhart7 1y ago
      "to embrace getting rapid feedback/iterating" that's the problem noo?? big company is sucks at that, you cant do that in certain company because sometimes its just not possible
    - skinkestek 1y ago
      > People were writing off Google in terms of AI until this year.. and oh how attitudes have changed. Just give Google a year or two. Google has a pretty amazing history of both messing up products generally and especially "ai like" things, including search. (Yes I used to defend Google until a few years ago.)
- 1y ago
anonzzzies 1y ago
Did anyone say that? They are an issue everywhere, including for code. But with code at least I can have tooling to automatically check and feed back that it hallucinated libraries, functions etc, but with just normal research / problems there is no such thing and you will spend a lot of time verifying everything.
- manmal 1y ago
  You get some superficial checking by the compiler and test cases, but hallucinations that pass both are still an issue.
  - anonzzzies 1y ago
    Absolutely, but at least you have some lines of defence while with real world info you have nothing. And the most offending stuff like importing a package that doesn't exist or using a function that doesn't exist does get caught and can be auto fixed.
    - skyfaller 1y ago
      Such errors can be caught and auto-fixed for now, because LLMs haven't yet rotted the code that catches and auto-fixes errors. If slop makes it into your compiler etc., I wouldn't count on that being true in the future.
- threeseed 1y ago
  I use Scala which has arguably the best compiler/type system with Cursor. There is no world in which a compiler or tooling will save you from the absolute mayhem it can do. I’ve had it routinely try to re-implement third party libraries, modify code unrelated to what it was asked, quietly override functions etc. It’s like a developer who is on LSD.
  - Terr_ 1y ago
    Yeah, everyone wanted a thinking machine, but the best we can do right now is a dreaming machine... And dreams don't have to make sense.
lynguist 1y ago
https://www.anthropic.com/research/tracing-thoughts-language-model https://www.anthropic.com/research/tracing-thoughts-language... The section about hallucinations is deeply relevant. Namely, Claude sometimes provides a plausible but incorrect chain-of-thought reasoning when its “true” computational path isn’t available. The model genuinely believes it’s giving a correct reasoning chain, but the interpretability microscope reveals it is constructing symbolic arguments backward from a conclusion. https://en.wikipedia.org/wiki/On_Bullshit https://en.wikipedia.org/wiki/On_Bullshit This empirically confirms the “theory of bullshit” as a category distinct from lying. It suggests that “truth” emerges secondarily to symbolic coherence and plausibility. This means knowledge itself is fundamentally symbolic-social, not merely correspondence to external fact. Knowledge emerges from symbolic coherence, linguistic agreement, and social plausibility rather than purely from logical coherence or factual correctness.
- jmaker 1y ago
  I haven’t used Cursor yet. Some colleagues have and seemed happy. I’ve had GitHub Copilot on for what feels like a couple years, a few days ago VS Code was extended to provide an agentic workflow, MCP, bring-your-own-key, it interprets instructions in a codebase. But the UX and the outputs are bad in over 3/4 of cases. It’s a nuisance to me. It injects bad code even though it has the full context. Is Cursor genuinely any better? To me it feels like people that benefit from or at least enjoy that sort of assistance and I solve vastly different problems and code very differently. I’ve done exhausting code reviews on juniors’ and middles’ PRs but what I’ve been feeling lately is that I’m reviewing changes introduced by a very naive poster. It doesn’t even type-check. Regardless of whether it’s Claude 3.7, o1, o3-mini, or a few models from Hugging Face. I don’t understand how people find that useful. Yesterday I literally wasted half an hour for a test suite setup a colleague of mine introduced to the codebase that wasn’t good, and I tried delegating that fix to several of the Copilot models. All of them missed the point, some even introduced security vulnerabilities in the process invalidating JWT validation, I tried “vide coding” it till it works, until I gave up in frustration and just used an ordinary search engine, which led me to the docs, in which I immediately found the right knob. I reverted all that crap and did the simple and correct thing. So my conclusion was simple: vibe coding and LLMs made the codebase unnecessarily more complicated and wasted my time. How on earth do people code whole apps with that?
_jonas 1y ago
I see this fallacy often too. My company provides hallucination detection software: https://cleanlab.ai/tlm/ https://cleanlab.ai/tlm/ But we somehow end up in sales meetings where the person who requested the meeting claims their AI does not hallucinate ...
learningstud 1y ago
People hallucinate all the time out of pressure or habit. We don't need AI for that. It's hard to tell most people from AI. Most people would fail Turing tests as subjects.

ModernMech 1y ago

It's a huge problem. I just can't get past it and I get burned by it every time I try one of these products. Cursor in particular was one of the worst; the very first time I allowed it to look at my codebase, it hallucinated a missing brace (my code parsed fine), "helpfully" inserted it, and then proceeded to break everything. How am I supposed to trust and work with such a tool? To me, it seems like the equivalent of lobbing a live hand grenade into your codebase. Don't get me wrong, I use AI every day, but it's mostly as a localized code complete or to help me debug tricky issues. Meaning I've written and understand the code myself, and the AI is there to augment my abilities. AI works great if it's used as a deductive tool. Where it runs into issues is when it's used inductively, to create things that aren't there. When it does this, I feel the hallucinations can be off the charts -- inventing APIs, function names, entire libraries, and even entire programming languages on occasion. The AI is more than happy to deliver any kind of information you want, no matter how wrong it is. AI is not a tool, it's a tiny Kafkaesque bureaucracy inside of your codebase. Does it work today? Yes! Why does it work? Who can say! Will it work tomorrow? Fingers crossed!

mediaman 1y ago
I'd add that the deductive abilities translate to well-defined spec. I've found it does well when I know what APIs I want it to use, and what general algorithmic approaches I want (which are still sometimes brainstormed separately with an AI, but not within the codebase). I provide it a numbered outline of the desired requirements and approach to take, and it usually does a good job. It does poorly without heavy instruction, though, especially with anything more than toy projects. Still a valuable tool, but far from the dreamy autonomous geniuses that they often get described as.
simeonww 1y ago
[flagged]
Mountain_Skies 1y ago
Versioning in source control for even personal projects just got far more important.
- AdrianEGraphene 1y ago
  It's wild how people write without version control... Maybe I'm missing something.
  - chrisweekly 1y ago
    yeah, "git init" (if you haven't botherered to create a template repo) is not exactly cumbersome.
- o11c 1y ago
  Thankfully modern source control doesn't reuse user-supplied filenames for its internals. In the dark ages, I destroyed more than one checkout using commands of the form: find -name '*somepattern*' -exec clobbering command ...
yodsanklai 1y ago
You're not supposed to trust the tool, you're supposed to review and rework the code before submitting for external review. I use AI for rather complex tasks. It's impressive. It can make a bunch of non-trivial changes to several files, and have the code compile without warnings. But I need to iterate a few times so that the code looks like what I want. That being said, I also lose time pretty regularly. There's a learning curve, and the tool would be much more useful if it was faster. It takes a few minutes to make changes, and there may be several iterations.
- deleted 1y ago
  [deleted]
- ryandrake 1y ago
  > You're not supposed to trust the tool, you're supposed to review and rework the code before submitting for external review. It sounds like the guys in this article should not have trusted AI to go fully open loop on their customer support system. That should be well understood by all "customers" of AI. You can't trust it to do anything correctly without human feedback/review and human quality control.
- schmichael 1y ago
  > You're not supposed to trust the tool This is just an incredible statement. I can't think of another development tool we'd say this about. I'm not saying you're wrong, or that it's wrong to have tools we can't just, just... wow... what a sea change.
  - theonething 1y ago
    > I can't think of another development tool we'd say this about. Because no other dev tool actually generates unique code like AI does. So you treat it like the other components of your team that generates code, the other developers. Do you trust other developers to write good code without mistakes without getting it reviewed by others. Of course not.
    - 1y ago
theonething 1y ago
> it hallucinated a missing brace (my code parsed fine), "helpfully" inserted it, and then proceeded to break everything. Your tone is rather hyperbolic here, making it sound like an extra brace resulted in a disaster. It didn't. It was easy to detect and easy to fix. Not a big deal.
- ModernMech 1y ago
  It's not a big deal in the sense that it's easily reversed, but it is a big deal in that it means the tool is unpredictably unhelpful. Of the properties that good tools in my workflow possess, "unpredictably unhelpful" does not make the top 100. When a tool starts confidently inserting random wrong code into my 100% correct code, there's not much more I need to see to know it's not a tool for me. That's less like a tool and more like a vandal. That's not something I need in my toolbox, and I'm certainly not going to replace my other tools with it.
- nothrabannosir 1y ago
  https://dwheeler.com/essays/apple-goto-fail.html https://dwheeler.com/essays/apple-goto-fail.html
skissane 1y ago
> the very first time I allowed it to look at my codebase, it hallucinated a missing brace (my code parsed fine), "helpfully" inserted it, and then proceeded to break everything. This is not an inherent flaw of LLMs, rather it is a flaw of a particular implementation-if you use guided sampling, so during sampling you only consider tokens allowed by the programming language grammar at that position, it becomes impossible for the LLM to generate ungrammatical output > When it does this, I feel the hallucinations can be off the charts -- inventing APIs, function names, entire libraries, They can use guided sampling for this too - if you know the set of function names which exist in the codebase and its dependencies, you can reject tokens that correspond to non-existent function names during sampling Another approach, instead of or as well as guided sampling, is to use an agent with function calling - so the LLM can try compiling the modified code itself, and then attempt to recover from any errors which occur.

cryptoegorophy 1y ago

I think that’s why Apple is very slow at rolling out AI if it ever actually will. Downside is way too big than the upside.

sillyfluke 1y ago
They already rolled out an "AI" product. Got humiliated pretty bad, and rolled it back. [0] [0] https://www.bbc.com/news/articles/cq5ggew08eyo https://www.bbc.com/news/articles/cq5ggew08eyo
- furyofantares 1y ago
  They also have text thread and email summaries. I still think it counts as a slow rollout.
- VenturingVole 1y ago
  They had an opportunity to actually adapt, to embrace getting rapid feedback/iterating: But they are not equipped for it culturally. Major lost opportunity as it could have been a driver of internal change. I'm certain they'll get it right soon enough though. People were writing off Google in terms of AI until this year.. and oh how attitudes have changed.
  - tonyhart7 1y ago
    "to embrace getting rapid feedback/iterating" that's the problem noo?? big company is sucks at that, you cant do that in certain company because sometimes its just not possible
  - skinkestek 1y ago
    > People were writing off Google in terms of AI until this year.. and oh how attitudes have changed. Just give Google a year or two. Google has a pretty amazing history of both messing up products generally and especially "ai like" things, including search. (Yes I used to defend Google until a few years ago.)
1y ago

anonzzzies 1y ago

Did anyone say that? They are an issue everywhere, including for code. But with code at least I can have tooling to automatically check and feed back that it hallucinated libraries, functions etc, but with just normal research / problems there is no such thing and you will spend a lot of time verifying everything.

manmal 1y ago
You get some superficial checking by the compiler and test cases, but hallucinations that pass both are still an issue.
- anonzzzies 1y ago
  Absolutely, but at least you have some lines of defence while with real world info you have nothing. And the most offending stuff like importing a package that doesn't exist or using a function that doesn't exist does get caught and can be auto fixed.
  - skyfaller 1y ago
    Such errors can be caught and auto-fixed for now, because LLMs haven't yet rotted the code that catches and auto-fixes errors. If slop makes it into your compiler etc., I wouldn't count on that being true in the future.
threeseed 1y ago
I use Scala which has arguably the best compiler/type system with Cursor. There is no world in which a compiler or tooling will save you from the absolute mayhem it can do. I’ve had it routinely try to re-implement third party libraries, modify code unrelated to what it was asked, quietly override functions etc. It’s like a developer who is on LSD.
- Terr_ 1y ago
  Yeah, everyone wanted a thinking machine, but the best we can do right now is a dreaming machine... And dreams don't have to make sense.

lynguist 1y ago

https://www.anthropic.com/research/tracing-thoughts-language-model https://www.anthropic.com/research/tracing-thoughts-language... The section about hallucinations is deeply relevant. Namely, Claude sometimes provides a plausible but incorrect chain-of-thought reasoning when its “true” computational path isn’t available. The model genuinely believes it’s giving a correct reasoning chain, but the interpretability microscope reveals it is constructing symbolic arguments backward from a conclusion. https://en.wikipedia.org/wiki/On_Bullshit https://en.wikipedia.org/wiki/On_Bullshit This empirically confirms the “theory of bullshit” as a category distinct from lying. It suggests that “truth” emerges secondarily to symbolic coherence and plausibility. This means knowledge itself is fundamentally symbolic-social, not merely correspondence to external fact. Knowledge emerges from symbolic coherence, linguistic agreement, and social plausibility rather than purely from logical coherence or factual correctness.

jmaker 1y ago
I haven’t used Cursor yet. Some colleagues have and seemed happy. I’ve had GitHub Copilot on for what feels like a couple years, a few days ago VS Code was extended to provide an agentic workflow, MCP, bring-your-own-key, it interprets instructions in a codebase. But the UX and the outputs are bad in over 3/4 of cases. It’s a nuisance to me. It injects bad code even though it has the full context. Is Cursor genuinely any better? To me it feels like people that benefit from or at least enjoy that sort of assistance and I solve vastly different problems and code very differently. I’ve done exhausting code reviews on juniors’ and middles’ PRs but what I’ve been feeling lately is that I’m reviewing changes introduced by a very naive poster. It doesn’t even type-check. Regardless of whether it’s Claude 3.7, o1, o3-mini, or a few models from Hugging Face. I don’t understand how people find that useful. Yesterday I literally wasted half an hour for a test suite setup a colleague of mine introduced to the codebase that wasn’t good, and I tried delegating that fix to several of the Copilot models. All of them missed the point, some even introduced security vulnerabilities in the process invalidating JWT validation, I tried “vide coding” it till it works, until I gave up in frustration and just used an ordinary search engine, which led me to the docs, in which I immediately found the right knob. I reverted all that crap and did the simple and correct thing. So my conclusion was simple: vibe coding and LLMs made the codebase unnecessarily more complicated and wasted my time. How on earth do people code whole apps with that?

_jonas 1y ago

I see this fallacy often too. My company provides hallucination detection software: https://cleanlab.ai/tlm/ https://cleanlab.ai/tlm/ But we somehow end up in sales meetings where the person who requested the meeting claims their AI does not hallucinate ...

learningstud 1y ago

People hallucinate all the time out of pressure or habit. We don't need AI for that. It's hard to tell most people from AI. Most people would fail Turing tests as subjects.