6 ms·

It's a huge problem. I just can't get past it and I get burned by it every time I try one of these products. Cursor in particular was one of the worst; the very

It's a huge problem. I just can't get past it and I get burned by it every time I try one of these products. Cursor in particular was one of the worst; the very first time I allowed it to look at my codebase, it hallucinated a missing brace (my code parsed fine), "helpfully" inserted it, and then proceeded to break everything. How am I supposed to trust and work with such a tool? To me, it seems like the equivalent of lobbing a live hand grenade into your codebase. Don't get me wrong, I use AI every day, but it's mostly as a localized code complete or to help me debug tricky issues. Meaning I've written and understand the code myself, and the AI is there to augment my abilities. AI works great if it's used as a deductive tool. Where it runs into issues is when it's used inductively, to create things that aren't there. When it does this, I feel the hallucinations can be off the charts -- inventing APIs, function names, entire libraries, and even entire programming languages on occasion. The AI is more than happy to deliver any kind of information you want, no matter how wrong it is. AI is not a tool, it's a tiny Kafkaesque bureaucracy inside of your codebase. Does it work today? Yes! Why does it work? Who can say! Will it work tomorrow? Fingers crossed!

mediaman 2y ago
I'd add that the deductive abilities translate to well-defined spec. I've found it does well when I know what APIs I want it to use, and what general algorithmic approaches I want (which are still sometimes brainstormed separately with an AI, but not within the codebase). I provide it a numbered outline of the desired requirements and approach to take, and it usually does a good job. It does poorly without heavy instruction, though, especially with anything more than toy projects. Still a valuable tool, but far from the dreamy autonomous geniuses that they often get described as.
simeonww 2y ago
[flagged]
Mountain_Skies 2y ago
Versioning in source control for even personal projects just got far more important.
- AdrianEGraphene 2y ago
  It's wild how people write without version control... Maybe I'm missing something.
  - chrisweekly 2y ago
    yeah, "git init" (if you haven't botherered to create a template repo) is not exactly cumbersome.
- o11c 2y ago
  Thankfully modern source control doesn't reuse user-supplied filenames for its internals. In the dark ages, I destroyed more than one checkout using commands of the form: find -name '*somepattern*' -exec clobbering command ...
yodsanklai 2y ago
You're not supposed to trust the tool, you're supposed to review and rework the code before submitting for external review. I use AI for rather complex tasks. It's impressive. It can make a bunch of non-trivial changes to several files, and have the code compile without warnings. But I need to iterate a few times so that the code looks like what I want. That being said, I also lose time pretty regularly. There's a learning curve, and the tool would be much more useful if it was faster. It takes a few minutes to make changes, and there may be several iterations.
- deleted 2y ago
  [deleted]
- ryandrake 2y ago
  > You're not supposed to trust the tool, you're supposed to review and rework the code before submitting for external review. It sounds like the guys in this article should not have trusted AI to go fully open loop on their customer support system. That should be well understood by all "customers" of AI. You can't trust it to do anything correctly without human feedback/review and human quality control.
- schmichael 2y ago
  > You're not supposed to trust the tool This is just an incredible statement. I can't think of another development tool we'd say this about. I'm not saying you're wrong, or that it's wrong to have tools we can't just, just... wow... what a sea change.
  - theonething 2y ago
    > I can't think of another development tool we'd say this about. Because no other dev tool actually generates unique code like AI does. So you treat it like the other components of your team that generates code, the other developers. Do you trust other developers to write good code without mistakes without getting it reviewed by others. Of course not.
    - 2y ago
theonething 2y ago
> it hallucinated a missing brace (my code parsed fine), "helpfully" inserted it, and then proceeded to break everything. Your tone is rather hyperbolic here, making it sound like an extra brace resulted in a disaster. It didn't. It was easy to detect and easy to fix. Not a big deal.
- ModernMech 2y ago
  It's not a big deal in the sense that it's easily reversed, but it is a big deal in that it means the tool is unpredictably unhelpful. Of the properties that good tools in my workflow possess, "unpredictably unhelpful" does not make the top 100. When a tool starts confidently inserting random wrong code into my 100% correct code, there's not much more I need to see to know it's not a tool for me. That's less like a tool and more like a vandal. That's not something I need in my toolbox, and I'm certainly not going to replace my other tools with it.
- nothrabannosir 2y ago
  https://dwheeler.com/essays/apple-goto-fail.html https://dwheeler.com/essays/apple-goto-fail.html
skissane 2y ago
> the very first time I allowed it to look at my codebase, it hallucinated a missing brace (my code parsed fine), "helpfully" inserted it, and then proceeded to break everything. This is not an inherent flaw of LLMs, rather it is a flaw of a particular implementation-if you use guided sampling, so during sampling you only consider tokens allowed by the programming language grammar at that position, it becomes impossible for the LLM to generate ungrammatical output > When it does this, I feel the hallucinations can be off the charts -- inventing APIs, function names, entire libraries, They can use guided sampling for this too - if you know the set of function names which exist in the codebase and its dependencies, you can reject tokens that correspond to non-existent function names during sampling Another approach, instead of or as well as guided sampling, is to use an agent with function calling - so the LLM can try compiling the modified code itself, and then attempt to recover from any errors which occur.

mediaman 2y ago

I'd add that the deductive abilities translate to well-defined spec. I've found it does well when I know what APIs I want it to use, and what general algorithmic approaches I want (which are still sometimes brainstormed separately with an AI, but not within the codebase). I provide it a numbered outline of the desired requirements and approach to take, and it usually does a good job. It does poorly without heavy instruction, though, especially with anything more than toy projects. Still a valuable tool, but far from the dreamy autonomous geniuses that they often get described as.

simeonww 2y ago

[flagged]

Mountain_Skies 2y ago

Versioning in source control for even personal projects just got far more important.

AdrianEGraphene 2y ago
It's wild how people write without version control... Maybe I'm missing something.
- chrisweekly 2y ago
  yeah, "git init" (if you haven't botherered to create a template repo) is not exactly cumbersome.
o11c 2y ago
Thankfully modern source control doesn't reuse user-supplied filenames for its internals. In the dark ages, I destroyed more than one checkout using commands of the form: find -name '*somepattern*' -exec clobbering command ...

yodsanklai 2y ago

You're not supposed to trust the tool, you're supposed to review and rework the code before submitting for external review. I use AI for rather complex tasks. It's impressive. It can make a bunch of non-trivial changes to several files, and have the code compile without warnings. But I need to iterate a few times so that the code looks like what I want. That being said, I also lose time pretty regularly. There's a learning curve, and the tool would be much more useful if it was faster. It takes a few minutes to make changes, and there may be several iterations.

deleted 2y ago
[deleted]
ryandrake 2y ago
> You're not supposed to trust the tool, you're supposed to review and rework the code before submitting for external review. It sounds like the guys in this article should not have trusted AI to go fully open loop on their customer support system. That should be well understood by all "customers" of AI. You can't trust it to do anything correctly without human feedback/review and human quality control.
schmichael 2y ago
> You're not supposed to trust the tool This is just an incredible statement. I can't think of another development tool we'd say this about. I'm not saying you're wrong, or that it's wrong to have tools we can't just, just... wow... what a sea change.
- theonething 2y ago
  > I can't think of another development tool we'd say this about. Because no other dev tool actually generates unique code like AI does. So you treat it like the other components of your team that generates code, the other developers. Do you trust other developers to write good code without mistakes without getting it reviewed by others. Of course not.
  - 2y ago

theonething 2y ago

> it hallucinated a missing brace (my code parsed fine), "helpfully" inserted it, and then proceeded to break everything. Your tone is rather hyperbolic here, making it sound like an extra brace resulted in a disaster. It didn't. It was easy to detect and easy to fix. Not a big deal.

ModernMech 2y ago
It's not a big deal in the sense that it's easily reversed, but it is a big deal in that it means the tool is unpredictably unhelpful. Of the properties that good tools in my workflow possess, "unpredictably unhelpful" does not make the top 100. When a tool starts confidently inserting random wrong code into my 100% correct code, there's not much more I need to see to know it's not a tool for me. That's less like a tool and more like a vandal. That's not something I need in my toolbox, and I'm certainly not going to replace my other tools with it.
nothrabannosir 2y ago
https://dwheeler.com/essays/apple-goto-fail.html https://dwheeler.com/essays/apple-goto-fail.html

skissane 2y ago

> the very first time I allowed it to look at my codebase, it hallucinated a missing brace (my code parsed fine), "helpfully" inserted it, and then proceeded to break everything. This is not an inherent flaw of LLMs, rather it is a flaw of a particular implementation-if you use guided sampling, so during sampling you only consider tokens allowed by the programming language grammar at that position, it becomes impossible for the LLM to generate ungrammatical output > When it does this, I feel the hallucinations can be off the charts -- inventing APIs, function names, entire libraries, They can use guided sampling for this too - if you know the set of function names which exist in the codebase and its dependencies, you can reject tokens that correspond to non-existent function names during sampling Another approach, instead of or as well as guided sampling, is to use an agent with function calling - so the LLM can try compiling the modified code itself, and then attempt to recover from any errors which occur.