4 ms·

I disagree with the claim that "AI agents don't get lost." What I've observed instead is that they don't experience the sensation of feeling lost. Which is quit

I disagree with the claim that "AI agents don't get lost." What I've observed instead is that they don't experience the sensation of feeling lost. Which is quite different. This summer I spent quite a while using a coding agent to help me untangle a deep and complicated data processing pipeline. It had itself been built by agents, in a remarkably short amount of time. But it had also become clear that it was riddled with errors and was producing lots of bad data. What I quickly discovered was that upwards of half of my questions would receive very confidently wrong answers. And even once I had finally diagnosed whatever problem I was currently working on, it was difficult to trust the agent with any bug fixes. Since it was having an even harder time tracing data flows than I was (I'll take this chance to submit for your consideration that faster is not necessarily better), it was proving to be a bit of a monkey's paw. Yes, it would fix the exact bug I asked it to fix, but typically introduce new defects in the process. And yes, I was having this struggle with all of the latest & greatest models. I ultimately concluded that, in this codebase, the agent was indeed deeply, hopelessly lost. (edit: And probably this code got so bad in the first place because the agents that were used to build it had been lost for a while, but unable to recognize this problem and call their operators' attention to it.)

twosdai 1mo ago
This is a really good take. Thanks for sharing this. I havent been able to put to words how I have felt about agents being untrustworthy.
julien_dev 1mo ago
I'm curious about your exact case. In my experience I often had luck with evidence based approaches where the agent had to prove something first in order to make a statement (or write/do some tests first before making claims). I agree though that one has to be very careful when trying to "fix" things with agents in a big codebase without it introducing new defects.
Sivart13 1mo ago
"Confidently wrong" is definitely a standard behavior model for LLMs. I agree with the other reply that you're likely to get better results if it has some kind of test case to run that's more authoritative than its own reasoning.
rosenfeld 1mo ago
Yes, I've also noticed a few occasions when Claude would get it wrong, but in most cases it's able to find issues I didn't even consider because of a very deep analysis in a confusing (to humans) code. It detects some rare situations where a defect could exist. This happens when I explicitly ask Opus to review a PR and there's some harness around this ability, but I'm really impressed at how deep their analysis can be and correct as well. Of course, sometimes they're going to fail, but I don't see them getting lost often.

twosdai 1mo ago

This is a really good take. Thanks for sharing this. I havent been able to put to words how I have felt about agents being untrustworthy.

julien_dev 1mo ago

I'm curious about your exact case. In my experience I often had luck with evidence based approaches where the agent had to prove something first in order to make a statement (or write/do some tests first before making claims). I agree though that one has to be very careful when trying to "fix" things with agents in a big codebase without it introducing new defects.

Sivart13 1mo ago

"Confidently wrong" is definitely a standard behavior model for LLMs. I agree with the other reply that you're likely to get better results if it has some kind of test case to run that's more authoritative than its own reasoning.

rosenfeld 1mo ago

Yes, I've also noticed a few occasions when Claude would get it wrong, but in most cases it's able to find issues I didn't even consider because of a very deep analysis in a confusing (to humans) code. It detects some rare situations where a defect could exist. This happens when I explicitly ask Opus to review a PR and there's some harness around this ability, but I'm really impressed at how deep their analysis can be and correct as well. Of course, sometimes they're going to fail, but I don't see them getting lost often.