4 ms·

> I see. It seems I can't rename the directory I'm currently in. > Let's try a different approach. “Let’s try a different approach” always makes me nervous wi

by nojs 1y ago

> I see. It seems I can't rename the directory I'm currently in. > Let's try a different approach. “Let’s try a different approach” always makes me nervous with Claude too. It usually happens when something critical prevents the task being possible, and the correct response would be to stop and tell me the problem. But instead, Claude goes into paperclip mode making sure the task gets done no matter what.

ghm2180 1y ago
Yeah, it's "let's fix this no matter what" is really weird. In this mode everything becomes worst, it begins to comment code to make tests work, add pytest.mark.skip or xfail. It's almost like it was trained on data where it asks I gotta pick a tool to fix which one do I use and it was given ToNS of weird uncontrolled choices to train on that makes the code work, except instead of a scalpel its in home depot and it takes a random aisle and that makes it chooses anything from duct tape to super glue.
stingraycharles 1y ago
This is something that proper prompting can fix.
- syndeo 1y ago
  You seem to be getting downvoted, but I have to agree. I put it in my rules to ask me for confirmation before going down alternate paths like this, that it's critically important to not "give up" and undo its changes without first making a case to me about why it thinks it ought to do so. So far, at least, that seems to help.
  - stingraycharles 1y ago
    Yeah I don’t understand why, it seems like people think that “everything should be in the model”, which is just not true. Tuning the system prompt and user prompts to your needs is absolutely required before you’ll have a great time with these tools. Just take a look at zen-mcp to see what you can achieve with proper prompting and workflow management.
    - philipwhiuk 1y ago
      Because companies are claiming this stuff is intelligent
      - stingraycharles 1y ago
        Intelligence is one thing, context is the other. Prompts provide context and instructions and are tailored towards your needs.
- antonvs 1y ago
  Yes, but it's also something that proper training can fix, and that's the level at which the fix should probably be implemented. The current behavior amounts to something like "attempt to complete the task at all costs," which is unlikely to provide good results, and in practice, often doesn't.
  - samrus 1y ago
    But are LLMs the right models to even be able to learn such long horizon goals and how to not cheat at them? I feel like we need a new base model where the next token prodiction itself is dynamical and RL based to be able to handle this issue properly
    - antonvs 1y ago
      I was including RLHF in "training". And even the system prompt, really. If it's true that models can be prevented from spiraling into dead ends with "proper prompting" as the comment above claimed, then it's also true that this can be addressed earlier in the process. As it stands, this behavior isn't likely to be useful for any normal user, and it's certainly a blocker to "agentic" use.
      - samrus 1y ago
        The RLHF is happening too late i think. I think the reinforcement learning needs to be during the initial next token prodiction. On that note we need something to represent a complex world state than just language.
- samrus 1y ago
  Tgats running into the bitter lesson again. The model should genwralize and understand when its reached a road block in its higher level goal. The fact that it needs a uuman to decide that for it means it wont be able to do that on its own. This is critical for the software engineer tasks we are expecting agentic models to do
- 4ndrewl 1y ago
  "works with my prompt" is the new "works on my machine"
- voidUpdate 1y ago
  "you're holding the prompt wrong"
- rs186 1y ago
  Imagine an intern did the same thing, and you say "we just need better instructions". No! The intern needs to actually understand what they are doing. It is not just one more sentence "by the way, if this fails, check ...", because you can never enumerate all the possible situations (and you shouldn't even try), but instead you need to figure out why as soon as possible.
anonzzzies 1y ago
Yes, when Claude code says that, it usually means its going to attempt some hacky workaround that I do not want. Most commonly, in our case, if a client used one of those horrible orms like prisma or drizzle, it (claude) can never run the migrations and then wants to try to just manually go run the sql on the db, with 'interesting' outcomes.
- eclipxe 1y ago
  I've found both Prisma and Drizzle to be very nice and useful tools. Claude Code for me knows how to run my migrations for Prisma.
dawnerd 1y ago
On the flipside, GPT4.1 in Agent mode in VSCode is the outright laziest agent out there. You can give it a task to do, it'll tell you vaguely what needs to happen and ask if you want it to do it. Doesn't bother to verify its work, refuses to make use of tools. It's a joke frankly. Claude is too damn pushy to just make it work at all costs like you said, probably I'd guess to chew through tokens since they're bleeding money.
theshrike79 1y ago
I always think of LLMs as offshore teams with a strong cultural aversion to saying "no". They will do ANYTHING but tell the client they don't know what to do. Mocking the tests so far they're only testing the mocks? Yep! Rewriting the whole crap to do something different, but it compiles? Great! Stopping and actually saying "I can't solve this, please give more instructions"? NEVER!
- oc1 1y ago
  This is exactly how dumb these SOTA models feel. A real AI would stop and tell me it doesn't know for sure how to continue and that it needs more information from me instead of wild guessing. Sonnet, Opus, Gemini, Codex, they all have this fundamental error that they are unable to stop in case of uncertainty. Therefore producing shit solutions to problems i never had but now have..
  - cess11 1y ago
    These things are intelligent in the same way Aloy of Horizon fame is brave.
  - falcor84 1y ago
    I don't see a reason to believe that this is a "fundamental error". I think it's just an artifact of the way they are trained, and if the training penalized them more for taking a bad path than for stopping for instructions, then the situation would be different.
    - oc1 1y ago
      Fine. I'll cancel all other ai subscriptions if finally an ai doesn't aim to please me but behaves like a real professional. If your ai doesn't assume that my personality is trump-like and needs constant flattery . If you respect your users on a level that don't outsource RLHF to the lowest bider but pay actual senior (!) professionals in the respective fields you're training the model for. No Provider does this - they all went down the path to please some kind of low-iq population. Yes, i'm looking at you sama and fellows.
herbst 1y ago
It's like a anti pattern. My Claude basically always needs to try a different approach as soon as it runs commands. It's hard to tell when it starts to go berserk again or just trying all the system commands from 0 again. It does seem to constantly forget that is not Windows nor Ubuntu it's running on
alienbaby 1y ago
I see it a lot where it doesn't catch terminal output from it's own tests, and assumes it was wrong when it passed, so it goes through a everal iterations of trying simpler approaches until it succeeds in reading the terminal output. Lots of wasted time and tokens. (Using Claude sonnet with vscode where it consistently has issues reading output from terminal commands it executes)
justbees 1y ago
When Claude says “Let’s try a different approach” I immediately hit escape and give it more detailed instructions or try and steer it to the approach I want. It still has the previous context and then can use that with the more specific instructions. It really is like guiding a very smart intern or temp. You can't just let them run wild in the codebase. They need explicit parameters.
nancyminusone 1y ago
https://xkcd.com/416/ https://xkcd.com/416/
jofzar 1y ago
"let's try a different approach" 95% of the time involves deleting the file and trying to recreate it. It's mind-blowing it happens so often.
mvieira38 1y ago
I suspect this has to do with newer training procedures for reasoning models, where they are injecting stuff like "wait a minute" to force the model to reason more, as described in the Deepseek R1 training docs

by nojs 1y ago

ghm2180 1y ago

Yeah, it's "let's fix this no matter what" is really weird. In this mode everything becomes worst, it begins to comment code to make tests work, add pytest.mark.skip or xfail. It's almost like it was trained on data where it asks I gotta pick a tool to fix which one do I use and it was given ToNS of weird uncontrolled choices to train on that makes the code work, except instead of a scalpel its in home depot and it takes a random aisle and that makes it chooses anything from duct tape to super glue.

stingraycharles 1y ago

This is something that proper prompting can fix.

syndeo 1y ago
You seem to be getting downvoted, but I have to agree. I put it in my rules to ask me for confirmation before going down alternate paths like this, that it's critically important to not "give up" and undo its changes without first making a case to me about why it thinks it ought to do so. So far, at least, that seems to help.
- stingraycharles 1y ago
  Yeah I don’t understand why, it seems like people think that “everything should be in the model”, which is just not true. Tuning the system prompt and user prompts to your needs is absolutely required before you’ll have a great time with these tools. Just take a look at zen-mcp to see what you can achieve with proper prompting and workflow management.
  - philipwhiuk 1y ago
    Because companies are claiming this stuff is intelligent
    - stingraycharles 1y ago
      Intelligence is one thing, context is the other. Prompts provide context and instructions and are tailored towards your needs.
antonvs 1y ago
Yes, but it's also something that proper training can fix, and that's the level at which the fix should probably be implemented. The current behavior amounts to something like "attempt to complete the task at all costs," which is unlikely to provide good results, and in practice, often doesn't.
- samrus 1y ago
  But are LLMs the right models to even be able to learn such long horizon goals and how to not cheat at them? I feel like we need a new base model where the next token prodiction itself is dynamical and RL based to be able to handle this issue properly
  - antonvs 1y ago
    I was including RLHF in "training". And even the system prompt, really. If it's true that models can be prevented from spiraling into dead ends with "proper prompting" as the comment above claimed, then it's also true that this can be addressed earlier in the process. As it stands, this behavior isn't likely to be useful for any normal user, and it's certainly a blocker to "agentic" use.
    - samrus 1y ago
      The RLHF is happening too late i think. I think the reinforcement learning needs to be during the initial next token prodiction. On that note we need something to represent a complex world state than just language.
samrus 1y ago
Tgats running into the bitter lesson again. The model should genwralize and understand when its reached a road block in its higher level goal. The fact that it needs a uuman to decide that for it means it wont be able to do that on its own. This is critical for the software engineer tasks we are expecting agentic models to do
4ndrewl 1y ago
"works with my prompt" is the new "works on my machine"
voidUpdate 1y ago
"you're holding the prompt wrong"
rs186 1y ago
Imagine an intern did the same thing, and you say "we just need better instructions". No! The intern needs to actually understand what they are doing. It is not just one more sentence "by the way, if this fails, check ...", because you can never enumerate all the possible situations (and you shouldn't even try), but instead you need to figure out why as soon as possible.

anonzzzies 1y ago

Yes, when Claude code says that, it usually means its going to attempt some hacky workaround that I do not want. Most commonly, in our case, if a client used one of those horrible orms like prisma or drizzle, it (claude) can never run the migrations and then wants to try to just manually go run the sql on the db, with 'interesting' outcomes.

eclipxe 1y ago
I've found both Prisma and Drizzle to be very nice and useful tools. Claude Code for me knows how to run my migrations for Prisma.

dawnerd 1y ago

On the flipside, GPT4.1 in Agent mode in VSCode is the outright laziest agent out there. You can give it a task to do, it'll tell you vaguely what needs to happen and ask if you want it to do it. Doesn't bother to verify its work, refuses to make use of tools. It's a joke frankly. Claude is too damn pushy to just make it work at all costs like you said, probably I'd guess to chew through tokens since they're bleeding money.

theshrike79 1y ago

I always think of LLMs as offshore teams with a strong cultural aversion to saying "no". They will do ANYTHING but tell the client they don't know what to do. Mocking the tests so far they're only testing the mocks? Yep! Rewriting the whole crap to do something different, but it compiles? Great! Stopping and actually saying "I can't solve this, please give more instructions"? NEVER!

oc1 1y ago
This is exactly how dumb these SOTA models feel. A real AI would stop and tell me it doesn't know for sure how to continue and that it needs more information from me instead of wild guessing. Sonnet, Opus, Gemini, Codex, they all have this fundamental error that they are unable to stop in case of uncertainty. Therefore producing shit solutions to problems i never had but now have..
- cess11 1y ago
  These things are intelligent in the same way Aloy of Horizon fame is brave.
- falcor84 1y ago
  I don't see a reason to believe that this is a "fundamental error". I think it's just an artifact of the way they are trained, and if the training penalized them more for taking a bad path than for stopping for instructions, then the situation would be different.
  - oc1 1y ago
    Fine. I'll cancel all other ai subscriptions if finally an ai doesn't aim to please me but behaves like a real professional. If your ai doesn't assume that my personality is trump-like and needs constant flattery . If you respect your users on a level that don't outsource RLHF to the lowest bider but pay actual senior (!) professionals in the respective fields you're training the model for. No Provider does this - they all went down the path to please some kind of low-iq population. Yes, i'm looking at you sama and fellows.

herbst 1y ago

It's like a anti pattern. My Claude basically always needs to try a different approach as soon as it runs commands. It's hard to tell when it starts to go berserk again or just trying all the system commands from 0 again. It does seem to constantly forget that is not Windows nor Ubuntu it's running on

alienbaby 1y ago

I see it a lot where it doesn't catch terminal output from it's own tests, and assumes it was wrong when it passed, so it goes through a everal iterations of trying simpler approaches until it succeeds in reading the terminal output. Lots of wasted time and tokens. (Using Claude sonnet with vscode where it consistently has issues reading output from terminal commands it executes)

justbees 1y ago

When Claude says “Let’s try a different approach” I immediately hit escape and give it more detailed instructions or try and steer it to the approach I want. It still has the previous context and then can use that with the more specific instructions. It really is like guiding a very smart intern or temp. You can't just let them run wild in the codebase. They need explicit parameters.

nancyminusone 1y ago

https://xkcd.com/416/ https://xkcd.com/416/

jofzar 1y ago

"let's try a different approach" 95% of the time involves deleting the file and trying to recreate it. It's mind-blowing it happens so often.

mvieira38 1y ago

I suspect this has to do with newer training procedures for reasoning models, where they are injecting stuff like "wait a minute" to force the model to reason more, as described in the Deepseek R1 training docs