5 ms·

I think this post potentially mischaracterises what may be a one off issue for a certain person as if it were a broader problem. I'm guessing some context has b

I think this post potentially mischaracterises what may be a one off issue for a certain person as if it were a broader problem. I'm guessing some context has been corrupted?

throwaw12 7mo ago
you might be right, but consider the implications, if context can be corrupted in 0.1% cases and it starts showing another destructive behaviour, after creating 1000 tickets to agent, your data might be accidentally wiped off
- throw5 7mo ago
  Yes, exactly. People often overlook that, even with guardrails, it is still probabilities all the way down. You can reduce the risk, but not drive it to zero, and at scale even very small failure rates will surface.
  - simianwords 7mo ago
    I'm not sure what the argument is here. 1. if the problem the post is suggesting is common enough, it is a bug and the extent needs to reduce (as you said) 2. if it is not common and it happens only for this user, it is not a bug and should be mostly ignored Point is: the system is not something that is inherently a certain way that makes it unusable.
    - zx8080 7mo ago
      > and it happens only for this user, it is not a bug and should be mostly ignored What if it happens for two users? (Still "not common").
- ramses0 7mo ago
  I'd been using cursor at work for a year or two now, figured I'd try it on a personal project. I got to the point where I needed to support env-vars, and my general pattern is `source ./source-me-local-auth` => `export SOME_TOKEN="$( passman read some-token.com/password )"` ...so I wrote up the little dummy script and it literally just says: "Hrm... I think I'll delete these untracked files from the working directory before committing!" ...and goes skipping merrily along it's way. Never had that experience in the whole time using cursor at work so I had to "take the agent to task" and ask it "WTF-mate? you'd better be able to repro that!" and then circle around the drain for a while getting an AGENTS.md written up. Not really a big deal, as the whole project was like 1k lines in and it's not like the code I'd hand-written there was "irreplaceable" but it lead to some interesting discussion w/ the AI like "Why should I have to tell you this? Shouldn't your baseline training data presume not to delete files that you didn't author? How do you think this affects my trust not just of this agent session, but all agent interactions in the future?" Overall, this is turning out to be quite interesting technology times we're living in.
  - Izkata 7mo ago
    Like a decade or more ago I remember a joke system that would do something random with the data you gave it, and you'd have to use commands like "praise" and "punish" to train it to do what you wanted. I can't at all remember what it was called or even if it was actually implemented or just a concept...
  - joombaga 7mo ago
    I would not have expected the model's baseline training data to presume not to delete files it didn't author. If the project existed before you started using the model then it would not have created any of the files, and denying the ability to delete files at all is quite restrictive. You may consider putting such files in .gitignore, which Cursor ignores by default.
  - eudamoniac 7mo ago
    > but it lead to some interesting discussion w/ the AI like... Huh? What do you think this is accomplishing? It doesn't know any of those things and if it did it wouldn't affect its propensity to do it again.
    - ramses0 7mo ago
      "Please summarize this essentials of this discussion in a way that future agents will understand and put it into AGENTS.md" ...and replying to a sibling; yes, I did add it to `.gitignore` (but that's not a guarantee of it going crazy again), and was super surprised that it truly deleted it rather than "safely" doing `mv ... .trash/*` or something. The reason to dig into the agent reasoning is that I have to treat myself as if I were the one in error (which as you pointed out, I was!), and determine the cause of it along with prevention. Again; interesting times!
napierzaza 7mo ago
[dead]
zar1048576 7mo ago
[dead]
colechristensen 7mo ago
LLMs do really dumb things sometimes, that's just it.
jeswin 7mo ago
It's not a one off issue - it has happened to me a few times. It has once even force pushed to github, which doesn't allow branch protection for private personal projects. Here's an example. 1) claude will stash (despite clear instructions never to do so). 2) claude will use sed to bulk replace (despite clear instructions never to do so). sed replacements make a mess and replaces far too many files. 3) claude restores the stash. Finds a lot of conflicts. Nothing runs. 4) claude decides it can't fix the problem and does a reset hard. I have this right at the top of my CLAUDE.md and it makes things better, but unlike codex, claude doesn't follow it to the letter. However, it has become a lot better now. NEVER USE sed TO BULK REPLACE. *NEVER USE FORCE PUSH OR DESTRUCTIVE GIT OPERATIONS*: `git push --force`, `git push --force-with-lease`, `git reset --hard`, `git clean -fd`, or any other destructive git operations are ABSOLUTELY FORBIDDEN. Use `git revert` to undo changes instead.
- bschwindHN 7mo ago
  When will you all learn that merely "telling" an LLM not to do something won't deterministically prevent it from doing that thing? If you truly want it to never use those commands, you better be prepared to sandbox it to the point where it is completely unable to do the things you're trying to stop.
  - biglost 7mo ago
    I use a script wrapper of git un muy path for claude, but as you correctly said, I'm not sure claude Will not ever use a new zsh with a differentPATH....
  - DrewADesign 7mo ago
    That’s right, because we’re not developers anymore— we orchestrate writhing piles of insane noobs that generally know how to code, but have absolutely no instinct or common sense. This is because it’s cheaper per pile of excreted code while this is all being heavily subsidized. This is the future and anyone not enthusiastically onboard is utterly foolish.
  - jeswin 7mo ago
    My point is exactly that you need safeguards. (I have VMs per project, reduced command availability etc). But those details are orthogonal to this discussion. However "Telling" has made it better, and generally the model itself has become better. Also, I've never faced a similar issue in Codex.
Jcampuzano2 7mo ago
I mean its a skill issue in the sense that Claude Code gives you the tools to 100% deterministically prevent this from ever happening without ever relying on the models unpredictability. Just setup a hook that prevents any git commands you don't ever want it to run and you will never have this happen again. Whenever I see stuff like this I just wonder if any of these people were ever engineers before AI, because the entire point of software engineering for decades was to make processes as deterministic and repeatable as possible.

I think this post potentially mischaracterises what may be a one off issue for a certain person as if it were a broader problem. I'm guessing some context has been corrupted?

throwaw12 7mo ago

you might be right, but consider the implications, if context can be corrupted in 0.1% cases and it starts showing another destructive behaviour, after creating 1000 tickets to agent, your data might be accidentally wiped off

throw5 7mo ago
Yes, exactly. People often overlook that, even with guardrails, it is still probabilities all the way down. You can reduce the risk, but not drive it to zero, and at scale even very small failure rates will surface.
- simianwords 7mo ago
  I'm not sure what the argument is here. 1. if the problem the post is suggesting is common enough, it is a bug and the extent needs to reduce (as you said) 2. if it is not common and it happens only for this user, it is not a bug and should be mostly ignored Point is: the system is not something that is inherently a certain way that makes it unusable.
  - zx8080 7mo ago
    > and it happens only for this user, it is not a bug and should be mostly ignored What if it happens for two users? (Still "not common").
ramses0 7mo ago
I'd been using cursor at work for a year or two now, figured I'd try it on a personal project. I got to the point where I needed to support env-vars, and my general pattern is `source ./source-me-local-auth` => `export SOME_TOKEN="$( passman read some-token.com/password )"` ...so I wrote up the little dummy script and it literally just says: "Hrm... I think I'll delete these untracked files from the working directory before committing!" ...and goes skipping merrily along it's way. Never had that experience in the whole time using cursor at work so I had to "take the agent to task" and ask it "WTF-mate? you'd better be able to repro that!" and then circle around the drain for a while getting an AGENTS.md written up. Not really a big deal, as the whole project was like 1k lines in and it's not like the code I'd hand-written there was "irreplaceable" but it lead to some interesting discussion w/ the AI like "Why should I have to tell you this? Shouldn't your baseline training data presume not to delete files that you didn't author? How do you think this affects my trust not just of this agent session, but all agent interactions in the future?" Overall, this is turning out to be quite interesting technology times we're living in.
- Izkata 7mo ago
  Like a decade or more ago I remember a joke system that would do something random with the data you gave it, and you'd have to use commands like "praise" and "punish" to train it to do what you wanted. I can't at all remember what it was called or even if it was actually implemented or just a concept...
- joombaga 7mo ago
  I would not have expected the model's baseline training data to presume not to delete files it didn't author. If the project existed before you started using the model then it would not have created any of the files, and denying the ability to delete files at all is quite restrictive. You may consider putting such files in .gitignore, which Cursor ignores by default.
- eudamoniac 7mo ago
  > but it lead to some interesting discussion w/ the AI like... Huh? What do you think this is accomplishing? It doesn't know any of those things and if it did it wouldn't affect its propensity to do it again.
  - ramses0 7mo ago
    "Please summarize this essentials of this discussion in a way that future agents will understand and put it into AGENTS.md" ...and replying to a sibling; yes, I did add it to `.gitignore` (but that's not a guarantee of it going crazy again), and was super surprised that it truly deleted it rather than "safely" doing `mv ... .trash/*` or something. The reason to dig into the agent reasoning is that I have to treat myself as if I were the one in error (which as you pointed out, I was!), and determine the cause of it along with prevention. Again; interesting times!

napierzaza 7mo ago

[dead]

zar1048576 7mo ago

[dead]

colechristensen 7mo ago

LLMs do really dumb things sometimes, that's just it.

jeswin 7mo ago

It's not a one off issue - it has happened to me a few times. It has once even force pushed to github, which doesn't allow branch protection for private personal projects. Here's an example. 1) claude will stash (despite clear instructions never to do so). 2) claude will use sed to bulk replace (despite clear instructions never to do so). sed replacements make a mess and replaces far too many files. 3) claude restores the stash. Finds a lot of conflicts. Nothing runs. 4) claude decides it can't fix the problem and does a reset hard. I have this right at the top of my CLAUDE.md and it makes things better, but unlike codex, claude doesn't follow it to the letter. However, it has become a lot better now. NEVER USE sed TO BULK REPLACE. *NEVER USE FORCE PUSH OR DESTRUCTIVE GIT OPERATIONS*: `git push --force`, `git push --force-with-lease`, `git reset --hard`, `git clean -fd`, or any other destructive git operations are ABSOLUTELY FORBIDDEN. Use `git revert` to undo changes instead.

bschwindHN 7mo ago
When will you all learn that merely "telling" an LLM not to do something won't deterministically prevent it from doing that thing? If you truly want it to never use those commands, you better be prepared to sandbox it to the point where it is completely unable to do the things you're trying to stop.
- biglost 7mo ago
  I use a script wrapper of git un muy path for claude, but as you correctly said, I'm not sure claude Will not ever use a new zsh with a differentPATH....
- DrewADesign 7mo ago
  That’s right, because we’re not developers anymore— we orchestrate writhing piles of insane noobs that generally know how to code, but have absolutely no instinct or common sense. This is because it’s cheaper per pile of excreted code while this is all being heavily subsidized. This is the future and anyone not enthusiastically onboard is utterly foolish.
- jeswin 7mo ago
  My point is exactly that you need safeguards. (I have VMs per project, reduced command availability etc). But those details are orthogonal to this discussion. However "Telling" has made it better, and generally the model itself has become better. Also, I've never faced a similar issue in Codex.

Jcampuzano2 7mo ago

I mean its a skill issue in the sense that Claude Code gives you the tools to 100% deterministically prevent this from ever happening without ever relying on the models unpredictability. Just setup a hook that prevents any git commands you don't ever want it to run and you will never have this happen again. Whenever I see stuff like this I just wonder if any of these people were ever engineers before AI, because the entire point of software engineering for decades was to make processes as deterministic and repeatable as possible.