7 ms·
There is something darkly comical about using an LLM to write up your “a coding agent deleted our production database” Twitter post. On another note, I conside
by pierrekin 5mo ago
There is something darkly comical about using an LLM to write up your “a coding agent deleted our production database” Twitter post.
On another note, I consider users asking a coding agent “why did you do that” to be illustrating a misunderstanding in the users mind about how the agent works. It doesn’t decide to do something and then do it, it just outputs text. Then again, anthropic has made so many changes that make it harder to see the context and thinking steps, maybe this is an attempt at clawing back that visibility.
- NewsaHackO 5mo agoTwitter users get paid for these 'articles' based on engagement, correct? That may be the reason why it is so dramatized.
- dentemple 5mo agoIt's one way for the company to make its money back, I guess.
- jeremyccrane 5mo agoNaw, we just want people to know. We followed all Cursor rules, thought we had protected all API keys, and trusted the backups of a heavily used infrastructure company. Cautionary tale sharing with others.
- iainmerrick 5mo agoIt’s a good cautionary tale -- in hindsight the danger signs are clear, but it’s also clear why you thought it was OK and how third parties unfortunately let you down. The “agent’s confession” is the least interesting and useful part of the whole saga. Nothing there helps to explain why the disaster happened or what kind of prompting might help avoid it. The key mistake is accidentally giving the agent the API key, and the key letdown is the lack of capability scoping or backups in the service. The main lessons I take are “don’t give LLMs the keys to prod” and “keep backups”. Oh, and “even if you think your setup is safe, double-check it!”
- GreenWatermelon 5mo agoNo all that dramatization is just what LLMs belch out by default when told to tell a story.
- 59nadir 5mo ago> a misunderstanding in the users mind about how the agent work On top of that the agent is just doing what the LLM says to do, but somehow Opus is not brought up except as a parenthetical in this post. Sure, Cursor markets safety when they can't provide it but the model was the one that issued the tool call. If people like this think that their data will be safe if they just use the right agent with access to the same things they're in for a rude awakening. From the article, apparently an instruction: > "NEVER FUCKING GUESS!" Guessing is literally the entire point, just guess tokens in sequence and something resembling coherent thought comes out.
- deleted 5mo ago[deleted]
- sieste 5mo agoGood point, it's like having an instruction "Never fucking output a token just because it's the one most likely to occur next!!1!"
- jeremyccrane 5mo agoThat is actually pretty good, LLM's gonna LLM
- deleted 5mo ago[deleted]
- oofbey 5mo ago> It doesn’t decide to do something and then do it, it just outputs text. We can debate philosophy and theory of mind (I’d rather not) but any reasonable coding agent totally DOES consider what it’s going to do before acting. Reasoning. Chain of thought. You can hide behind “it’s just autoregressively predicting the next token, not thinking” and pretend none of the intuition we have for human behavior apply to LLMs, but it’s self-limiting to do so. Many many of their behaviors mimic human behavior and the same mechanisms for controlling this kind of decision making apply to both humans and AI.
- pierrekin 5mo agoI suspect we are not describing the same thing. When a human asks another human “why did you do X?”, the other human can of course attempt to recall the literal thoughts they had while they did X (which I would agree with you are quite analogous to the LLMs chain of thought). But they can do something beyond that, which is to reason about why they may have the beliefs that they had. “Why did you run that command?” “Because I thought that the API key did not have access to the production system.” When a human responds with this they are introspecting their own mind and trying to project into words the difference in understanding they had before and after. Whereas for an agent it will happily include details that are not literally in its chain of thought as justifications for its decisions. In this case, I would argue that it’s not actually doing the same thing humans do, it is creating a new plausible reason why an agent might do the thing that it itself did, but it no longer has access to its own internal “thought state” beyond what was recorded in the chain of thought.
- cortesoft 5mo ago> Whereas for an agent it will happily include details that are not literally in its chain of thought as justifications for its decisions. Humans do this too, ALL THE TIME. We rationalize decisions after we make them, and truly believe that is why we made the decision. We do it for all sorts of reasons, from protecting our ego to simply needing to fill in gaps in our memory. Honestly, I feel like asking an AI it’s train of thought for a decision is slightly more useful than asking a human (although not much more useful), since an LLM has a better ability to recreate a decision process than a human does (an LLM can choose to perfectly forget new information to recreate a previous decision). Of course, I don’t think it is super useful for either humans or LLMs. Trying to get the human OR LLM to simply “think better next time” isn’t going to work. You need actual process changes. This was a rule we always had at my company for any after incident learning reviews: Plan for a world where we are just as stupid tomorrow as we are today. In other words, the action item can’t be “be more careful next time”, because humans forget sometimes (just like LLMs). You will THINK you are being careful, but a detail slips your mind, or you misremember what situation you are in, or you didn’t realize the outside situation changed (e.g. you don’t realize you bumped the keyboard and now you are typing in another console window). Instead, the safety improvements have to be about guardrails you put up, or mitigations you put in place to prevent disaster the NEXT time you fail to be as careful as you are trying to be. Because there is always a next time. Honestly, I think the biggest struggle we are having with LLMs is not knowing when to treat it like a normal computer program and when to treat it like a more human-like intelligence. We run across both issues all the time. We expect it to behave like a human when it doesn’t and then turn around and expect it to behave like a normal computer program when it doesn’t. This is BRAND NEW territory, and we are going to make so many mistakes while we try to figure it out. We have to expect that if you want to use LLMs for useful things.
- vidarh 5mo agoIf you ask humans to explain why we did something, Sperry's split brain experiment gives reason to think you can't trust our accounts of why we did something either (his experiments showed the brain making up justifications for decisions it never made) Bit it can still be useful, as long as you interpret it as "which stimuli most likely triggered the behaviour?" You can't trust it uncritically, but models do sometimes pinpoint useful things about how they were prompted.
- emp17344 5mo agoThat is absolutely not what the split brain experiment reveals. Why would you take results received from observing the behavior of a highly damaged brain, and use them to predict the behavior of a healthy brain? Stop spreading misinformation.
- nuancebydefault 5mo agoSuch 'highly damaged' brain is still 90 percent or more structured the same as a normal human brain. See it as a brain that runs in debug mode. It is known that the narrative part of the brain is separate from the decision taking brain. If someone asks you, in a very convincing, persuasive way, why you did something a year ago and you can't clearly remember you did, it can happen that you become positive that you did so anyway. And then the mind just hallucinates a reason. That's a trait of brains.
- Jensson 5mo ago> If someone asks you, in a very convincing, persuasive way, why you did something a year ago and you can't clearly remember you did, it can happen that you become positive that you did so anyway. And then the mind just hallucinates a reason. That's a trait of brains. Yes brains can hallucinate reasons, doesn't mean they always do. If all reasons given were hallucinations then introspection would be impossible, but clearly introspection do help people.
- vidarh 5mo agoBecause said "highly damaged brain" in most respects still functions pretty much like a healthy one. There is no misinformation in what I wrote.
- jayd16 5mo agoBeyond that, isn't it just going to make up a narrative to fit what's in the prompt and context? I don't think there's any special introspection that can be done even from a mechanical sense, is there? That is to say, asking any other model or a human to read what was done and explain why would give you just an accounting that is just as fictional.
- mike_hearn 5mo agoNot necessarily. The people saying that in this thread seem to be forgetting about the encrypted reasoning tokens. The why of a decision is often recorded in a part of the context window you can't see with modern models. If you ask a model, "why did you do that" it isn't necessarily going to make up a plausible answer - it can see the reasoning traces that led up to that decision and just summarize them.
- gobdovan 5mo ago> asking a coding agent “why did you do that” to be illustrating a misunderstanding in the users mind about how the agent works I think the same thing, but about agents in general. I am not saying that we humans are automata, but most of the time explanation diverges profoundly from motivation, since motivation is what generated our actions, while explanation is the process of observing our actions and giving ourselves, and others around us, plausible mechanics for what generated them.
- badgersnake 5mo agoSeems like they’ve already reached the point where they’ve forgotten how to think.
- khazhoux 5mo ago> systemic failures across two heavily-marketed vendors that made this not only possible but inevitable. > No confirmation step. No "type DELETE to confirm." No "this volume contains production data, are you sure?" No environment scoping. Nothing. > The agent that made this call was Cursor running Anthropic's Claude Opus 4.6 — the flagship model. The most capable model in the industry. The most expensive tier. Not Composer, not Cursor's small/fast variant, not a cost-optimized auto-routed model. The flagship. The tropes, the tropes!! https://tropes.fyi/ https://tropes.fyi/
- levlaz 5mo agoSo if tropes.md works it doesn’t actually solve the problem. You’ll be reading stuff that you think an LLM didn’t write.
- jeremyccrane 5mo agoNot some vibe coder, and AI agents can be incredibly powerful. But yes, the irony is not lost on us!
- xnx 5mo agoAn LLM will reply with a plausible explanation of why someone would have written the code that it just wrote. Seems about the same.
- josephg 5mo ago> There is something darkly comical about using an LLM to write up It feels like a modern greek tragedy. Man discovers LLMs are untrustworthy, then immediately uses an LLM as his mouthpiece. Delicious!
- razorbeamz 5mo ago> There is something darkly comical about using an LLM to write up your “a coding agent deleted our production database” Twitter post. Which calls into question if this is even real.
- foota 5mo agoWhile I largely agree, it does raise the prospect of testing this iteratively. E.g., give a model some fake environment, prompt it random things until it does something "bad" in your fake environment, and then fix whatever it claims led to its taking that action. If you can do this and reliably reduce the rate at which it does bad things, then you could reasonably claim that it is aware of meaningful introspection.
- mtrifonov 5mo agoYes, you're right, in that there's no decision module separate from the output. It overcommits in the other direction. The post-hoc reasoning the model produces when you ask "why did you do that" is also just text, and yet that text often matches independent third-party analysis of the same behavior at well above chance. If it really were uncorrelated text-completion, the post-hoc explanation should not align with the actual causes more than randomly. It does, frequently enough that I've stopped using it as evidence the user is naive. "just outputs text" is doing more work than we acknowledge. The person asking the agent "why did you do that" might be an idiot for expecting anything more than a post-hoc rationalization, but that's exactly what you'd expect from a human too.
- xnx 5mo agoOn social media, a reasonable first assumption is that all content is written primarily for views/engagement. Any component of truth is incidental.