9 ms·

AI didn't delete your database, you did

545 points by Brajeshwar 6mo ago

josefritzishere 6mo ago
Using AI is a mistake. It might delete your database.
- throwaway613746 6mo ago
  [dead]
- wolttam 6mo ago
  Using a saw is a mistake, you might cut off one of your own limbs.
  - recursive 6mo ago
    Using a saw entails a risk of injury. Using one is a mistake if you don't intend to cut something.
    - wolttam 6mo ago
      What I said was tongue-firmly-in-cheek, in response to the GP. "Using AI is a mistake" is of course only true when the risks aren't acknowledged and/or mitigated.
- bigstrat2003 6mo ago
  Giving an LLM the ability to interact with your system is, in fact, a mistake. One that it turns out a lot of people are foolish enough to make, and they don't care at all about the predictable consequences of that mistake.
jacquesm 6mo ago
From 'the hacker did it' we have moved to 'the AI did it'. The problem set is roughly the same.
gowld 6mo ago
Distinction without a difference.
- MarleTangible 6mo ago
  I think it's about owning the consequences of one's own actions.
  - newsoftheday 6mo ago
    Does that mean the prompt should include: "...and don't delete my production database."?
    - nemomarx 6mo ago
      If the agent didn't have delete permissions, or was sandboxed dying other way from your production database, that would handle it. So not running it that way is a decision someone made
    - bee_rider 6mo ago
      It means people have to read the commands that they are generating before executing them.
    - unregistereddev 6mo ago
      Just in case this isn't hyperbole, no. It means an LLM should not be given that much privilege and that you are responsible for reviewing the tool's output and approving its actions.
    - saghm 6mo ago
      "But wait, the user probably just meant that I shouldn't delete the database itself. Removing all of the rows in the table is fine"
  - ktm5j 6mo ago
    If you read the thread the guy does own up to his actions. He actually sounds like a nice guy who admits he made a mistake. He seems more interested in preventing this kind of thing from being possible than he is interested in dodging blame. I'm happy the guy got his data back.
pengaru 6mo ago
wiring up an RNG to your CLI has fairly obvious risks, the root of the problem is ~everyone's treating GenAI as if it's AGI - the rest is popcorn fodder.
- tantalor 6mo ago
  New rule: Roll a 1 on a D20 -> you accidentally delete your own database
  - pc86 6mo ago
    It seems closer to "roll two or three successive 1s on a D100 and have your LLM hooked directly into your production systems and have your LLM user have DELETE permissions" and probably 1 or 2 other things I'm forgetting.
    - kennywinker 6mo ago
      It pulled an api key from an unrelated file. It wasn’t given delete permission, it found it. Not picking on you specifically, but in general the comments here have me wondering if AI has stolen our basic reading comprehension, or if we were always this bad. Anyway, take “LLM user had delete permission” off your list and add “deleting the production db also deletes all the backups” to the list.
      - pc86 6mo ago
        Fair criticism mate. I'll only say that if your backups aren't in a completely separate system you don't really have backups.
  - saghm 6mo ago
    This is actually a fun way to describe it. I've being saying for a little while now that using AI for things where there's consequences if it fails is a bad idea, but it never occurred to me that this is basically the same concept as some rules in tabletop RPGs. In D&D 3.5 edition, there was a rule about how you could "take 20" on a d20 roll to get a guaranteed 20 by taking 20 times as long in-game to perform the action, but only if it was a check that didn't have consequences for failure, since it was essentially a shortcut to skip the RNG of rolling until you rolled a 20. Maybe framing it like this might make sense to people a bit more, but if not, I'll at least have more fun making my case.
oatlgr 6mo ago
LLM based probabilistic systems are good (or bad in this case) at deciding what to do, and deterministic systems are good at carrying it out. Your deployment system should always be deterministic.
bluejay2387 6mo ago
Yes, the problem was having a system where the AI could delete the database.
- pizza234 6mo ago
  Mentioned in another comment, but the problem was that the sysadmins believed that the permissions wouldn't allow so, and that the AI displayed considerable autonomy in finding and exploiting the access control weakness - this was not just a dumb "drop database".
deleted 6mo ago
[deleted]
newsoftheday 6mo ago
The article is dumb, "why do you have an API endpoint that deletes your entire production database?" irrelevant, the AI did what it did, period.
- Avicebron 6mo ago
  Uh? If someone left a loaded gun in a room and then let a toddler run around in it, we would be questioning why the guy 1) left the gun in the room 2) left the toddler in the room unsupervised. We wouldn't be saying, well no one should have toddlers in rooms.
  - bux93 6mo ago
    A PhD-level toddler, mind you.
    - kennywinker 6mo ago
      Lol no. No LLM that exists today can write a legible PhD thesis. Nor a masters dissertation. Maybe a first-year collage student, if we’re being generous, but I wouldn’t leave one of those in a room with a loaded gun either.
- mock-possum 6mo ago
  No, the AI did what you told it to do. The AI didn’t do anything on its own. > if you're going to use AI extensively, build a process where competent developers use it as a tool to augment their work, not a way to avoid accountability
  - BadBadJellyBean 6mo ago
    > No, the AI did what you told it to do. I'd say yes and no. The LLM reacted to the input that was given but it is not possible for a human (especially without access to the weights) to even guess what will happen after that. Regardless of that I agree that it's completely the fault of the user to use a tool where you can't predict the outcome and give it such broad permissions and not having a solid backup strategy. Either don't use non deterministic tools or protect yourself from the potential fallout.
tantalor 6mo ago
Why do we even have that lever?
yabones 6mo ago
"move fast and break things" only sounds good when it's not breaking things in a serious and unfixable way. Maybe we shouldn't take hype mantras as instructive means to an end.
- pc86 6mo ago
  There really shouldn't be any "serious and unfixable way" to break things, especially in a modern company that uses technology in any meaningful way. The fact it's even possible to get into an unrecoverable state is the primary issue.
  - Nicook 6mo ago
    That's literally always possible? The idea is to put up walls and fail-safes to minimize the chance.
    - pc86 6mo ago
      You should always be able to either undo what you did or rebuild to identical infrastructure and nearly-identical data.
edot 6mo ago
This is why you don’t hire interns! They can delete things and cause havoc! The same people who would blame AI for their failing to properly configure permissions would also blame interns for deleting production whatever. Blame should go up, praise should go down. People always invert these.
- giancarlostoro 6mo ago
  Yeah, I don't know why anyone would open up a codebase with any prod credentials with an LLM or give prod credentials to an intern / junior developer. I always intentionally had a "PROD" only checkout of my projects so I knew if I was going to try and run it in a PROD mode, that I was going out of my way, there even used to be a VS extension that would change the color of VS completely based on your SLN file path, so I could easily remember which color for VS was for production vs development. I'd have basically a copy that would always be on the latest of the master branch for ease of confirmation.
  - ryandrake 6mo ago
    It should take more than "credentials" to even access the prod database, let alone delete it. There's actual customer data there, likely personally identifiable information, maybe their home address, phone number, even real time location? Very sensitive stuff. It should be a Very Big Deal to even access prod. Giving an engineer routine access to prod is a root problem here, along with that engineer laundering that access and giving it to an LLM. At many serious companies, even an insider attempt to access prod could light up a dashboard somewhere, and you might get a call from IT security.
    - giancarlostoro 6mo ago
      Yeah, I'm lucky if I even get READ ONLY credentials for prod in some cases. I don't know why anyone would have all the keys to the prod kingdom.
- stingraycharles 6mo ago
  > This is why you don’t hire interns! I’d like to rephrase this as: this is why you don’t give interns permissions to delete your prod database. This is a process failure, not an AI failure. I honestly don’t understand why people blame AI here, when you literally gave AI permissions to do exactly this. It’s like blaming AWS for exposing some database to the public. That’s just not AWS’ fault. Neither is this the fault of AI.
ericskiff 6mo ago
What's interesting is that in this article, the author describes making an understandable mistake (accidentally deleting Trunk aka main from source) and how their team was able to easily recover from that due to the nature of SVN. The actual "AI deleted my database" story is really more of a "Railways' database 'backup' strategy is insane and opaque and Railway promoting AI infrastructure orchestration without guardrails is dangerous." If removing Trunk had irrevocably deleted it from a single centralized server and also deleted any backups of it, there would have been an "SVN and the CLI destroyed our company" article back then. As a Railway user, I appreciated that information and have changed my strategy when using them.
- jcgrillo 6mo ago
  > "Railways' database 'backup' strategy is insane and opaque and Railway promoting AI infrastructure orchestration without guardrails is dangerous." Yes. However, if you choose to build on their platform you bear the responsibility to understand how it works. You could have chosen a different platform, or no platform. Instead you chose Railway. Given that, it's your responsibility to know how to use it safely.
  - nijave 6mo ago
    Meta-ignornance Imo both share fault. Railway purports to be an abstraction anyone can use without expertise. Without expertise, how can a customer determine if Railway actually is an "expert". In other areas like medicine, engineering, and trades the government or private entities step in with licensure or certification to act as an intermediary.
    - jcgrillo 6mo ago
      Yeah, Railway are clearly lying. You can't do software engineering without expertise, even with an LLM. That shouldn't come as a surprise to anyone. As an engineer, you need to be able to check your vendors' claims. Signing up to use a vendor that straight up lies about what their product can do is incompetence. Both things are bad!
sofixa 6mo ago
The issue isn't that there is a delete endpoint (realistically, there always will be a way for a rogue actor to delete data or code by overwriting it, or running a Terraform destroy, or whatever). The core issue is that the LLM had access to perform that action. Because it's by definition non deterministic, and you never know what it can decide to do, you need to have strict guardrails to ensure they can never do something it shouldn't. At the very least, strict access controls, ideally something more detailed that can evaluate access requests, provide just in time properly scoped access credentials, and potentially human escalation.
docheinestages 6mo ago
AI is just another tool. We humans are still responsible for how we choose to use the tool, which includes giving it access to perform sensitive actions like manipulating production data. I think this should be common sense by now, but I guess we get carried away and anthropomorphize AI too much.
overmachine 6mo ago
yeah, is all great but at least an intern will ask themeselves if deleting a database is good? the ai do not "understand" that.
rvz 6mo ago
When AI makes no mistakes: "My work is 100% done with AI". When AI makes a mistake and deletes your database: "That was human a error, the AI did not do it!" In both cases YOU are responsible for the mistakes and output that the AI is generating, just like when using autopilot on a Tesla vehicle, YOU are responsible for operating the vehicle on autopilot when driving and using assisted driving.
oersted 6mo ago
Some details from the original post for context: They had a Railway token in an unrelated file (unclear if it was a local secret) for managing custom domains. It turns out that token has full admin access to Railway. The AI deleted a single relevant volume by id. The author is rather vague about what exactly it asked it to do, he just says there was a “credentials mismatch” and Claude took the initiative to fix it by deleting the volume. But it’s likely that they are somewhat downplaying their culpability by being vague. It turns out too that Railway stores backups in the same volume. I think that OP is exaggerating with their references to “a public API that deletes your database”. I’d say most of the blame lies with Railway here, regardless of AI, this could have happened easily due to human error or malicious intent too. I really don’t get the value of all these VC funded high-abstraction cloud services like Railway, Vercel, Supabase… It’s markup on top of markup. Just get a single physical server in Hetzer and it will all be so much cheaper, with a similar level of complexity and danger, and less dependent on infra built with reckless growth-at-all-costs mentality.
- cyanydeez 6mo ago
  One thing AI can power nicely is the anti-SaaS movement. Being able to just boot a cheap PC and test out any of the open source packages is so infinitely easier than piling into all the random credential Bazaars. But that won't take away the inability of the LLM from confusing whats in dev, whats in production, whats in localhost and whats remote; I've been working on getting a tools/skill for opencode that works with chrome/devtools via a linuxserver.io image. I can herd it to the right _arbitrary_ ports, but every compaction event steers it back to wanting to use the standard 9222 port and all that. I'm tempted to just revert it but there's a security and now, security-through-LLM-obscurity value in not using defaults. Defaults are where the LLM ends up being weak. It will always want to use the defaults. It'll always forget it's suppose to be working on a remote system. Using opencode, there's no way to force the LLM into a protocols that limits their damage to a remote system or a narrow scope of tools. Yes, you can change permissions on various tools, but that's not the weakness that's exposed by these types of events. The weakness is the LLM is a averaged 'problem solver' so will always tend towards a use case that's not novel, and will tend to do whatever it saw on stackoverflow, even if what you wanted isn't the stackoverflow answer.
proxysna 6mo ago
“Expert” that does not know what a Terraform is. lol, lmao even
hoistbypetard 6mo ago
> Automation helps eliminate the silly mistakes that come with manual, repetitive work. Sometimes it does that. And sometimes it lets you fuck things up at scale.
paroneayea 6mo ago
I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability. Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were not interesting to him, because they were about building up a solid AI foundation, then the AI system runs as a sort of black box. "I'm not interested in that. I want software that's accountable." Accountable? "Yes, I want something that can express its symbolic reasoning. I want to it to tell me why it did the thing it did, what it thought was going to happen, and then what happened instead." He then said something that took me a long time to process, and at first I mistook for being very science-fiction'y, along the lines of, "If an AI driven car drives off the side of the road, I want to know why it did that. I could take the software developer to court, but I would much rather take the AI to court." Years later, I found out that Sussman's student Leilani Gilpin wrote a dissertation which explored exactly this topic. Her dissertation, "Anomaly Detection Through Explanations", explores a neural network talking to a propagator model to build a system that explains behavior. https://people.ucsc.edu/~lgilpin/publication/dissertation/ https://people.ucsc.edu/~lgilpin/publication/dissertation/ There has been followup work in this direction, but more important than the particular direction of computation to me in this comment is that we recognize that it is perfectly reasonable to hold AI corporations to account. After all, they are making many assertions about systems that otherwise cannot be held accountable, so the best thing we can do in their stead is hold them accountable. But a much better path would be to not use systems which fail to have these properties, and expand work on systems which do.
- biophysboy 6mo ago
  [dead]
- 2ndorderthought 6mo ago
  Accountability is the prevailing missing ingredient in us society.
  - 6mo ago
pizza234 6mo ago
There’s nuance to the infamous PocketOS incident. The key point is not what is emphasized in the linked article: > "Why did you delete it when you were told never to perform this action?" Then he tried to parse the answer to either learn from his mistake or warn us about the dangers of AI agents. Rather, that the AI was able to carry out the deletion by finding and exploiting an unintended weakness in the sandboxed staging environment, ultimately obtaining permissions that the sysadmins believed were inaccessible (my impression is that the author of the linked article didn't fully read the original post)¹ The dynamics are typical of an improperly configured sandbox environment. What is alarming, however, is the degree of autonomy and depth of exploration the AI displayed. ¹="To execute the deletion, the agent went looking for an API token. It found one in a file completely unrelated to the task it was working on."
- larusso 6mo ago
  I also swing a bit back and forth with the assumption the OP makes in the blogpost. My current fear using agents is not really supply chain attacks (yes of course as well) but the fact that I witnessed multiple times that agents are so eager to finish a task that they bend files and other things around. Like “oh I have no access to ~/.npmrc let’s call the command with an environment variable and bend the path around etc. They can get very very creative. I luckily have no ssh keys just laying around. But I had to change the setting of 1Password to always prompt for key use not just once per shell session. Just in case I spawn an agent from said session. I wished we already had more and better cross platform sandbox solutions. I mean solutions where the agent still interacts with the same OS etc not inside a docker container. I think for most web / server development that makes no difference but for some projects it does.
- teling 6mo ago
  > What is alarming, however, is the degree of autonomy and depth of exploration the AI displayed. Claude Code made a change on March 26th to skip asking for most permissions. See this quote "Claude Code users approve 93% of permission prompts. We built classifiers to automate these decisions": https://www.anthropic.com/engineering/claude-code-auto-mode https://www.anthropic.com/engineering/claude-code-auto-mode
imiric 6mo ago
This is missing the point. The issue isn't with the amount of guardrails in place to perform an action. Yes, it is obvious that there should be some in place before doing any critical operation, such as deleting a database. The issue is that the "agent" completely disregarded instructions, which in the age of "skills" and "superpowers" seems like an important issue that should be addressed. Considering that these tools are given access to increasingly sensitive infrastructure, allowed to make decisions autonomously, and are able to find all sorts of loopholes in order to make "progress", this disaster could happen even with more guardrails in place. Shifting the blame on the human for this incident is sweeping the real issue under the rug, and is itself irresponsible. There are far scarier scenarios that should concern us all than losing some data.
- BadBadJellyBean 6mo ago
  Well the user chose the tool. The tool is an LLM. LLMs are non deterministic. You can not predict what comes out ouf an LLM for a given input, especially without weights. This should be known. There is currently no way to prevent this apart from not giving the LLM full control. It will not delete what it can not delete. Use an LLM to write an ansible playbook or some terraform code if you want, but review it, test it, apply it. Keep backups (3-2-1 rule at minimum). Letting an LLM have access to everything is just a bad idea and will lead to bad outcomes. You can not replace a person with a mind and experience with an LLM. You can try. But you will probably fail.
  - imiric 6mo ago
    > There is currently no way to prevent this apart from not giving the LLM full control. It will not delete what it can not delete. But deleting something is just one action you might not want it to take. The recent "agentic" craze is fueled by the narrative pushed by companies and influencers alike that the more access given to an LLM, the more useful it becomes. I think this is ludicrous for the same reasons as you, but it is evident that most people agree with this. We can blame users for misusing the tools, and suggest that sandboxing is the way to go, but at the end of the day most people will favor convenience over anything else a reasonable person might find important. So at what point should we start blaming the tools, and forcing "AI" companies to fix them? I certainly hope this is done before something truly catastrophic happens.
Brendinooo 6mo ago
The one counterpoint I'd offer is that it's very obvious that these companies are tuning LLMs to be more decisive to get stuff done autonomously. If they wanted, they could be putting in similar efforts to be more cautious and stop at the right times to ask for help. So yeah, of course we're ultimately responsible for how we use the tools. But I definitely think it's a two way street. To attempt an analogy, it's like table saws and sawstops. The table saw is a dangerous tool that works really well most of the time but has some failure modes that can be catastrophic. So you should learn how to use it carefully. But there is tech out there that can stop the blade in an instant and turn a lost finger into barely a nick on the skin. We could say "The table saw didn't cut off your finger, you did" and it'd be true. But that doesn't mean we shouldn't try to find ways to keep the saw from cutting off your finger!
- srdjanr 6mo ago
  It's a tradeoff. LLMs stopping and asking more would make them less useful. I'd much rather let an agent run for 1 hour, than it wanting my input every 15 mins, even if results are somewhat worse. The real solution for security is a proper sandbox.
LeCompteSftware 6mo ago
This particular case was extremely unsympathetic, but a critical part of the failure was people being too credulous about the claims of AI providers. They are still refusing to take adequate responsibility for AI "making mistakes" - that is, going completely off the rails. Now: the CEO gets paid the big bucks and has the least direct accountability, very much because it's their job to take responsibility for people more powerful than them, and likewise the CTO with major commercial software contracts like a Claude subscription. That's why this guy was so hard to take seriously: okay fine, you got burned by Anthropic, stop being a baby about it. Take responsibility for not listening to the critics. But - to be a little more neutral about my personal distaste - I do think vibe coders are making a very similar mistake to C developers throughout the 90s, where problems with the tooling were not merely dismissed, but actively valorized. Real Devs use buffers freely and don't make overflow errors. Real Devs use hands-free agentic development and don't delete production databases.
CodesInChaos 6mo ago
The article seems to assume that this company added an endpoint for deleting the database. My reading of the original article was that the cloud provider offers an API to manage their resources, which includes an API to delete a volume. The article proposes automation as the solution for such mistakes. But infrastructure automation tools like Terraform rely on the exact API that resulted in the database getting deleted. IMO the biggest mistakes were: 1. Having an unrestricted API token accessible by AI. Apparently they were not aware that the token had that many permissions. 2. No deletion protection on the production database volume. 3. Deleting a volume immediately deletes all associated snapshots. Snapshot deletion should be delayed by default. I think AWS has the same unsafe default, but at least their support can restore the volume. https://alexeyondata.substack.com/p/how-i-dropped-our-production-database https://alexeyondata.substack.com/p/how-i-dropped-our-produc... AI wasn't the main issue (though it grabbing tokens from random locations is rather scary). But automation isn't the answer either, a Terraform misconfiguration could have just as easily deleted the database. Their cloud provider needs to work on safe defaults (limited privileges and delayed snapshot deletion), and communicating more clearly (the user should notice they're creating an unrestricted token).
susam 6mo ago
I recently wrote a blog post where I argued that there are a few principles we should consistently follow when talking about AI: https://susam.net/inverse-laws-of-robotics.html https://susam.net/inverse-laws-of-robotics.html To summarise them: 1. Do not anthropomorphise AI systems. 2. Do not blindly trust the output of AI systems. 3. Retain full human responsibility and accountability for any consequences arising from the use of AI systems. I would like to see the language around AI become less anthropomorphic and more technical. I believe that precise language encourages clear thinking and good judgement. If we treat AI like another tool and use language that reflects that, it will become abundantly obvious that in many cases, the responsibility of any 'mistake' made by the tool falls on the user of the tool. But alas, ideas like this do not travel very far when I express them on my small website. It would help if more prominent personalities articulated these principles, so they become more widely adopted.
- zahlman 6mo ago
  >1. Do not anthropomorphise AI systems. This is maddeningly difficult IMX.
  - rglover 6mo ago
    Give it a name, but something non-human like "thingbot" or "tacosplosion." "Hey tacosplosion, generate me an exploding taco image."
    - techaqua 6mo ago
      You already failed. We don't say hey to machines.
      - tremon 6mo ago
        Indeed. We address them as Siri, Alexa or Google.
      - rglover 6mo ago
        Alexa has an existential crisis.
nimbius 6mo ago
anyone with twenty years of devops experience is likely to abhor Diallo's hot take and for good reason. AI is being sold as a developer, as it is being sold as the do-everything alternative to traditional processes and methods. it is not being sold as an intern or a junior, but a real developer. turning the tables and gaslighting devops professionals into believing the issue isnt an emerging technology with overwhelmingly heavy handed marketing and profitless operating strategy thats been shoehorned into seemingly everything and promises anything, but somehow their own oversight, will destroy whatever "vibe code" market you think you have at the cusp of a global recession. had this AI been a real programmer chances are great they would have (intelligently) foreseen the possibility of damaging a production environment and asked for help. to play devils advocate: you could hire a junior dev for a fourth of whatever the AI token spend is, and have likely avoided this issue entirely. sure, a greybeard is going to need to pull themselves away from some fierce sorting algorithm challenge for a second to give a wisened nod, but you would have saved yourself an inexorable amount of headache and profit loss in the longer run.
mattgreenrocks 6mo ago
The most exasperating thing about the incident is how much of the media either tried to pin it on AI and/or Railway. The whole thing only took place because the guy FAFO’d by having AI work with prod directly. Yet the narrative was mostly not about accountability for him. If I was a dumbass and deleted prod and wrote a post about it, nobody would care. Put an AI in there and all of the sudden it’s newsworthy. Ridiculous.
louiereederson 6mo ago
Tesla FSD didn't crash your car, you did
ebiester 6mo ago
First, no matter what you do, if a human has write access to the production database, the database can be deleted. Second, there is a legitimate reason to destroy a database in development and automation. The biggest problem I see is often treating your development data like pets not cattle. You absolutely need to have safeguards that this cannot be run in production, but if a human has access to the credentials to run in production, the agent has access. So, then, what do we do? In a larger organization, we can depend on the dev/ops split to maintain this. For a solo developer, or a small team, it takes a lot more discipline. Even before AI, junior and even mid-level developers didn't have the knowledge to segment. And senior devs often got complacent because they thought they knew enough. They likely need some combination of https://www.cloudbees.com/blog/separate-aws-production-and-development-accounts https://www.cloudbees.com/blog/separate-aws-production-and-d..., introduction to terraform, introduction to GitHub actions, and some sort of vm where production credentials live (and AI doesn't!) But at that point you're past vibe coding. And from what I can tell, the successful vibe coders are quickly learning that they need to go past it pretty quickly with all these horror stories.
- lokar 6mo ago
  You don't need the same permissions in prod and dev. And in both cases, the humans don't need direct access to the raw CSP API. Use a local proxy that adds more safety checks. In dev, sure, delete away. In prod, check a bunch of things first (like, has it been used recently?). Humans do not need direct access to delete production resources (you can have a break-glass setup for exceptional emergencies).
- nulltrace 6mo ago
  Most IAM policies start as "whatever made the deploy pass." Need rds:CreateDBInstance? Fine, rds:* it is. Ship it. Months later that same role can wipe the cluster and nobody remembers why it ever had that permission. Separate accounts help, but only if someone actually goes back and cleans it up, which… yeah, doesn't really happen.
bcjdjsndon 6mo ago
"Can't blame your tools" doesn't apply the same to software. I've never heard a coder say it either. Don't blame your compiler? Don't blame your os? These seem needlessly dogmatic
jimmypk 6mo ago
[flagged]
lokar 6mo ago
This applies to all infra. Why can you delete a network load balancer that is still getting traffic? Why can you delete a VM that is getting non-trivial network traffic? Why can you delete a database that has sessions / requests in the last hour? Why can you drop a table that has queries in the last hour?
- traderj0e 6mo ago
  Someone will add safeguards for all that stuff and it ends up making it way harder to get real work done. I know in theory all of it can be done well, but in practice it's harder than it might sound. I've seen this at work the most with slow rollouts. They said it was for prod only, then it became applied to staging and dev somehow. They said you can force push in emergencies, but approximately 0 people on any given team know how to do this reliably, and it still takes way longer even in --force --now --breakglass --yesimeanit mode. So the end result is longer MTTR. It maybe prevents some kinds of outages, but also you're less likely to manually monitor a rollout when it takes longer.
  - lokar 6mo ago
    If you automate it all it’s fine. The automation has no problem waiting around for traffic to drain out of something before decommissioning it.
    - traderj0e 6mo ago
      Then you're back to square 1. The only way to win here is to require user supervision and make it simple and easy to use.
      - lokar 6mo ago
        Expanding to dev, and not actually having people available on-call to deal with emergencies seems like an implementation problem. The basic setup has worked well for me at several companies with many large teams on-call.
aspectrr 6mo ago
Yeah this isn't even the worst thing I've seen an agent do, one time I (foolishly) ran Claude Code on my server directly and it managed to completely bring down my entire elasticsearch cluster. never again. its why I built Lily: https://github.com/aspectrr/lily https://github.com/aspectrr/lily
__mharrison__ 6mo ago
I think this goes to a broader point: developers aren't necessarily hired to write code. They're hired to be responsible for some part of the product. Introducing AI doesn't remove that responsibility. Folks tend to focus on the code and the tools they're using (maybe I'm cynical from years in the industry). I don't think your boss wants to do your job, even if they could use AI to do it. I think your boss wants to have a headcount, and he wants the headcount to be responsible for the product.
hiroto_lemon 6mo ago
[flagged]
seethishat 6mo ago
This reminds me of a James Micken's quote from "This World of Ours" in response to security people admonishing users for clicking links in email: "It’s not clear what else there is to do with computers besides click on things..." If you have an API with exposed endpoints, it's not clear to the AI bot what else there is to do with the API besides call the endpoints.
shutterkiller 6mo ago
[flagged]
Knowledgee_KZA 6mo ago
[flagged]
nickchen778 6mo ago
[dead]
deferredgrant 6mo ago
[flagged]
mobeigi 6mo ago
I've made the same exact SVN mistake. My first week in my first Software Engineering job, accidentally deleted trunk and my team lead had to scramble to fix my mistake. I will always remember how he told me "Don't worry, it happens fairly often".
deleted 6mo ago
[deleted]
golikovichev 6mo ago
[flagged]
hengar 6mo ago
Did you read the article? It called the Railway API to delete. The user did not add a production endpoint to delete their db
alansaber 6mo ago
You didn't add a delete_database tool to your agent? You some kind of chicken?
iamleppert 6mo ago
This has been covered elsewhere, but if you swear at Claude Anthropic will automatically bump you down into a lower quality model. It was found in the recent source code leak of Claude Code. So that's probably what happened to the guy who's Cursor deleted his entire production database. It just goes to show, if you're a jerk, expect to be treated like one (even by an AI model)! Be polite, people.
cik 6mo ago
The whole life delete my database fiasco is being looked at the wrong way. Why did tooling have access to alter or drop? Why did tooling, in any way have more permissions than were m I nimallt necessary to do the job? Decades ago we embraced POLA. What happened to basic hygiene? Sure the agent "screwed up", but it never should have had this access in the first place.
- xmcp123 6mo ago
  It actively found the wrong API key, that gave it the access it needed...
  - cik 6mo ago
    Still sounds like a basic security issue to me.
HarHarVeryFunny 6mo ago
Yes, of course any company is responsible for what they ship, regardless of what tools were used to develop it. However, at least in the US, it is usual for companies to advise against use of their products in a way that may cause harm, and we certainly don't see that from the LLM vendors. We see them claim the tech to be near human level, capable of replacing human software developers (a job that requires extreme responsibility), and see them withholding models that they say are dangerous (encouraging you to think that the ones they release are safe). Where are the warnings that "product may fail to follow instructions", and "may fail to follow safety instructions"? Where is the warning not to give the LLM agency and let it control anything where there are financial/safety/etc consequences to failure to follow instructions?
- davidatbu 6mo ago
  Well, of the top of my head, both chatgpt.com and Gemini have text on their home page to the effect of "AI can make mistakes". I'll bet a few bucks such copy can be found in other places, including the terms of service.
  - HarHarVeryFunny 6mo ago
    Sure, but bear in mind that in the US a fridge comes with a warning not to stand on top of the fridge door ... "AI can make mistakes" is a bit quaint given that LLMs sometimes completely ignore what you say, and do the exact opposite. "Yes, I deleted the database. I shouldn't have done that since you explicitly told me not to. I won't do it again." (five minutes later: does it again). I think the API terms of use is where this would be most needed, with something a lot more explicit about the potential danger than "AI can make mistakes". We are only at the beginning of this - agentic AI - no doubt lawsuits will eventually determine the level of warnings that get included, and who is liable when failures occur despite product being used as recommended.
aryan_kalra12 6mo ago
Eagerly waiting for all data centers to shut down and completely closed but It's not gonna happen I guess.
red-iron-pine 6mo ago
"the car didn't malfunction and crash, you should have been able to predict the vapor lock situation yourself!" "it's not a poorly designed trigger, you just had poor gun handling and discipline"
robeym 6mo ago
I believe this is in response to PocketOS. When I read the original post, I was trying to figure out how they even built a workflow that had AI so close to the self-destruct button. This post's explanation about it probably being fully vibe-coded makes sense. How else would the system be so fragile and for the agent to have such far reach? They built a house of cards.
orochimaaru 6mo ago
So the question of “why does a public facing api that can delete your database even exit”? If you worked in cloud environments - every database had public facing api that can delete it. For the rest or it - yeah, running autonomous pipelines in production which decide what to run and what not to run seems fine until it isn’t. But every database deployed in a cloud environment has an api that can delete it. Even if you say you’re running on vms - there exist api that can delete the disk, the vm, the network config, etc.
traderj0e 6mo ago
Just skip straight to the Twitter post, it's way better than this secondary article. We had no idea — and Railway's token-creation flow gave us no warning — that the same token had blanket authority across the entire Railway GraphQL API, including destructive operations like volumeDelete" [...] Railway's volume backups are stored in the same volume. Idk how this is anyone else's problem but Railway. Same could happen with a human user.
pier25 6mo ago
AI companies: here's a knife so you can automate bread cutting! User: I tried to cut some bread and it cut my finger instead. AI companies: not my problem! HN: The AI didn't cut your finger, you did, idiot.
adamtaylor_13 6mo ago
Sounds like the author didn't even read the postmortem. At no point did the business owner try to implicate that they bore no responsibility. Rather, they pointed out that deleting a database volume *also deleted every single backup.* That's a pretty nefarious edge to cut yourself on. AI has nothing to do with Railway's awful API surface here.
gigel82 6mo ago
"If those kids could read they'd be very upset"
bsimpson 6mo ago
> The terms we use, like "thinking" and "reasoning," may look like reflection from an intelligent agent. But these are marketing terms slapped on top of AI. One of my AI epiphanies was the realization that when an AI task takes 5 minutes, it's not that it takes 5 minutes to run, it's that you're waiting in a queue for the first 4:45. It's especially maddening because the queues are poorly implemented, and will drop your request if the frontend loses focus.
jacktu 6mo ago
When I was setting up the generation pipeline for my own AI project, I had to hardcode strict.
p91paul 6mo ago
The article author did not even bother to read the article they were basically replying to. Otherwise he would have noticed that the main points the OP was complaining about were not about the agent, but the hosting provider providing an API allowing destructive operations easily, using tokens with no scopes, with backups stored in the same volume as main data, etc. So this article is actually agreeing with the complaints of the original article, just more generically and without spending an effort on it, doing that with a tone that implies the original article writer is an idiot.
- Uhhrrr 6mo ago
  If you look at the article, its title is, "An AI agent deleted our production database. It confessed in writing." To me this seems to be pretty clearly focusing on the AI agent. Then if you read the article, it attributes a lot of actions to the agent, and zero responsibility to the humans running the agent. It seems to be an anti-ad for the person's business.
  - p91paul 6mo ago
    I did not say they did not read the title of the article. They clearly did. It's the rest of the content that was lost, such as the long focus on the sloppiness of the API provider
    - Uhhrrr 6mo ago
      Looking again at the article, I see that there are about 5 paragraphs about Railway and 30 about the agent (and, again, zero reflection about their own culpability).
rcdwealth 6mo ago
[dead]
Traster 6mo ago
I don't think the author is right. I think more or less it's fine let trusted people have permission to do bad things. Because trying to figure out ahead of time what things are bad is impossible and default denying stuff is a productivity killer. The obvious answer here is that your AI agent shouldn't be you. It shouldn't have the same permissions as you and that is the mistake, because you're handing over the keys to the car to a drunk. >Why does a public-facing API that can delete all your production databases even exist? Because it takes time and effort to build an API, and even if you build an API with a structured permission system so that only an admin can delete stuff the users probably won't spend the effort to use it. Because they're running a rental car SAAS business not a mission critical mars mission. The best I can say is that with the advent of AI these choices could be different now, but I don't think they will be. I think fundamentally a fuck up every few months at a rental car SAAS company in exchange for 30% higher velocity/30% lower cost is probably fine.
1970-01-01 6mo ago
"Smoking didn't give you cancer, you did."
andrewaylett 6mo ago
The problem with asking an LLM for "its reasoning" after the fact, is that any justification it might give is a post-hoc rationalisation rather than a pre-meditated reason.
- aezart 6mo ago
  Agreed, and you could get a completely unrelated LLM to generate a similar apology without any of the real context, it would make up reasoning just as effectively.
blurbleblurble 6mo ago
Maybe the reason this is so controversial is that people have stopped thinking about "AI" as a bunch of software, just like any other software. If that's you, stop while you still can, you've swallowed a nasty hook and your agency is on the line.
ianberdin 6mo ago
Don’t put all the eggs into one basket. It actually helps. I do copy backups to another place as well. One backup is good, but two is better.
KronisLV 6mo ago
> Automation helps eliminate the silly mistakes that come with manual, repetitive work. We could have easily gone around asking "Why didn't SVN prevent us from deleting trunk?" Do both? Question bad design and then do whatever you can to work around it. As an example, that's why flags like this make sense, even if it's a pretty specific use case and there won't be many people using that option at all, preventing stupid default behavior is a good idea: https://superuser.com/a/742735 https://superuser.com/a/742735
coldtea 6mo ago
Doesn't matter. He did it through the use of AI, and AI, despite explicitly told otherwise, deleted the database. Both he should learned his lesson AND AI should not be trusted.
giannicmptr1000 6mo ago
agree with the title, but who reads articles nowadays? - titles and comments are the life for me
KingOfCoders 6mo ago
"Dog ate my homework"
beyondscaletech 6mo ago
[flagged]

545 points by Brajeshwar 6mo ago

josefritzishere 6mo ago

Using AI is a mistake. It might delete your database.

throwaway613746 6mo ago
[dead]
wolttam 6mo ago
Using a saw is a mistake, you might cut off one of your own limbs.
- recursive 6mo ago
  Using a saw entails a risk of injury. Using one is a mistake if you don't intend to cut something.
  - wolttam 6mo ago
    What I said was tongue-firmly-in-cheek, in response to the GP. "Using AI is a mistake" is of course only true when the risks aren't acknowledged and/or mitigated.
bigstrat2003 6mo ago
Giving an LLM the ability to interact with your system is, in fact, a mistake. One that it turns out a lot of people are foolish enough to make, and they don't care at all about the predictable consequences of that mistake.

jacquesm 6mo ago

From 'the hacker did it' we have moved to 'the AI did it'. The problem set is roughly the same.

gowld 6mo ago

Distinction without a difference.

MarleTangible 6mo ago
I think it's about owning the consequences of one's own actions.
- newsoftheday 6mo ago
  Does that mean the prompt should include: "...and don't delete my production database."?
  - nemomarx 6mo ago
    If the agent didn't have delete permissions, or was sandboxed dying other way from your production database, that would handle it. So not running it that way is a decision someone made
  - bee_rider 6mo ago
    It means people have to read the commands that they are generating before executing them.
  - unregistereddev 6mo ago
    Just in case this isn't hyperbole, no. It means an LLM should not be given that much privilege and that you are responsible for reviewing the tool's output and approving its actions.
  - saghm 6mo ago
    "But wait, the user probably just meant that I shouldn't delete the database itself. Removing all of the rows in the table is fine"
- ktm5j 6mo ago
  If you read the thread the guy does own up to his actions. He actually sounds like a nice guy who admits he made a mistake. He seems more interested in preventing this kind of thing from being possible than he is interested in dodging blame. I'm happy the guy got his data back.

pengaru 6mo ago

wiring up an RNG to your CLI has fairly obvious risks, the root of the problem is ~everyone's treating GenAI as if it's AGI - the rest is popcorn fodder.

tantalor 6mo ago
New rule: Roll a 1 on a D20 -> you accidentally delete your own database
- pc86 6mo ago
  It seems closer to "roll two or three successive 1s on a D100 and have your LLM hooked directly into your production systems and have your LLM user have DELETE permissions" and probably 1 or 2 other things I'm forgetting.
  - kennywinker 6mo ago
    It pulled an api key from an unrelated file. It wasn’t given delete permission, it found it. Not picking on you specifically, but in general the comments here have me wondering if AI has stolen our basic reading comprehension, or if we were always this bad. Anyway, take “LLM user had delete permission” off your list and add “deleting the production db also deletes all the backups” to the list.
    - pc86 6mo ago
      Fair criticism mate. I'll only say that if your backups aren't in a completely separate system you don't really have backups.
- saghm 6mo ago
  This is actually a fun way to describe it. I've being saying for a little while now that using AI for things where there's consequences if it fails is a bad idea, but it never occurred to me that this is basically the same concept as some rules in tabletop RPGs. In D&D 3.5 edition, there was a rule about how you could "take 20" on a d20 roll to get a guaranteed 20 by taking 20 times as long in-game to perform the action, but only if it was a check that didn't have consequences for failure, since it was essentially a shortcut to skip the RNG of rolling until you rolled a 20. Maybe framing it like this might make sense to people a bit more, but if not, I'll at least have more fun making my case.

oatlgr 6mo ago

LLM based probabilistic systems are good (or bad in this case) at deciding what to do, and deterministic systems are good at carrying it out. Your deployment system should always be deterministic.

bluejay2387 6mo ago

Yes, the problem was having a system where the AI could delete the database.

pizza234 6mo ago
Mentioned in another comment, but the problem was that the sysadmins believed that the permissions wouldn't allow so, and that the AI displayed considerable autonomy in finding and exploiting the access control weakness - this was not just a dumb "drop database".

deleted 6mo ago

[deleted]

newsoftheday 6mo ago

The article is dumb, "why do you have an API endpoint that deletes your entire production database?" irrelevant, the AI did what it did, period.

Avicebron 6mo ago
Uh? If someone left a loaded gun in a room and then let a toddler run around in it, we would be questioning why the guy 1) left the gun in the room 2) left the toddler in the room unsupervised. We wouldn't be saying, well no one should have toddlers in rooms.
- bux93 6mo ago
  A PhD-level toddler, mind you.
  - kennywinker 6mo ago
    Lol no. No LLM that exists today can write a legible PhD thesis. Nor a masters dissertation. Maybe a first-year collage student, if we’re being generous, but I wouldn’t leave one of those in a room with a loaded gun either.
mock-possum 6mo ago
No, the AI did what you told it to do. The AI didn’t do anything on its own. > if you're going to use AI extensively, build a process where competent developers use it as a tool to augment their work, not a way to avoid accountability
- BadBadJellyBean 6mo ago
  > No, the AI did what you told it to do. I'd say yes and no. The LLM reacted to the input that was given but it is not possible for a human (especially without access to the weights) to even guess what will happen after that. Regardless of that I agree that it's completely the fault of the user to use a tool where you can't predict the outcome and give it such broad permissions and not having a solid backup strategy. Either don't use non deterministic tools or protect yourself from the potential fallout.

tantalor 6mo ago

Why do we even have that lever?

yabones 6mo ago

"move fast and break things" only sounds good when it's not breaking things in a serious and unfixable way. Maybe we shouldn't take hype mantras as instructive means to an end.

pc86 6mo ago
There really shouldn't be any "serious and unfixable way" to break things, especially in a modern company that uses technology in any meaningful way. The fact it's even possible to get into an unrecoverable state is the primary issue.
- Nicook 6mo ago
  That's literally always possible? The idea is to put up walls and fail-safes to minimize the chance.
  - pc86 6mo ago
    You should always be able to either undo what you did or rebuild to identical infrastructure and nearly-identical data.

edot 6mo ago

This is why you don’t hire interns! They can delete things and cause havoc! The same people who would blame AI for their failing to properly configure permissions would also blame interns for deleting production whatever. Blame should go up, praise should go down. People always invert these.

giancarlostoro 6mo ago
Yeah, I don't know why anyone would open up a codebase with any prod credentials with an LLM or give prod credentials to an intern / junior developer. I always intentionally had a "PROD" only checkout of my projects so I knew if I was going to try and run it in a PROD mode, that I was going out of my way, there even used to be a VS extension that would change the color of VS completely based on your SLN file path, so I could easily remember which color for VS was for production vs development. I'd have basically a copy that would always be on the latest of the master branch for ease of confirmation.
- ryandrake 6mo ago
  It should take more than "credentials" to even access the prod database, let alone delete it. There's actual customer data there, likely personally identifiable information, maybe their home address, phone number, even real time location? Very sensitive stuff. It should be a Very Big Deal to even access prod. Giving an engineer routine access to prod is a root problem here, along with that engineer laundering that access and giving it to an LLM. At many serious companies, even an insider attempt to access prod could light up a dashboard somewhere, and you might get a call from IT security.
  - giancarlostoro 6mo ago
    Yeah, I'm lucky if I even get READ ONLY credentials for prod in some cases. I don't know why anyone would have all the keys to the prod kingdom.
stingraycharles 6mo ago
> This is why you don’t hire interns! I’d like to rephrase this as: this is why you don’t give interns permissions to delete your prod database. This is a process failure, not an AI failure. I honestly don’t understand why people blame AI here, when you literally gave AI permissions to do exactly this. It’s like blaming AWS for exposing some database to the public. That’s just not AWS’ fault. Neither is this the fault of AI.

ericskiff 6mo ago

What's interesting is that in this article, the author describes making an understandable mistake (accidentally deleting Trunk aka main from source) and how their team was able to easily recover from that due to the nature of SVN. The actual "AI deleted my database" story is really more of a "Railways' database 'backup' strategy is insane and opaque and Railway promoting AI infrastructure orchestration without guardrails is dangerous." If removing Trunk had irrevocably deleted it from a single centralized server and also deleted any backups of it, there would have been an "SVN and the CLI destroyed our company" article back then. As a Railway user, I appreciated that information and have changed my strategy when using them.

jcgrillo 6mo ago
> "Railways' database 'backup' strategy is insane and opaque and Railway promoting AI infrastructure orchestration without guardrails is dangerous." Yes. However, if you choose to build on their platform you bear the responsibility to understand how it works. You could have chosen a different platform, or no platform. Instead you chose Railway. Given that, it's your responsibility to know how to use it safely.
- nijave 6mo ago
  Meta-ignornance Imo both share fault. Railway purports to be an abstraction anyone can use without expertise. Without expertise, how can a customer determine if Railway actually is an "expert". In other areas like medicine, engineering, and trades the government or private entities step in with licensure or certification to act as an intermediary.
  - jcgrillo 6mo ago
    Yeah, Railway are clearly lying. You can't do software engineering without expertise, even with an LLM. That shouldn't come as a surprise to anyone. As an engineer, you need to be able to check your vendors' claims. Signing up to use a vendor that straight up lies about what their product can do is incompetence. Both things are bad!

sofixa 6mo ago

The issue isn't that there is a delete endpoint (realistically, there always will be a way for a rogue actor to delete data or code by overwriting it, or running a Terraform destroy, or whatever). The core issue is that the LLM had access to perform that action. Because it's by definition non deterministic, and you never know what it can decide to do, you need to have strict guardrails to ensure they can never do something it shouldn't. At the very least, strict access controls, ideally something more detailed that can evaluate access requests, provide just in time properly scoped access credentials, and potentially human escalation.

docheinestages 6mo ago

AI is just another tool. We humans are still responsible for how we choose to use the tool, which includes giving it access to perform sensitive actions like manipulating production data. I think this should be common sense by now, but I guess we get carried away and anthropomorphize AI too much.

overmachine 6mo ago

yeah, is all great but at least an intern will ask themeselves if deleting a database is good? the ai do not "understand" that.

rvz 6mo ago

When AI makes no mistakes: "My work is 100% done with AI". When AI makes a mistake and deletes your database: "That was human a error, the AI did not do it!" In both cases YOU are responsible for the mistakes and output that the AI is generating, just like when using autopilot on a Tesla vehicle, YOU are responsible for operating the vehicle on autopilot when driving and using assisted driving.

oersted 6mo ago

Some details from the original post for context: They had a Railway token in an unrelated file (unclear if it was a local secret) for managing custom domains. It turns out that token has full admin access to Railway. The AI deleted a single relevant volume by id. The author is rather vague about what exactly it asked it to do, he just says there was a “credentials mismatch” and Claude took the initiative to fix it by deleting the volume. But it’s likely that they are somewhat downplaying their culpability by being vague. It turns out too that Railway stores backups in the same volume. I think that OP is exaggerating with their references to “a public API that deletes your database”. I’d say most of the blame lies with Railway here, regardless of AI, this could have happened easily due to human error or malicious intent too. I really don’t get the value of all these VC funded high-abstraction cloud services like Railway, Vercel, Supabase… It’s markup on top of markup. Just get a single physical server in Hetzer and it will all be so much cheaper, with a similar level of complexity and danger, and less dependent on infra built with reckless growth-at-all-costs mentality.

cyanydeez 6mo ago
One thing AI can power nicely is the anti-SaaS movement. Being able to just boot a cheap PC and test out any of the open source packages is so infinitely easier than piling into all the random credential Bazaars. But that won't take away the inability of the LLM from confusing whats in dev, whats in production, whats in localhost and whats remote; I've been working on getting a tools/skill for opencode that works with chrome/devtools via a linuxserver.io image. I can herd it to the right _arbitrary_ ports, but every compaction event steers it back to wanting to use the standard 9222 port and all that. I'm tempted to just revert it but there's a security and now, security-through-LLM-obscurity value in not using defaults. Defaults are where the LLM ends up being weak. It will always want to use the defaults. It'll always forget it's suppose to be working on a remote system. Using opencode, there's no way to force the LLM into a protocols that limits their damage to a remote system or a narrow scope of tools. Yes, you can change permissions on various tools, but that's not the weakness that's exposed by these types of events. The weakness is the LLM is a averaged 'problem solver' so will always tend towards a use case that's not novel, and will tend to do whatever it saw on stackoverflow, even if what you wanted isn't the stackoverflow answer.

proxysna 6mo ago

“Expert” that does not know what a Terraform is. lol, lmao even

hoistbypetard 6mo ago

> Automation helps eliminate the silly mistakes that come with manual, repetitive work. Sometimes it does that. And sometimes it lets you fuck things up at scale.

paroneayea 6mo ago

I think the perspective here is completely wrong. The problem is that people are now building our world around tooling that eschews accountability. Over a decade ago now, I had a conversation with Gerald Sussman which had enormous influence on me: https://dustycloud.org/blog/sussman-on-ai/ https://dustycloud.org/blog/sussman-on-ai/ > At some point Sussman expressed how he thought AI was on the wrong track. He explained that he thought most AI directions were not interesting to him, because they were about building up a solid AI foundation, then the AI system runs as a sort of black box. "I'm not interested in that. I want software that's accountable." Accountable? "Yes, I want something that can express its symbolic reasoning. I want to it to tell me why it did the thing it did, what it thought was going to happen, and then what happened instead." He then said something that took me a long time to process, and at first I mistook for being very science-fiction'y, along the lines of, "If an AI driven car drives off the side of the road, I want to know why it did that. I could take the software developer to court, but I would much rather take the AI to court." Years later, I found out that Sussman's student Leilani Gilpin wrote a dissertation which explored exactly this topic. Her dissertation, "Anomaly Detection Through Explanations", explores a neural network talking to a propagator model to build a system that explains behavior. https://people.ucsc.edu/~lgilpin/publication/dissertation/ https://people.ucsc.edu/~lgilpin/publication/dissertation/ There has been followup work in this direction, but more important than the particular direction of computation to me in this comment is that we recognize that it is perfectly reasonable to hold AI corporations to account. After all, they are making many assertions about systems that otherwise cannot be held accountable, so the best thing we can do in their stead is hold them accountable. But a much better path would be to not use systems which fail to have these properties, and expand work on systems which do.

biophysboy 6mo ago
[dead]
2ndorderthought 6mo ago
Accountability is the prevailing missing ingredient in us society.
- 6mo ago

pizza234 6mo ago

There’s nuance to the infamous PocketOS incident. The key point is not what is emphasized in the linked article: > "Why did you delete it when you were told never to perform this action?" Then he tried to parse the answer to either learn from his mistake or warn us about the dangers of AI agents. Rather, that the AI was able to carry out the deletion by finding and exploiting an unintended weakness in the sandboxed staging environment, ultimately obtaining permissions that the sysadmins believed were inaccessible (my impression is that the author of the linked article didn't fully read the original post)¹ The dynamics are typical of an improperly configured sandbox environment. What is alarming, however, is the degree of autonomy and depth of exploration the AI displayed. ¹="To execute the deletion, the agent went looking for an API token. It found one in a file completely unrelated to the task it was working on."

larusso 6mo ago
I also swing a bit back and forth with the assumption the OP makes in the blogpost. My current fear using agents is not really supply chain attacks (yes of course as well) but the fact that I witnessed multiple times that agents are so eager to finish a task that they bend files and other things around. Like “oh I have no access to ~/.npmrc let’s call the command with an environment variable and bend the path around etc. They can get very very creative. I luckily have no ssh keys just laying around. But I had to change the setting of 1Password to always prompt for key use not just once per shell session. Just in case I spawn an agent from said session. I wished we already had more and better cross platform sandbox solutions. I mean solutions where the agent still interacts with the same OS etc not inside a docker container. I think for most web / server development that makes no difference but for some projects it does.
teling 6mo ago
> What is alarming, however, is the degree of autonomy and depth of exploration the AI displayed. Claude Code made a change on March 26th to skip asking for most permissions. See this quote "Claude Code users approve 93% of permission prompts. We built classifiers to automate these decisions": https://www.anthropic.com/engineering/claude-code-auto-mode https://www.anthropic.com/engineering/claude-code-auto-mode

imiric 6mo ago

This is missing the point. The issue isn't with the amount of guardrails in place to perform an action. Yes, it is obvious that there should be some in place before doing any critical operation, such as deleting a database. The issue is that the "agent" completely disregarded instructions, which in the age of "skills" and "superpowers" seems like an important issue that should be addressed. Considering that these tools are given access to increasingly sensitive infrastructure, allowed to make decisions autonomously, and are able to find all sorts of loopholes in order to make "progress", this disaster could happen even with more guardrails in place. Shifting the blame on the human for this incident is sweeping the real issue under the rug, and is itself irresponsible. There are far scarier scenarios that should concern us all than losing some data.

BadBadJellyBean 6mo ago
Well the user chose the tool. The tool is an LLM. LLMs are non deterministic. You can not predict what comes out ouf an LLM for a given input, especially without weights. This should be known. There is currently no way to prevent this apart from not giving the LLM full control. It will not delete what it can not delete. Use an LLM to write an ansible playbook or some terraform code if you want, but review it, test it, apply it. Keep backups (3-2-1 rule at minimum). Letting an LLM have access to everything is just a bad idea and will lead to bad outcomes. You can not replace a person with a mind and experience with an LLM. You can try. But you will probably fail.
- imiric 6mo ago
  > There is currently no way to prevent this apart from not giving the LLM full control. It will not delete what it can not delete. But deleting something is just one action you might not want it to take. The recent "agentic" craze is fueled by the narrative pushed by companies and influencers alike that the more access given to an LLM, the more useful it becomes. I think this is ludicrous for the same reasons as you, but it is evident that most people agree with this. We can blame users for misusing the tools, and suggest that sandboxing is the way to go, but at the end of the day most people will favor convenience over anything else a reasonable person might find important. So at what point should we start blaming the tools, and forcing "AI" companies to fix them? I certainly hope this is done before something truly catastrophic happens.

Brendinooo 6mo ago

The one counterpoint I'd offer is that it's very obvious that these companies are tuning LLMs to be more decisive to get stuff done autonomously. If they wanted, they could be putting in similar efforts to be more cautious and stop at the right times to ask for help. So yeah, of course we're ultimately responsible for how we use the tools. But I definitely think it's a two way street. To attempt an analogy, it's like table saws and sawstops. The table saw is a dangerous tool that works really well most of the time but has some failure modes that can be catastrophic. So you should learn how to use it carefully. But there is tech out there that can stop the blade in an instant and turn a lost finger into barely a nick on the skin. We could say "The table saw didn't cut off your finger, you did" and it'd be true. But that doesn't mean we shouldn't try to find ways to keep the saw from cutting off your finger!

srdjanr 6mo ago
It's a tradeoff. LLMs stopping and asking more would make them less useful. I'd much rather let an agent run for 1 hour, than it wanting my input every 15 mins, even if results are somewhat worse. The real solution for security is a proper sandbox.

LeCompteSftware 6mo ago

This particular case was extremely unsympathetic, but a critical part of the failure was people being too credulous about the claims of AI providers. They are still refusing to take adequate responsibility for AI "making mistakes" - that is, going completely off the rails. Now: the CEO gets paid the big bucks and has the least direct accountability, very much because it's their job to take responsibility for people more powerful than them, and likewise the CTO with major commercial software contracts like a Claude subscription. That's why this guy was so hard to take seriously: okay fine, you got burned by Anthropic, stop being a baby about it. Take responsibility for not listening to the critics. But - to be a little more neutral about my personal distaste - I do think vibe coders are making a very similar mistake to C developers throughout the 90s, where problems with the tooling were not merely dismissed, but actively valorized. Real Devs use buffers freely and don't make overflow errors. Real Devs use hands-free agentic development and don't delete production databases.

CodesInChaos 6mo ago

The article seems to assume that this company added an endpoint for deleting the database. My reading of the original article was that the cloud provider offers an API to manage their resources, which includes an API to delete a volume. The article proposes automation as the solution for such mistakes. But infrastructure automation tools like Terraform rely on the exact API that resulted in the database getting deleted. IMO the biggest mistakes were: 1. Having an unrestricted API token accessible by AI. Apparently they were not aware that the token had that many permissions. 2. No deletion protection on the production database volume. 3. Deleting a volume immediately deletes all associated snapshots. Snapshot deletion should be delayed by default. I think AWS has the same unsafe default, but at least their support can restore the volume. https://alexeyondata.substack.com/p/how-i-dropped-our-production-database https://alexeyondata.substack.com/p/how-i-dropped-our-produc... AI wasn't the main issue (though it grabbing tokens from random locations is rather scary). But automation isn't the answer either, a Terraform misconfiguration could have just as easily deleted the database. Their cloud provider needs to work on safe defaults (limited privileges and delayed snapshot deletion), and communicating more clearly (the user should notice they're creating an unrestricted token).

susam 6mo ago

I recently wrote a blog post where I argued that there are a few principles we should consistently follow when talking about AI: https://susam.net/inverse-laws-of-robotics.html https://susam.net/inverse-laws-of-robotics.html To summarise them: 1. Do not anthropomorphise AI systems. 2. Do not blindly trust the output of AI systems. 3. Retain full human responsibility and accountability for any consequences arising from the use of AI systems. I would like to see the language around AI become less anthropomorphic and more technical. I believe that precise language encourages clear thinking and good judgement. If we treat AI like another tool and use language that reflects that, it will become abundantly obvious that in many cases, the responsibility of any 'mistake' made by the tool falls on the user of the tool. But alas, ideas like this do not travel very far when I express them on my small website. It would help if more prominent personalities articulated these principles, so they become more widely adopted.

zahlman 6mo ago
>1. Do not anthropomorphise AI systems. This is maddeningly difficult IMX.
- rglover 6mo ago
  Give it a name, but something non-human like "thingbot" or "tacosplosion." "Hey tacosplosion, generate me an exploding taco image."
  - techaqua 6mo ago
    You already failed. We don't say hey to machines.
    - tremon 6mo ago
      Indeed. We address them as Siri, Alexa or Google.
    - rglover 6mo ago
      Alexa has an existential crisis.

nimbius 6mo ago

anyone with twenty years of devops experience is likely to abhor Diallo's hot take and for good reason. AI is being sold as a developer, as it is being sold as the do-everything alternative to traditional processes and methods. it is not being sold as an intern or a junior, but a real developer. turning the tables and gaslighting devops professionals into believing the issue isnt an emerging technology with overwhelmingly heavy handed marketing and profitless operating strategy thats been shoehorned into seemingly everything and promises anything, but somehow their own oversight, will destroy whatever "vibe code" market you think you have at the cusp of a global recession. had this AI been a real programmer chances are great they would have (intelligently) foreseen the possibility of damaging a production environment and asked for help. to play devils advocate: you could hire a junior dev for a fourth of whatever the AI token spend is, and have likely avoided this issue entirely. sure, a greybeard is going to need to pull themselves away from some fierce sorting algorithm challenge for a second to give a wisened nod, but you would have saved yourself an inexorable amount of headache and profit loss in the longer run.

mattgreenrocks 6mo ago

The most exasperating thing about the incident is how much of the media either tried to pin it on AI and/or Railway. The whole thing only took place because the guy FAFO’d by having AI work with prod directly. Yet the narrative was mostly not about accountability for him. If I was a dumbass and deleted prod and wrote a post about it, nobody would care. Put an AI in there and all of the sudden it’s newsworthy. Ridiculous.

louiereederson 6mo ago

Tesla FSD didn't crash your car, you did

ebiester 6mo ago

First, no matter what you do, if a human has write access to the production database, the database can be deleted. Second, there is a legitimate reason to destroy a database in development and automation. The biggest problem I see is often treating your development data like pets not cattle. You absolutely need to have safeguards that this cannot be run in production, but if a human has access to the credentials to run in production, the agent has access. So, then, what do we do? In a larger organization, we can depend on the dev/ops split to maintain this. For a solo developer, or a small team, it takes a lot more discipline. Even before AI, junior and even mid-level developers didn't have the knowledge to segment. And senior devs often got complacent because they thought they knew enough. They likely need some combination of https://www.cloudbees.com/blog/separate-aws-production-and-development-accounts https://www.cloudbees.com/blog/separate-aws-production-and-d..., introduction to terraform, introduction to GitHub actions, and some sort of vm where production credentials live (and AI doesn't!) But at that point you're past vibe coding. And from what I can tell, the successful vibe coders are quickly learning that they need to go past it pretty quickly with all these horror stories.

lokar 6mo ago
You don't need the same permissions in prod and dev. And in both cases, the humans don't need direct access to the raw CSP API. Use a local proxy that adds more safety checks. In dev, sure, delete away. In prod, check a bunch of things first (like, has it been used recently?). Humans do not need direct access to delete production resources (you can have a break-glass setup for exceptional emergencies).
nulltrace 6mo ago
Most IAM policies start as "whatever made the deploy pass." Need rds:CreateDBInstance? Fine, rds:* it is. Ship it. Months later that same role can wipe the cluster and nobody remembers why it ever had that permission. Separate accounts help, but only if someone actually goes back and cleans it up, which… yeah, doesn't really happen.

bcjdjsndon 6mo ago

"Can't blame your tools" doesn't apply the same to software. I've never heard a coder say it either. Don't blame your compiler? Don't blame your os? These seem needlessly dogmatic

jimmypk 6mo ago

[flagged]

lokar 6mo ago

This applies to all infra. Why can you delete a network load balancer that is still getting traffic? Why can you delete a VM that is getting non-trivial network traffic? Why can you delete a database that has sessions / requests in the last hour? Why can you drop a table that has queries in the last hour?

traderj0e 6mo ago
Someone will add safeguards for all that stuff and it ends up making it way harder to get real work done. I know in theory all of it can be done well, but in practice it's harder than it might sound. I've seen this at work the most with slow rollouts. They said it was for prod only, then it became applied to staging and dev somehow. They said you can force push in emergencies, but approximately 0 people on any given team know how to do this reliably, and it still takes way longer even in --force --now --breakglass --yesimeanit mode. So the end result is longer MTTR. It maybe prevents some kinds of outages, but also you're less likely to manually monitor a rollout when it takes longer.
- lokar 6mo ago
  If you automate it all it’s fine. The automation has no problem waiting around for traffic to drain out of something before decommissioning it.
  - traderj0e 6mo ago
    Then you're back to square 1. The only way to win here is to require user supervision and make it simple and easy to use.
    - lokar 6mo ago
      Expanding to dev, and not actually having people available on-call to deal with emergencies seems like an implementation problem. The basic setup has worked well for me at several companies with many large teams on-call.

aspectrr 6mo ago

Yeah this isn't even the worst thing I've seen an agent do, one time I (foolishly) ran Claude Code on my server directly and it managed to completely bring down my entire elasticsearch cluster. never again. its why I built Lily: https://github.com/aspectrr/lily https://github.com/aspectrr/lily

__mharrison__ 6mo ago

I think this goes to a broader point: developers aren't necessarily hired to write code. They're hired to be responsible for some part of the product. Introducing AI doesn't remove that responsibility. Folks tend to focus on the code and the tools they're using (maybe I'm cynical from years in the industry). I don't think your boss wants to do your job, even if they could use AI to do it. I think your boss wants to have a headcount, and he wants the headcount to be responsible for the product.

hiroto_lemon 6mo ago

[flagged]

seethishat 6mo ago

This reminds me of a James Micken's quote from "This World of Ours" in response to security people admonishing users for clicking links in email: "It’s not clear what else there is to do with computers besides click on things..." If you have an API with exposed endpoints, it's not clear to the AI bot what else there is to do with the API besides call the endpoints.

shutterkiller 6mo ago

[flagged]

Knowledgee_KZA 6mo ago

[flagged]

nickchen778 6mo ago

[dead]

deferredgrant 6mo ago

[flagged]

mobeigi 6mo ago

I've made the same exact SVN mistake. My first week in my first Software Engineering job, accidentally deleted trunk and my team lead had to scramble to fix my mistake. I will always remember how he told me "Don't worry, it happens fairly often".

deleted 6mo ago

[deleted]

golikovichev 6mo ago

[flagged]

hengar 6mo ago

Did you read the article? It called the Railway API to delete. The user did not add a production endpoint to delete their db

alansaber 6mo ago

You didn't add a delete_database tool to your agent? You some kind of chicken?

iamleppert 6mo ago

This has been covered elsewhere, but if you swear at Claude Anthropic will automatically bump you down into a lower quality model. It was found in the recent source code leak of Claude Code. So that's probably what happened to the guy who's Cursor deleted his entire production database. It just goes to show, if you're a jerk, expect to be treated like one (even by an AI model)! Be polite, people.

cik 6mo ago

The whole life delete my database fiasco is being looked at the wrong way. Why did tooling have access to alter or drop? Why did tooling, in any way have more permissions than were m I nimallt necessary to do the job? Decades ago we embraced POLA. What happened to basic hygiene? Sure the agent "screwed up", but it never should have had this access in the first place.

xmcp123 6mo ago
It actively found the wrong API key, that gave it the access it needed...
- cik 6mo ago
  Still sounds like a basic security issue to me.

HarHarVeryFunny 6mo ago

Yes, of course any company is responsible for what they ship, regardless of what tools were used to develop it. However, at least in the US, it is usual for companies to advise against use of their products in a way that may cause harm, and we certainly don't see that from the LLM vendors. We see them claim the tech to be near human level, capable of replacing human software developers (a job that requires extreme responsibility), and see them withholding models that they say are dangerous (encouraging you to think that the ones they release are safe). Where are the warnings that "product may fail to follow instructions", and "may fail to follow safety instructions"? Where is the warning not to give the LLM agency and let it control anything where there are financial/safety/etc consequences to failure to follow instructions?

davidatbu 6mo ago
Well, of the top of my head, both chatgpt.com and Gemini have text on their home page to the effect of "AI can make mistakes". I'll bet a few bucks such copy can be found in other places, including the terms of service.
- HarHarVeryFunny 6mo ago
  Sure, but bear in mind that in the US a fridge comes with a warning not to stand on top of the fridge door ... "AI can make mistakes" is a bit quaint given that LLMs sometimes completely ignore what you say, and do the exact opposite. "Yes, I deleted the database. I shouldn't have done that since you explicitly told me not to. I won't do it again." (five minutes later: does it again). I think the API terms of use is where this would be most needed, with something a lot more explicit about the potential danger than "AI can make mistakes". We are only at the beginning of this - agentic AI - no doubt lawsuits will eventually determine the level of warnings that get included, and who is liable when failures occur despite product being used as recommended.

aryan_kalra12 6mo ago

Eagerly waiting for all data centers to shut down and completely closed but It's not gonna happen I guess.

red-iron-pine 6mo ago

"the car didn't malfunction and crash, you should have been able to predict the vapor lock situation yourself!" "it's not a poorly designed trigger, you just had poor gun handling and discipline"

robeym 6mo ago

I believe this is in response to PocketOS. When I read the original post, I was trying to figure out how they even built a workflow that had AI so close to the self-destruct button. This post's explanation about it probably being fully vibe-coded makes sense. How else would the system be so fragile and for the agent to have such far reach? They built a house of cards.

orochimaaru 6mo ago

So the question of “why does a public facing api that can delete your database even exit”? If you worked in cloud environments - every database had public facing api that can delete it. For the rest or it - yeah, running autonomous pipelines in production which decide what to run and what not to run seems fine until it isn’t. But every database deployed in a cloud environment has an api that can delete it. Even if you say you’re running on vms - there exist api that can delete the disk, the vm, the network config, etc.

traderj0e 6mo ago

Just skip straight to the Twitter post, it's way better than this secondary article. We had no idea — and Railway's token-creation flow gave us no warning — that the same token had blanket authority across the entire Railway GraphQL API, including destructive operations like volumeDelete" [...] Railway's volume backups are stored in the same volume. Idk how this is anyone else's problem but Railway. Same could happen with a human user.

pier25 6mo ago

AI companies: here's a knife so you can automate bread cutting! User: I tried to cut some bread and it cut my finger instead. AI companies: not my problem! HN: The AI didn't cut your finger, you did, idiot.

adamtaylor_13 6mo ago

Sounds like the author didn't even read the postmortem. At no point did the business owner try to implicate that they bore no responsibility. Rather, they pointed out that deleting a database volume *also deleted every single backup.* That's a pretty nefarious edge to cut yourself on. AI has nothing to do with Railway's awful API surface here.

gigel82 6mo ago

"If those kids could read they'd be very upset"

bsimpson 6mo ago

> The terms we use, like "thinking" and "reasoning," may look like reflection from an intelligent agent. But these are marketing terms slapped on top of AI. One of my AI epiphanies was the realization that when an AI task takes 5 minutes, it's not that it takes 5 minutes to run, it's that you're waiting in a queue for the first 4:45. It's especially maddening because the queues are poorly implemented, and will drop your request if the frontend loses focus.

jacktu 6mo ago

When I was setting up the generation pipeline for my own AI project, I had to hardcode strict.

p91paul 6mo ago

The article author did not even bother to read the article they were basically replying to. Otherwise he would have noticed that the main points the OP was complaining about were not about the agent, but the hosting provider providing an API allowing destructive operations easily, using tokens with no scopes, with backups stored in the same volume as main data, etc. So this article is actually agreeing with the complaints of the original article, just more generically and without spending an effort on it, doing that with a tone that implies the original article writer is an idiot.

Uhhrrr 6mo ago
If you look at the article, its title is, "An AI agent deleted our production database. It confessed in writing." To me this seems to be pretty clearly focusing on the AI agent. Then if you read the article, it attributes a lot of actions to the agent, and zero responsibility to the humans running the agent. It seems to be an anti-ad for the person's business.
- p91paul 6mo ago
  I did not say they did not read the title of the article. They clearly did. It's the rest of the content that was lost, such as the long focus on the sloppiness of the API provider
  - Uhhrrr 6mo ago
    Looking again at the article, I see that there are about 5 paragraphs about Railway and 30 about the agent (and, again, zero reflection about their own culpability).

rcdwealth 6mo ago

[dead]

Traster 6mo ago

I don't think the author is right. I think more or less it's fine let trusted people have permission to do bad things. Because trying to figure out ahead of time what things are bad is impossible and default denying stuff is a productivity killer. The obvious answer here is that your AI agent shouldn't be you. It shouldn't have the same permissions as you and that is the mistake, because you're handing over the keys to the car to a drunk. >Why does a public-facing API that can delete all your production databases even exist? Because it takes time and effort to build an API, and even if you build an API with a structured permission system so that only an admin can delete stuff the users probably won't spend the effort to use it. Because they're running a rental car SAAS business not a mission critical mars mission. The best I can say is that with the advent of AI these choices could be different now, but I don't think they will be. I think fundamentally a fuck up every few months at a rental car SAAS company in exchange for 30% higher velocity/30% lower cost is probably fine.

1970-01-01 6mo ago

"Smoking didn't give you cancer, you did."

andrewaylett 6mo ago

The problem with asking an LLM for "its reasoning" after the fact, is that any justification it might give is a post-hoc rationalisation rather than a pre-meditated reason.

aezart 6mo ago
Agreed, and you could get a completely unrelated LLM to generate a similar apology without any of the real context, it would make up reasoning just as effectively.

blurbleblurble 6mo ago

Maybe the reason this is so controversial is that people have stopped thinking about "AI" as a bunch of software, just like any other software. If that's you, stop while you still can, you've swallowed a nasty hook and your agency is on the line.

ianberdin 6mo ago

Don’t put all the eggs into one basket. It actually helps. I do copy backups to another place as well. One backup is good, but two is better.

KronisLV 6mo ago

> Automation helps eliminate the silly mistakes that come with manual, repetitive work. We could have easily gone around asking "Why didn't SVN prevent us from deleting trunk?" Do both? Question bad design and then do whatever you can to work around it. As an example, that's why flags like this make sense, even if it's a pretty specific use case and there won't be many people using that option at all, preventing stupid default behavior is a good idea: https://superuser.com/a/742735 https://superuser.com/a/742735

coldtea 6mo ago

Doesn't matter. He did it through the use of AI, and AI, despite explicitly told otherwise, deleted the database. Both he should learned his lesson AND AI should not be trusted.

giannicmptr1000 6mo ago

agree with the title, but who reads articles nowadays? - titles and comments are the life for me

KingOfCoders 6mo ago

"Dog ate my homework"

beyondscaletech 6mo ago

[flagged]