9 ms·
>Well, here's the thing not enough people talk about: we're giving these tools god-mode permissions. Tools built by people we've never met. People we have zero
by Dilettante_ 1y ago
>Well, here's the thing not enough people talk about: we're giving these tools god-mode permissions. Tools built by people we've never met. People we have zero way to vet. And our AI assistants? We just... trust them. Completely.
I keep seeing this pattern in articles: "Did you know that if you point the gun at your foot and pull the trigger, yOu ShOoT yOuRsElF iN tHe FoOt??!? I couldn't believe it myself!! What a discovery!!1!"
Are people really this oblivious or are these articles written about non-issues just to have written 'content'?
- BobbyTables2 1y agoThese also remind me on the early 1980s mentality of some who thought anything printed by a computer must be correct. It’s an AI, it must be perfect! /s
- avs733 1y agoThis is unfair. It presumes a universal understanding of something largely because it is obvious to us. Most computer users have little to know detailed understanding of how any computer technology works, and because they are invisible and abstract, even less understanding of the risks they expose themselves to. The answer to you gun analogy is false because it assumes basic knowledge of a gun. This is part of why so many kids shoot themselves or family members with guns - because they don’t know if you pull the trigger something violent will happen until they are taught it.
- Dilettante_ 1y agoBut these are not children, and they were not just handed a gun. They went out and acquired one. That is what astounds me. How one can come into possession of a gun completely without understanding that it is dangerous. How fundamentally the worlds I and they live in must be for that to happen. Oh, now that I write it out like that, I've definitely been on the other side of that astoundment before, for lacking 'common sense'. Ain't that just the way.
- jacquesm 1y agoThat's because people in general are good. They don't understand that there is a small fraction of humanity that would happily erase their digital lives given half a chance because they themselves can not even conceive of someone doing a thing like that. The fact that we all can is a professional deformation, it is not normal. I lived in a place where the doors had no locks. Nobody could imagine anybody stealing from someone else's house. And so it just didn't happen. When buying the house there is the moment where the keys are transferred. The sellers somewhat sheepishly announced they didn't have any. The lawyer handling the transaction asked if they had lost their keys and if they did that they should pay for replacing the locks. Then it turned out they never had any locks in the first place and that this was pretty much the norm there. That distrust that we call common sense is where we go wrong, the fact that we've connected the whole world means that these little assholes now have access to everything and everybody and there isn't even a good way to figure out who they are and how to punish them unless they are inept at hiding their traces.
- PantaloonFlames 1y agoWhen I was growing up, we did not have a key to lock our house. I was a "latch key kid", both parents worked, but I never had a key. I remember meeting someone who locked their front door as a matter of course, and thinking that was so curious.
- sciencejerk 1y agoWhat country was this? I've heard of leaving house unlocked but never no locks at all
- jacquesm 1y agoNorthern Canada.
- lazide 1y agoDid you know that if you give a loaded gun to a chimpanzee, sometimes it will shoot you (or itself) and it didn’t even know that was going to happen?!? Even if you tell it several times?!? And that if that happens ‘smart’ people will tell you that it was really dumb to do that!!?!
- ljm 1y agoWhat’s obvious to the audience of HN isn’t necessarily obvious to anyone else. Articles like this are intended to serve the latter group of people. And it’s true, AI agents with MCP servers are pretty much unsafe by design, because security was never considered from the start. Until that changes, if it ever even does, the best thing to do is to inform.
- askl 1y agoDo people outside the HN audience exist that are using MCP servers?
- brazukadev 1y agoWell, yes but they are not less nerd (actually, HN audience is not like it used to be, too)
- didericis 1y ago> the best thing to do is to inform. While also not using them yourself/actively trying to find and strip them out of workflows you have control over.
- dawnerd 1y agoThe recent npm supply chain attacks were directly targeting people in the HN crowd. Don’t think because you’re here and know tech that you’re immune.
- lsaferite 1y agoThe npm supply chain attacks (or any similar ones) are essentially the same issue described in the article. You can't trust 3rd-party provided code implicitly. Even if the code is initially fine it's subject to change in later revisions. This issue goes all the way down the stack. Obviously, with a large user base the likelihood of quick detection goes up, but the issue never goes away.
- worik 1y ago
- kemayo 1y agoThere was that article a few months ago about an AI code assistant deleting a company's production database: https://fortune.com/2025/07/23/ai-coding-tool-replit-wiped-database-called-it-a-catastrophic-failure/ https://fortune.com/2025/07/23/ai-coding-tool-replit-wiped-d... There's layers here, of course: 1. The founder shot themselves in the foot by not understanding an AI tool couldn't be trusted, so clearly they really were that oblivious. 2. ...the founder had direct production access hooked up to their regular coding environment. They were always going to shoot themselves in the foot eventually.
- askl 1y agoThat's a really heartwarming story. Thanks.
- Aurornis 1y ago> There was that article a few months ago about an AI code assistant deleting a company's production database That wasn’t a company. It was a vibe coding experiment. It didn’t have any actual customers. It did delete the database marked “production” though. If it had been deployed it would have been a problem. It was just an experiment, though.
- m3047 1y agoThere was a time when email clients running scripts in emails from randos on the internet was "a game-changing opportunity for automation". Here we are again. There was a time when people thought letting their children use the internet as a babysitter was healthier than parking them in front of the television... guess that turned out another way.
- zobzu 1y agoThe problem, especially with AI, IMO is that folks are even more willing to shoot themselves in the foot. IMO this might be due to a few things like these: 1. Folks are trained on buffer overflows and SQL injections, they don't even question it, these are "bad". But an MCP interface to an API with god-like access to all data? The MCP of course will make sure its safe for them! (of course, it does not). It's learning and social issue rather than a technical issue. Sometimes, it makes me feel like we're all LLMs. 2. 300 things to fix, 100 vendors, and it needs to be done yesterday. As soon as you look into the vendor implementation you find 10 issues, because just like you, they have 300 things to fix and it needs to be done yesterday, so trade-offs become more dangerous. And who's going to go after you if you do not look into it too hard? No one right now. 3. Complete lack of oversight (and sometimes understanding). If you're just playing around by prompting an LLM you do not know what its doing. When it works, you ship. We've all seen it by now. Personally, I think this one could be improved by having a visual scheduler of tasks.
- collinmcnulty 1y agoI think the issue is that the security issue is inherent to the core story being told about AI. “It can read your emails and respond automatically” is both a pretty compelling story to certain people who aren’t waist deep in tech news, and also exceedingly dangerous.
- gdbsjjdn 1y ago[flagged]
- arcfour 1y agoDid you just desperately want to grind your political axe today? This has nothing to do with anything.
- DonHopkins 1y ago[flagged]
- arcfour 1y agoI'm sorry, we fundamentally disagree and there is no point in arguing as it is completely off-topic and our minds are made up. Your comment is unnecessarily inflammatory as is the OC's; the thread about MCP servers and their security model.
- gdbsjjdn 1y agoThere's something deeply ingrained in the American psyche that says that warnings and guardrails are for other, stupid people. You, personally, are no doubt an ubermensch who can perform every procedure flawlessly from memory every time. If there were a crisis you would be a hero and save everyone. This applies equally to "good guy with a gun" and human factors analysis of software.
- flowerthoughts 1y agohttps://xkcd.com/1053/ https://xkcd.com/1053/ Today, 10,000 Americans discovered that water is wet for the first time. Same thing tomorrow.
- scuff3d 1y agoSurely giving an unpredictable AI access to your system has absolutely no downsides at all.
- zahlman 1y ago> Are people really this oblivious Last I checked, basic SQL injection attacks were still causing massive economic damage every year and are at or near the top of the OWASP list. SQL injection is essentially the result of unintentionally giving god-mode permission to a database by failing to understand how queries are built and processed. The... agency available to AI agents might not be all that obvious either.
- OptionX 1y agoAnd we have decades of hindsight with sql injection to work with and make it obvious. No so much with all the fancy new AI tools. Yes MCP has next to no security features, but then again is it even a year old at this point? Not excusing it just pointing out something folks should me mindful of when using tool based on it, its an immature system. And heck, I still remember a time when most of the internet traffic just flew around in plain text. Insanity to us now.
- tomjen3 1y agoI am going to disagree with that. SQL injection attacks are an example of the age old issue of mixing up input and instructions. Smash the stack is older than many software devs, but it was essentially the same problem - its an inherit issue with Von Neumann architecture. This is also not an AI issue, or even an MCP issue. If the same issue had been in a client library for the Postmark API, it would likely have had a bigger impact. What we need is to make it much more likely to get caught and go to prison for stuff like this. That will change things.
- zahlman 1y ago> SQL injection attacks are an example of the age old issue of mixing up input and instructions. Yes, and attacks on AI are much the same. The AI gets "prompted" by something that was supposed to be inert processable data. (Or its basic conduct guidelines are overridden because the system doesn't and can't distinguish between the "system prompt" and "user prompt".)
- kordlessagain 1y agoSpeak for yourself.
- Terr_ 1y agoIf you look at human civilization in general, it is full of cases where people need to be convinced that their bullet-wounded feet aren't "just the way it works", that fixing it is possible, and that the wounds are serious enough to be worth it.
- ChuckMcM 1y ago> Are people really this oblivious ... Yes, yes they are. The vector of "I've been using this package and giving it full permissions for like forever and it has never been a problem." oblivious. One must understand the two-step here: Step 1) Use someone else's package that solves your problem and doesn't have any issues. Yay you're doing DRY and saving time and effort! Step 2) The package you got is now corrupted but you don't have any tools to checking to see if you're application is doing anything sus. The alternative is that you audit every release and look at every change on every file. So suddenly your DRY because a weekly/monthly audit exercise for every single imported package that changes. Use three packages, that's three audit schedules. Use ten? That's ten audit schedules. It's stupid because if you actually used this alternative you'd be spending all of your time just auditing packages and 90+% of the time not finding anything to complain about. So the ACTUAL alternative, is write your own damn code. You wrote it so you know how it works, and when you change it you know what you changed. And unless you are the bad guy, exploits don't "magically appear" in your code. Vulnerabilities will appear in your code if you don't do code reviews, but remember actualizing this stuff requires both knowing about the bug and exploiting it in the wild, you may end up coding in something that could be exploited but you are unlikely then to use your own vulnerability.
- PantaloonFlames 1y agoStep 1 sounds like npm, to me. and Step 2.... ? yeah, we know. "Write your own code" won't scale. Anyway that horse is out of the barn. We need better SBOM tools.
- ChuckMcM 1y agoI disagree, writing your own code is the only thing that does scale. Even if you do nothing more than take the NPM package get source, and then do your own maintenance on it. But having been in different organizations that chose one path or the other, the durable organizations chose to write their own code.
- cyanydeez 1y agoAre people just aching to constantly blame poor people for moral and ethical failings, and not the grifters who prey on them? Yes. People definitely blame victims all the time. There's no way LLMs, their proprietors, designers could be practicing dark patterns, marketing gimmicks and security shortcuts to convince people to shot these guns as often as possible...even when, they point them at their foot. This grift is deep and wide and you may want to re-evaluate why everyone is so keen to always blame victims for technology, society, and business marketing.
- jszymborski 1y agoThe problem with foot guns is that you know it's a foot gun and I know it's a foot gun, but the underpaid social media manager at my bank might not and suddenly it's a problem for you and I.
- tokioyoyo 1y agoIt’s a bit easy for us to say, but general public doesn’t know about the details. For example, lots of people talk how algorithms designs your personal social media feed. But the same people don’t know what an “algorithm” means. It’s just a generic word for them. Which is fine, because not everyone is in our industry.
- anal_reactor 1y agoWhat security people don't understand is that if insurance policy costs more than "cost of disaster" times "probability of a disaster" then it's most likely not worth it. They personally enjoy doing security stuff, so they don't internalize the costs of maintaining a secure environment the same way an average person does, or even a non-security developer.
- OJFord 1y agoPeople are doing this with MCP on large scale, it's not talked about enough; otherwise sensible people are dismissive of the risks.
- deleted 1y ago[deleted]
- latexr 1y ago> Are people really this oblivious There are hundreds of accidental gun deaths every year in the US. So yes, people do need to be told to not point guns at their feet (and other body parts) and pull the trigger. https://ammo.com/articles/accidental-shooting-statistics https://ammo.com/articles/accidental-shooting-statistics Just the other day I saw a video of a guy pulling a trigger on a gun in another person’s hand with the express goal of inuring them with the knock back. Another of a guy accidentally shooting a hole through his own ceiling. And a third one of a guy (there were at least three people present) shooting a rifle into a shallow river; they were lucky the barrel exploded like a looney tunes cartoon. And we’re talking guns, which are designed to be dangerous. Software is much more nebulous and people have been conditioned for years to just download whatever and install it.