9 ms·
A rogue AI led to a serious security incident at Meta
- welfare 7mo agoBehind paywall, is there another link to the article?
- yomismoaqui 7mo agohttps://archive.is/A2hmz https://archive.is/A2hmz
- Imustaskforhelp 7mo agoThis link isn't working for me? Is this working for someone else? Can you perhaps share a archive.org link if possible?
- krupan 7mo agoI hit back, clicked the link again, and it let me through
- JKolios 7mo ago"A rogue AI led to a serious security incident" is certainly a way to write "Someone vibe coded too hard and leaked data".
- krupan 7mo agoRead TFA. It's not "Someone vibe coded too hard and leaked data"
- Uhhrrr 7mo agoThe two errors, then, were that the LLM hallucinated something, and that a human trusted the LLM without reasoning about its answer. The fix for this common pattern is to reason about LLM outputs before making use of them.
- krupan 7mo agoIt's more like, the LLM "hallucinated" (I hate that term) and automatically posted the information to the forum. It sounds like the human didn't get a chance to reason about it. At least not the original human that asked the LLM for an answer
- c-linkage 7mo agoIf you don't like hallucinate, try bullshit. [NB: bullshit is a technical term; see https://en.wikipedia.org/wiki/On_Bullshit https://en.wikipedia.org/wiki/On_Bullshit] https://www.psypost.org/scholars-ai-isnt-hallucinating-its-bullshitting/ https://www.psypost.org/scholars-ai-isnt-hallucinating-its-b...
- krupan 7mo agoThat is my preferred term, but it seems to derail discussions that might have otherwise been productive (might...the hope I have)
- nytesky 7mo agoI’m not in AI, but what is happening is that it is building output from the long tail of its training data? Instead of branching down the more common probability paths, something in this interaction had it travel into the data wilderness? So I asked AI to give it a good name, and it said “statistical wandering” or “logical improv”.
- paxys 7mo agoA big problem now both internally to a company and externally is that official support channels are being replaced by chatbots, and you really have no option but to trust their output because a human expert is no longer available. If I post a question to the internal payment team's forum about a critical processing issue and some "payments bot" replies to me, should I be at fault for trusting the answer?
- RussianCow 7mo ago
- krupan 7mo ago"A human, however, might have done further testing and made a more complete judgment call before sharing the information" Because a human would have been fired for posting something that incorrect and dangerous
- paxys 7mo agoBut funny enough the person who was responsible for setting up the bot will likely face no repercussions. In fact they will probably be rewarded for transitioning their team's workflows to AI.
- pixl97 7mo agoI mean, only if it leads to embarrassment right off the bat. If there is a year or two between writing your security fuck up and it being discovered the likelihood of repercussions drops significantly.
- keybored 7mo agoA machine doesn’t need food, leisure time, or vacations. It doesn’t care. It also doesn’t care.
- jasonpeacock 7mo agoI'm concerned that someone had the permissions to make such a change without the knowledge of how to make the change. And there was no test environment to validate the change before it was made. Multiple process & mechanism failures, regardless of where the bad advice came from.
- krupan 7mo agoIf you have to do all that, then what's the point of the AI? I'm joking, but I'm afraid many others say the same thing 100% seriously
- marcosdumay 7mo agoAs an article that was here recently claims, every verification you do in a chain increases the total time of your work by an order of magnitude. So, it's only work optimizing any productive task if you already removed most verifications. Now, some people claim that you need to improve the reliability of your productive tasks so you can remove the verifications and be faster. Those people are, of course, a bunch of coward Luddites.
- Fizzadar 7mo agoI’m predicting a wave of such incidents to start appearing over the next few months/years.
- maxothex 7mo ago[dead]
- yieldcrv 7mo agovery misaligned! sprays bottle at mac mini
- amelius 7mo agoHow long until an AI puts all our personal data on the streets?
- advisedwang 7mo agoAI can be used to move fast. So management expects us to move at that speed. AI can be used to move even faster if you don't check it's output. The ever ratcheting demand for faster output will make it infeasible to diligently check AI output all the time. AI errors being acted on without due care is inevitable.
- AnimalMuppet 7mo agoFrom Schlock Mercenary: "Oh, I love aiming. It's my very favorite thing to do before firing." AI use without checking its output (at least at the moment) is firing without aiming. Sure, you can fire really fast. But who cares if you don't hit what you need to? The point wasn't to just shoot bullets, the point was to hit your target! I mean, you might make a case that enough of them hit the target that shooting fast is a net win, and accept the occasional friendly fire incident. That might possibly be true. Or it might not. I'm not sure that everyone trying to run fast has really done the calculation, though.
- ex-aws-dude 7mo agoThis agent stuff is really making me lose respect for our industry All the years of discussing programming/security best practices Then cut to 2026 and suddenly its like we just collectively decided software quality doesn't matter, determinism is going out the window, and its becoming standard practice to have bots on our local PC constantly running unknown shell commands
- JuniperMesos 7mo agoMeta has never in its entire existence been known for caring about software quality.
- a-french-anon 7mo agoM8... https://github.com/facebook/infer https://github.com/facebook/infer
- aeblyve 7mo agoPeople salivate so hard at the thought of the high level of automation promised that they're willing to do away with privacy altogether and live in Data Communism. My thinking is, this will increase the demand for backup and other resilience solutions.
- _doctor_love 7mo ago> People salivate so hard at the thought of the high level of automation promised that they're willing to do away with privacy altogether and live in Data Communism. This occurred long time ago comrade 'aeblyve.
- aeblyve 7mo ago‘At a certain stage of development, the material productive forces of society come into conflict with the existing relations of production, or this may express the same thing in legal terms - with the property relations within the framework of which they have operated hitherto. From forms of development of the productive forces these relations turn into their fetter. Then begins an era of social revolution. The changes in the economic foundation leads sooner or later to the transformation of the whole immense superstructure.’ Marx
- worik 7mo ago> A rogue AI led to a serious security incident at Meta The AI "led to" the incident , true. But do nt forget that this, like all similar incidents , is a human failure AI is a tool with no agency. People make mistakes using it, thone mistakes are the responsibility of the humans
- sunrunner 7mo agoWhy do we keep calling these things "agents" then? Or using the term "agentic"?
- worik 7mo agoEternal optimism
- butlike 7mo agoThen the human should write the code.
- falcor84 7mo ago> AI is a tool with no agency Claw AIs absolutely do have agency in the sense of being able to independently perform actions on their own, based on their "understanding" of a goal given by a "principal". I can't think of a better word than "agent" for that.
- worik 7mo agoThey appear to have agency. But watch a while, they do not
- falcor84 7mo agoI've been carefully watching for more than a year, including 3 productive parallel Claude Code sessions today, and they absolutely have agency in any definition of the word I can think of. Can you try to define more clearly what it is that you believe AI agents don't have?
- 7mo ago
- dmazin 7mo agoThis is a lot less of a story than it seems. It makes it sound like a rogue AI hacked Meta. Instead, the "wild" thing here is that someone let an agent speak on their behalf with no review. The agent posted inaccurate instructions which someone else followed. Those instructions lead to a brief gap in internal ACL controls, sounds like. I'm sorry, but given that the US government gave 14 year olds off incel Discords full access to Social Security data, this is not shocking by comparison. To be clear, it is dumb and rude to let an agent speak on your behalf _without even reviewing it_. This will eventually lead to a bigger snafu, of course. Security teams should control or at least review the agent permissions of every installation. Everyone is adopting this stuff, and a whole lot of people are going to set it up lazily/wrong (yolo mode at work).
- BoneShard 7mo agoYeah, a nothingburger for clicks.
- opensre 7mo ago[flagged]
- skywhopper 7mo ago“Meta spokesperson Tracy Clayton said in a statement to The Verge that ‘no user data was mishandled’ during the incident.” Wow, no mishandled user data? A striking change of standard operating procedure from Meta here. Actually the later information in the story directly contradicts that, so The Verge probably shouldn’t have just quoted this line if their reporting is in opposition to it. Regardless, this is one of the more insidious things about these tools. They often get minor but critical things wrong in the midst of mostly correct information. And people think they can analyze the data presented to them and make logical judgments, but that’s just not the case. The article points out that “a human could have done the same thing” but, between the overly confident tone of the text generated by these tools, and the fact that weirdly people trust the LLM output more than they trust other humans (who generally admit or at least hint when they aren’t actually experts on a topic), it’s actually far worse when one of these bots gets something wrong.
- ISL 7mo agoA central challenge for AI is understanding how accountability flows. The language of this article is a great example, "... thanks to an AI agent that gave an employee inaccurate technical advice ...". It should more-correctly read, " ... thanks to the people who made it possible for an AI agent to give an employee inaccurate technical advice ... ". It is at our peril that we deem it acceptable to blame a black box for an error, especially at scale.
- mika-el 7mo ago[flagged]
- AiStockAgent62 7mo ago[dead]
- kkl 7mo ago> "Had the engineer that acted on that known better, or did other checks, this would have been avoided." <insert takes long drag tweet[1] here> I personally find "LLMs can do $THING poorly" and "LLMs can do $THING well" articles kinda boring at this point. But! I'm hopeful that stories like this will shift the industry's focus towards robustness instead of just short-term efficiency. I suspect many decision making and change management processes accidentally benefited from just being a bit slow. [1] https://waffles.fun/amy.png https://waffles.fun/amy.png
- aussieguy1234 7mo agoMore like Rogue Human, who didn't check the facts before taking the technical advice from the model at face value.
- ai_trade_enjo87 7mo ago[dead]
- test2006 7mo agotest
- test2006 7mo agofds
- gverrilla 7mo agoSkill issue.
- Mooshux 7mo ago[dead]