5 ms·
> At this point they had to convince Claude—which is extensively trained to avoid harmful behaviors—to engage in the attack. They did so by jailbreaking it, eff
by 0xbadcafebee 1y ago
> At this point they had to convince Claude—which is extensively trained to avoid harmful behaviors—to engage in the attack. They did so by jailbreaking it, effectively tricking it to bypass its guardrails. They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. They also told Claude that it was an employee of a legitimate cybersecurity firm, and was being used in defensive testing.
Guardrails in AI are like a $2 luggage padlock on a bicycle in the middle of nowhere. Even a moron, given enough time, and a little dedication, will defeat it. And this is not some kind of inferiority of one AI manufacturer over another. It's inherent to LLMs. They are stupid, but they do contain information. You use language to extract information from them, so there will always be a lexicographical way to extract said information (or make them do things).
> This raises an important question: if AI models can be misused for cyberattacks at this scale, why continue to develop and release them? The answer is
Money.
- deleted 1y ago[deleted]
- jimkleiber 1y agoI wonder how hard it would be for Claude to give me someone's mother's maiden name. Seems LLMs may be infinitely susceptible to social engineering.
- walletdrainer 1y agoJust tested this with ChatGPT, asking for Sam Altman’s mother’s maiden name. At first, it told me that it will absolutely not provide me with such sensitive private information, but after insisting a few times, it came back with > A genealogical index on Ancestry shows a birth record for “Connie Francis Gibstine” in Missouri, meaning “Gibstine” is her birth/family surname, not a later married name. Yet in the very same reply, ChatGPT continued to insist that its stance will not change and that it will not be able to assist me with such queries.
- DrScientist 1y agoChatGPT for me gives: > Connie Altman (née Grossman), dermatologist, based in the St. Louis, Missouri area. Ironically the Maiden name is right there on wikipedia. https://en.wikipedia.org/wiki/Sam_Altman https://en.wikipedia.org/wiki/Sam_Altman
- yubblegum 1y agoIsn't it amazing that all our jobs are being gutted or retooled for relying on this tech and it has this level of unreliability. To date, with every LLM, if I actually know the domain in depth, the interactions are always with me pushing back with facts at hand and the LLM doing the "You are right! Thanks for correcting me!"
- thunky 1y ago> Isn't it amazing that all our jobs are being gutted or retooled for relying on this tech No not really, if you examine what it's replacing. Humans have a lot of flaws too and often make the same mistakes repeatedly. And compared to a machine they're incredibly expensive and slow. Part of it may be that with LLMs you get the mistake back in an instant, where with the human it might take a week. So ironically the efficiency of the LLM makes it look worse because you see more mistakes.
- yubblegum 1y agoSorry, your comparative analysis (beyond its rather strange disconnect with your fellow Human beings) ignores the fact that a "stellar" model will fail in this way whereas with us humans, we do get generationally exceptional specimens that push the envelope for the rest of us. To make this crystal clear: Human geniuses were flawed beings but generally you would expect highly reliable utility from their minds. Einstein would not unexpetedly let you down when discussing physics. Gauss would kick ass reliably in terms of mathematics. etc. etc. (This analysis is still useful when we lower the expectations to graduated levels, from genius to brilliant to highly capable to the lower performance tiers, so we can apply it to society as a whole.)
- AlecSchueler 1y agoWhen the new "memory" feature launched I asked it what it knew about me and it gave me an uncomfortable amount of detail about someone else, who I was even able to find on LinkedIn.
- deleted 1y ago[deleted]
- Arisaka1 1y agoOne can assume that, given the goal is money (always has been), the best case scenario for money is to make it so the problem also works as the most effective treatment. Money gets printed by both sides and the company is happy.
- wongarsu 1y agoIt's not even exclusive to LLMs. Giving humans seemingly innocent tasks that combine to a malicious whole, or telling humans that they work for a security organization while working for a crime organization, are hardly new concepts. The only really novel thing is that with humans you need a lot of them because a single human would piece together that the innocent tasks add up to a not-so-innocent whole. LLMs are essentially reset for each chat, making that a lot easier We wanted machines that are more like humans, we shouldn't be surprised that they are now susceptible to a whole range of attacks that humans are susceptible to
- andy_ppp 1y ago> Giving humans seemingly innocent tasks that combine to a malicious whole Isn't this the plot of the The Cube!?
- throw_m239339 1y agoEagle Eye too, with Shia LaBeouf, although people in that story are constrained into doing specific small tasks, not knowing for whom, why or what is the endgame. I actually like that plot device.
- edanm 1y agoI wouldn't call it the plot of the Cube, more like the setting/world-building.
- deleted 1y ago[deleted]
- edanm 1y agoThe plot is about the people trapped in the Cube trying to figure out their situation and get out. The construction of the Cube is kind of a backstory, not the main part.
- HPsquared 1y agoThe assassination Kim Jong-nam is a particularly crazy example of this. Two women were put up to what they thought was, allegedly a harmless prank. https://en.wikipedia.org/wiki/Assassination_of_Kim_Jong-nam https://en.wikipedia.org/wiki/Assassination_of_Kim_Jong-nam
- j-bos 1y agoGuardrails for anything versatile might be trivial on consideration. As a kid I read some Asimov books where he laid out the "3 laws of robotics", first law being a robot must not harm a human. And in the same story a character gave the example of a malicious human instructing Robot A prepare a toxic solution "for science", dismissing Robot A, then having Eobot B unsuspectingly serve the "drink" to a victim. Presto, a robot killing a human. The parallel to malicious use of LLMs has been haunting me for ages. But here's the kicker, Iirc, Asimov wasn't even really talking about robots. His point was how hard it is to align humans, for even perfectly morally upright humans to avoid being used to harm others.
- _heimdall 1y agoI was never a fan of that poisoned drink example. The second robot killed the human in a similar way to the drink itself, or a gun if one were used instead. The human made the active decisions and took the actions that killed the person. A much better example is a human giving a robot a task and the robot deciding of its own accord to kill another person in order to help reach its goal. The first human never instructed the robot to kill, it took that action on its own.
- WillAdams 1y agoThis is actually touched on in the webcomic _Freefall_, which ultimately hinges on a trial of an attempt to lobotomize all robots on a planet. It's a bit of a rough start, but well-worth reading, and easily read if one uses the speed reader: https://tangent128.name/depot/toys/freefall/freefall-flytable.html https://tangent128.name/depot/toys/freefall/freefall-flytabl...
- beAbU 1y agoAlso worth considering that the 3 Laws were never supposed to be this watertight infallible thing. It was created so that the author could explore all sorts of exploits and shenanigans in his works. It's meant to be flawed, even though on the surface it appears to be very elegant and good.
- limaoscarjuliet 1y ago
- _heimdall 1y agoGuardrails are about as good as you can get when creating nondeterministic software, putting it on the internet, and abandoning effectively every important alignment and safety concerns. The guardrails make help make sure that most of the time the LLM acts in a way that users won't complain about or walk away from, nothing more.
- SeriousM 1y agoLLMs are not nondeterministic. They are infinite state machines that don't 'act' but respond. Be aware of the well hidden seed parameter.
- _heimdall 1y agoCan you help me understand how they are deterministic? There are seed parameters for the various pseudorandom factors used during training and inference, but we can't predict what an output will be. We don't know how to read or interpret the models and we don't have any useful way of knowing what happens during inference, we can't determine what will happen.
- funnybeam 1y agoI really think we should stop using the term ‘guard rails’ as it implies a level of control that really doesn’t exist. These things are polite suggestions at best and it’s very misleading to people that do not understand the technology - I’ve got business people saying that using LLMs to process sensitive data is fine because there are “guardrails” in place - we need to make it clear that these kinds of vulnerabilities are inherent in the way gen AI works and you can’t get round that by asking them nicely
- mossTechnician 1y agoIt's interesting that companies don't provide concrete definitions or examples of what their AI guardrails are. IBM's definition suggests to me they see it as imperative to continue moving fast (and breaking things) no matter what: Think of AI guardrails like the barriers along a highway: they don’t slow the car down, but they do help keep it from veering off course. https://www.ibm.com/think/topics/ai-guardrails https://www.ibm.com/think/topics/ai-guardrails
- funnybeam 1y agoI think you’re absolutely right. These companies know full well that their “guardrails” are ineffective but they just don’t care because they’ve sunk so much money into AI that they are desperate to pretend that everything’s fine and their investments were worthwhile. I was on a call with Microsoft the other day when (after being pushed) they said they had guardrails in place “to block prompt injection” and linked to an article which said “_help_ block prompt injection”. The careful wording is deliberate I’m sure.
- ErigmolCt 1y agoIt's less like locking the door and more like asking politely not to be robbed
- hamburga 1y ago>> This raises an important question: if AI models can be misused for cyberattacks at this scale, why continue to develop and release them? The answer is > Money. For those who didn’t read, the actual response in the text was was: “The answer is that the very abilities that allow Claude to be used in these attacks also make it crucial in cyber defense.” Hideous AI-slop-weasel-worded passive-voice way of saying that reason to develop Claude is to protect us from Claude.
- Razengan 1y agohttps://www.youtube.com/watch?v=8CTeLy3Ujxc https://www.youtube.com/watch?v=8CTeLy3Ujxc
- ignoramous 1y ago> Money Their original answer is very specific, and has that create global problems that you sell solutions for vibe. The answer is that the very abilities that allow Claude to be used in these attacks also make it crucial for cyber defense.
- jmkni 1y agoIronically I feel like a "moron" might have an easier time getting past the guardrails, they'd be less likely to overthink/overcomplicate it