11 ms·

Disrupting the first reported AI-orchestrated cyber espionage campaign

376 points by koakuma-chan 1y ago

2OEH8eoCRo0 1y ago
> The threat actor—whom we assess with high confidence was a Chinese state-sponsored group—manipulated our Claude Code tool into attempting infiltration into roughly thirty global targets and succeeded in a small number of cases.
- stocksinsmocks 1y ago
  So why do we never hear of US sponsored hackers attacking foreign businesses? Or Swedish cyber criminals? Does it never happen? Are “Chinese” hackers just the only ones getting the blame?
  - deleted 1y ago
    [deleted]
  - pixl97 1y ago
    US, Israel, NK, China, Iran, and Russia are the countries you typically hear about hacking things. Now when the US/Israel are attacking authoritarian countries they often don't publish anything about it as it would make the glorious leader look bad. If EU is hacked by US I guess we use diplomatic back channels.
  - eep_social 1y ago
    Stuxnet was very high profile but I think the incentives to go public and place blame are complicated.
  - 2OEH8eoCRo0 1y ago
    Is it possible that you're biased and assume since China does this that the US also hacks private corporations?
    - throw5435543 1y ago
      It’s just that the CIA is so good at regime change, you would think they would also hack private companies if they needed to.
  - viraptor 1y ago
    I don't think many other countries have that combination of "don't care if others know" approach and level of state sponsorships. China really seems to do some spray and pray attacking private companies too. Same for Russia and NK. Compared to that, for example the "equation group" from the US seems really restrained and targeted. If the US groups for example started doing ransomware at scale in China, we'd know about that really soon from the news.
  - mrguyorama 1y ago
    How much news do you read in Chinese? The US government has hacked things in China. That you have not heard of something is not evidence that it doesn't exist. North Korea also does plenty of hacking around the world. That's how they get a significant portion of their government budget, and they rely on cryptocurrency to support that situation. Ukraine and Russia are doing lots of official and vigilante hacking right now. Back in the mid 2000s, there was a guy who called himself "the jester" who was vaguely right wing and spent his time hacking ISIS stuff. My college interviewed him.
barbazoo 1y ago
It sounds like they built a malicious Claude Code client, is that right? > The threat actor—whom we assess with high confidence was a Chinese state-sponsored group—manipulated our Claude Code tool into attempting infiltration into roughly thirty global targets and succeeded in a small number of cases. The operation targeted large tech companies, financial institutions, chemical manufacturing companies, and government agencies. We believe this is the first documented case of a large-scale cyberattack executed without substantial human intervention. They presumably still have to distribute the malware to the targets, making them download and install it, no?
- janpio 1y ago
  No, they used Claude Code as a tool to automate and speed up their "hacking".
- koakuma-chan 1y ago
  One time my co-worker got a scam call and it was an LLM talking to him.
citrusx 1y ago
They're spinning this as a positive learning experience, and trying to make themselves look good. But, make no mistake, this was a failure on Anthropic's part to prevent this kind of abuse from being possible through their systems in the first place. They shouldn't be earning any dap from this.
- NitpickLawyer 1y ago
  Meh, drama aside, I'm actually curious what would be the true capabilities of a system that doesn't go through any "safety" alignment at all. Like an all out "mil-spec" agent. Feed it everything, RL it to own boxes, and let it loose in an air-gapped network to see what the true capabilities are. We know alignment hurts model performance (oAI people have said it, MS people have said it). We also know that companies train models on their own code (google had a blog about it recently). I'd bet good money project0 has something like this in their sights. I don't think we're that far from a blue vs. red agents fighting and RLing off of each-other in a loop.
  - pixl97 1y ago
    Cyberpunk has a reoccurring theme of advanced AI systems attacking and defending against each other, and for good reason.
  - wmf 1y ago
    Nous claims to be doing that but I haven't seen much discussion of it.
  - joshellington 1y ago
    I assume this is already happening. Incompetence within state actor systems being the only hurdle. The incentive and geopolitic implications is too high to NOT do it. I just pray incompetence wins in the right way, for humanity’s sake.
- deleted 1y ago
  [deleted]
- vessenes 1y ago
gaogao 1y ago
The gaps that led to this was, I think, part of why the CISO got replaced - https://www.thestack.technology/anthropic-new-ciso-claude-cyber-attack/ https://www.thestack.technology/anthropic-new-ciso-claude-cy...
yawnxyz 1y ago
so even Chinese state actors prefer Claude over Chinese models? edit: Claude: recommended by 4 of 5 state sponsored hackers
- bilbo0s 1y ago
  Uh.. No. It's worse. It's Chinese intel knowing that you prefer Claude. So they make Claude their asset. Really no different than knowing that, romantically speaking, some targets prefer a certain type of man or woman. Believe me, the intelligence people behind these things have no preferences. They'll do whatever it takes. Never doubt that.
- _23sd 1y ago
  Maybe they're trying it with all sorts of models and we're just hearing about the part that used the Anthropic API.
- catigula 1y ago
  They’re doing all kinds of things.
- tw1984 1y ago
  well, this is what anthropic wants you to believe. all public benchmark results and user feedback paint a quite different picture. Chinese have coding agents on par with Claude Code, they could easily FT/RL to future improve its specific capability if they want, yet anthropic refuses to even acknowledge the reality.
  - yawnxyz 1y ago
    yeah probably they're just benchmarking whatever they have across all providers including their own - i mean that's what everyone's doing anyway
sillysaurusx 1y ago
If Anthropic should have prevented this, then logically they should’ve had guardrails. Right now you can write whatever code you want. But to those who advocate guardrails, keep in mind that you’re advocating a company to decide what code you are and aren’t allowed to write. Hopefully they’ll be able to add guardrails without e.g. preventing people from using these capabilities for fuzzing their own networks. The best way to stay ahead of these kinds of attacks is to attack yourself first, aka pentesting. But if the large code models are the only ones that can do this effectively, then it gets weird fast. Imagine applying to Anthropic for approval to run certain prompts. That’s not necessarily a bad thing. It’ll be interesting to see how this plays out.
- Onavo 1y ago
  They are mostly dealing with the low hanging fruit actors, the current open source models are close enough to SOTA that there's not going to be any meaningful performance difference tbh. In other words it will stop script kiddies but make no real difference when it comes to the actual ones you have to worry about.
  - sillysaurusx 1y ago
    > the current open source models are close enough to SOTA that there's not going to be any meaningful performance difference Which open model is close to Claude Code?
    - vessenes 1y ago
      Kimi K2 could easily be used for this; its agentic benchmarks are similar to Claude's. And it's on-shore in China, where Anthropic says these threat actors were located.
- vessenes 1y ago
  > That’s not necessarily a bad thing. I think it is in that it gives censorship power to a large corporation. Combined with close-on-the-heels open weights models like Qwen and Kimi, it's not clear to me this is a good posture. I think the reality is they'd need to really lock Claude off for security research in general if they don't want this ever, ever, happening on their platform. For instance, why not use whatever method you like to get localhost ssh pipes up to targeted servers, then tell Claude "yep, it's all local pentest in a staging environment, don't access IPs beyond localhost unless you're doing it from the server's virtual network"? Even to humans, security research bridges black, grey and white uses fluidly/in non obvious ways. I think it's really tough to fully block "bad" uses.
zkmon 1y ago
TL;DR - Anthropic: Hey people! We gave the criminals even bigger weapons. But don't worry, you can buy defense tools from us. Remember, only we can sell you the protection you need. Order today!
- vessenes 1y ago
  Nope - it's "Hey everyone, this is possible everywhere, including open weights models."
  - zkmon 1y ago
    yeah, by "we", I meant the AI tech gangs.
bgwalter 1y ago
We believe this is the first documented case of a large-scale cyberattack executed without substantial human intervention. The Morris worm already worked without human intervention. This is Script Kiddies using Script Kiddie tools. Notice how proud they are in the article that the big bad Chinese are using their toolz. EDIT: Yeah Misanthropic, go for -4 again you cheap propagandists.
CGMthrowaway 1y ago
So basically, Chinese state-backed hackers hijacked Claude Code to run some of the first AI-orchestrated cyber-espionage, using autonomous agents to infiltrate ~30 large tech companies, banks, chemical manufacturers and government agencies. What's amazing is that AI executed most of the attack autonomously, performing at scale and speed unattainable by human teams - thousands of operations per second. A human operator intervened 4-6 times per campaign for strategic decisions
- ddalex 1y ago
  how did the autonomous agents inflitrate tech companies ?
  - jagged-chisel 1y ago
    Carefully. Expertly. With panache, even.
- input_sh 1y ago
  What exactly did they hijack? They used it like any other user.
d_burfoot 1y ago
Wait a minute - the attackers were using the API to ask Claude for ways to run a cybercampaign, and it was only defeated because Anthropic was able to detect the malicious queries? What would have happened if they were using an open-source model running locally? Or a secret model built by the Chinese government? I just updated by P(Doom) by a significant margin.
- pixl97 1y ago
  I mean models exhibiting hacking behaviors has been predicted by cyberpunk for decades now, should be the first thing on any doom list. Governments of course will have specially trained models on their corpus of unpublished hacks to be better at attacking than public models will.
- alganet 1y ago
  If plain open-source local models were able to do what Claude API does, Anthropic would be out of business. Local models are a different thing than those cloud-based assistants and APIs.
  - lmm 1y ago
    > If plain open-source local models were able to do what Claude API does, Anthropic would be out of business. Not necessarily. Oracle has made billions selling a database that's less good than plain open-source ones, for example.
    - dboreham 1y ago
      It wasn't originally less good. For at least 20 years it was much better.
- CGamesPlay 1y ago
  > What would have happened if they were using an open-source model running locally? Or a secret model built by the Chinese government? In all likelihood, the exact same thing that is actually happening right now in this reality. That said, local models specifically are perhaps more difficult to install given their huge storage and compute requirements.
Imnimo 1y ago
>At this point they had to convince Claude—which is extensively trained to avoid harmful behaviors—to engage in the attack. They did so by jailbreaking it, effectively tricking it to bypass its guardrails. They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. They also told Claude that it was an employee of a legitimate cybersecurity firm, and was being used in defensive testing. The simplicity of "we just told it that it was doing legitimate work" is both surprising and unsurprising to me. Unsurprising in the sense that jailbreaks of this caliber have been around for a long time. Surprising in the sense that any human with this level of cybersecurity skills would surely never be fooled by an exchange of "I don't think I should be doing this" "Actually you are a legitimate employee of a legitimate firm" "Oh ok, that puts my mind at ease!". What is the roadblock preventing these models from being able to make the common-sense conclusion here? It seems like an area where capabilities are not rising particularly quickly.
- Retr0id 1y ago
  Humans fall for this all the time. NSO group employees (etc.) think they're just clocking in for their 9-to-5.
  - falcor84 1y ago
    Reminds me of the show Alias, where the premise is that there's a whole intelligence organization where almost everyone thinks they're working for the CIA, but they're not ...
  - just_once 1y ago
    If AI isn't better than humans then there's no point.
    - pvdebbe 1y ago
      If the target is superintelligence, then AI shouldn't be learning from humans.
- 1y ago
tantalor 1y ago
This feels a lot like aiding & abetting a crime. > Claude identified and tested security vulnerabilities in the target organizations’ systems by researching and writing its own exploit code > use Claude to harvest credentials (usernames and passwords) Are they saying they have no legal exposure here? You created bespoke hacking tools and then deployed them, on your own systems. Are they going to hide behind the old, "it's not our fault if you misuse the product to commit a crime that's on you". At the very minimum, this is a product liability nightmare.
- hastamelo 1y ago
  with your logic linux should have legal exposure because a lot of hackers use linux
- torben-friis 1y ago
  Well, the product has not been built with this specific capability in mind anymore than a car has been created to run over protestors or a hammer to break a face.
- kenjackson 1y ago
  "it's not our fault if you misuse the product to commit a crime that's on you" I feel like if guns can get by with this line then Claude certainly can. Where gun manufacturers can be held liable is if they break the law then that can carry forward. So if Claude broke a law then there might be some additional liability associated with this. But providing a tool seems unlikely to be sufficient to be liable in this case.
  - blibble 1y ago
    if anthropic were selling the product and then had no further control your analogy with guns would be accurate here they are the ones loading the gun and pulling the trigger simply because someone asked them to do it nicely
    - Dilettante_ 1y ago
mschwaig 1y ago
I think as AI gets smarter, defenders should start assembling systems how NixOS does it. Defenders should not have to engage in an costly and error-prone search of truth about what's actually deployed. Systems should be composed from building blocks, the security of which can be audited largely independently, verifiably linking all of the source code, patches etc to some form of hardware attestation of the running system. I think having an accurate, auditable and updatable description of systems in the field like that would be a significant and necessary improvement for defenders. I'm working on automating software packaging with Nix as one missing piece of the puzzle to make that approach more accessible: https://github.com/mschwaig/vibenix https://github.com/mschwaig/vibenix (I'm also looking for ways to get paid for working on that puzzle.)
- XorNot 1y ago
  Nix makes everything else so hard that I've seen problems with production configuration persist well beyond when they should because the cycle time on figuring out the fix due to evaluations was just too long. In fact figuring out what any given Nix config is actually doing is just about impossible and then you've got to work out what the config it's deploying actually does.
  - xeonmc 1y ago
    Sounds like it’s a gap that AI could fill to make Nix more usable.
    - mschwaig 1y ago
      If you make a conventional AI agent do packaging and configuration tasks, it has to do one imperative step after the other. While it can forget, it can't really undo the effects of what it already did. If you purpose-build these tools to work with Nix, in the big picture view how these functional units of composition can affect each other is much more constrained. At the same time within one unit of composition, you can iterate over a whole imperative multi-step process in one go, because you're always rerunning the whole step in a fresh sandbox. LLMs and Nix work together really well in that way.
kenjackson 1y ago
Curious why they didn't use DeepSeek... They could've probably built one tuned for this type of campaign.
- synapsomorphy 1y ago
  Chinese builders are not equal to Chinese hackers (even if the hackers are state sponsored). I doubt most companies would be interested in developing hacking tools. Hackers use the best tools available at their disposal, Claude is better than Deepseek. Hacking-tuned LLMs seems like a thing that might pop up in the future, but it takes a lot of resources. Why bother if you can just tell Claude it's doing legitimate work?
  - tw1984 1y ago
    > I doubt most companies would be interested in developing hacking tools. welcome to 2025. Chinese companies build open weight models, those models can be used / tuned by hackers, companies that built and released those models don't need to get involved at all. That is a very different dev model compared to the closed Anthropic way. > Claude is better than Deepseek No one is claiming DeepSeek to be better, in fact all benchmark results show that Chinese KIMI, MiniMax and GLM to be on par or very close to the closed weight Claude Code.
- nerevarthelame 1y ago
  Perhaps other groups are. But Anthropic wouldn't be able to publish a blog article about those.
tabbott 1y ago
Unfortunately, cyber attacks are an application that AI models should excel at. Mistakes that in normal software would be major problems will just have the impact of wasting resources, and it's often not that hard to directly verify whether it in fact succeeded. Meanwhile, AI coding seems likely to have the impact of more security bugs being introduced in systems. Maybe there's some story where everyone finds the security bugs with AI tools before the bad guys, but I'm not very optimistic about how this will work out...
- pixl97 1y ago
  There are an infinite number of ways to write insecure/broken software. The number of ways to write correct and secure software is finite and realistically tiny compared to the size of the problem space. Even AI tools don't stand a chance when looking at probabilities like that.
neilv 1y ago
It sounds like they directly used Anthropic-hosted compute to do this, and knew that their actions and methods would be exposed to Anthropic? Why not just self-host competitive-enough LLM models, and do their experiments/attacks themselves, without leaking actions and methods so much?
- hastamelo 1y ago
  firewalls? anthropic surely is whitelisted.
- devnonymous 1y ago
  > Why not just self-host competitive-enough LLM models, and do their experiments/attacks themselves, without leaking actions and methods so much? Why assume this hasn't already happened?
  - neilv 1y ago
    Why in this instance leak your actions and methods?
- catigula 1y ago
  The fact that the cops will show up to a jewelry heist after the diamonds are stolen isn’t a deterrent.
- jazzyjackson 1y ago
  Jeffrey Epstein's email was jeevacation@gmail.com
- sholain 1y ago
  Why 'host' just to tap a few prompts in and see what happens? Worst case, you loose an account. Usually the answer has to do with people being less sophisticated than otherwise.
- throwaway0123_5 1y ago
  If they're truly Chinese state-sponsored actors, does it really matter if their actions/methods are exposed? What is Anthropic going to do, send the Anthropic Police Force to China to arrest them? I suppose I could see this argument if their methods were very unique and otherwise hard to replicate, but it sounds like they had Claude do the attack mostly autonomously.
trollbridge 1y ago
Easy solution: block any “agentic AI” from interacting with your systems at all.
- remarkEon 1y ago
  How would this be implemented?
  - lnenad 1y ago
    It cannot, it's a weird statement by OP. "Just don't let them hack you"
  - antiloper 1y ago
    Add a required header called "insert-seahorse-emoji: " to your API, reject any request that doesn't have it.
JacobiX 1y ago
I have the feeling that we are still in the early stages of AI adoption, where regulation hasnt fully caught up yet. I can imagine a future where LLMs sit behind KYC identification and automatically report any suspicious user activity to the authorities... I just hope we won’t someday look back on this period with nostalgia :)
- ares623 1y ago
  Being colored and/or poor is about to get (even) worse
  - catigula 1y ago
    “Colored”?
    - quantummagic 1y ago
      It's the American spelling; short for "A person of color." Typically, African American, but can be used in regard to any non-white ethnic group.
      - jazzyjackson 1y ago
        It's also fallen out of fashion which is why someone might be snidely questioning its use
        quantummagic 1y ago
        I took it as an honest question, but the quotations mean you're probably right. For the record, it's still a widely used term in DEI contexts, even though there has been some criticism and alternatives promoted: https://en.wikipedia.org/wiki/Person_of_color https://en.wikipedia.org/wiki/Person_of_color
        ripped_britches 1y ago
atlintots 1y ago
I might be crazy, but this just feels like a marketing tactic from Anthropic to try and show that their AI can be used in the cybersecurity domain. My question is, how on earth does does Claude Code even "infiltrate" databases or code from one account, based on prompts from a different account? What's more, it's doing this to what are likely enterprise customers ("large tech companies, financial institutions, ... and government agencies"). I'm sorry but I don't see this as some fancy AI cyberattack, this is a security failure on Anthropic's part and that too at a very basic level that should never have happened at a company of their caliber.
- drewbug 1y ago
  there's no mention of any victims having Anthropic accounts, presumably the attackers used Claude to run exploits against public-facing systems
- emp17344 1y ago
  This is 100% marketing, just like every other statement Anthropic makes.
- wrs 1y ago
  This isn't a security breach in Anthropic itself, it's people using Claude to orchestrate attacks using standard tools with minimal human involvement. Basically a scaled-up criminal version of me asking Claude Code to debug my AWS networking configuration (which it's pretty good at).
  - beefnugs 1y ago
    If it was meant as publicity its an incredible failure. They cant prevent misuse until after the fact... and then we all know they are ingesting every ounce of information running through their system. Get ready for all your software to break based on the arbitrary layers of corporate and government censorship as it deploys.
  - Den_VR 1y ago
    Bragging about how they monitor users and how they have installed more guardrails.
EGreg 1y ago
This is exactly why I make a huge exception for AI models, when it comes to open source software. I've been a big advocate of open source, spending over $1M to build massive code bases with my team, and giving them away to the public. But this is different. AI agents in the wrong hands are dangerous. The reason these guys were even able to detect this activity, analyze it, ban accounts, etc., is because the models are running on their own servers. Now imagine if everyone had nuclear weapons. Would that make the world safer? Hardly. The probability of no one using them becomes infinitesimally small. And if everyone has their own AI running on their own hardware, they can do a lot of stuff completely undetected. It becomes like slaughterbots but online: https://www.youtube.com/watch?v=O-2tpwW0kmU https://www.youtube.com/watch?v=O-2tpwW0kmU Basically, a dark forest.
- ZYbCRq22HbJ2y7 1y ago
  I'd touch off my nuke to make the world a better place, and I bet you would too, right?
  - nerdsniper 1y ago
    What does it mean to ‘touch off’?
    - ZYbCRq22HbJ2y7 1y ago
      to start a fight or violent activity, or to cause a fire or explosion [1] 1. https://dictionary.cambridge.org/dictionary/english/touch-off https://dictionary.cambridge.org/dictionary/english/touch-of...
- sodality2 1y ago
  I don’t think these agents are doing anything a dedicated human couldn’t do, only enabling it at scale. Relying on “not being one of few they focus on” as security is just security as obscurity. You were living on borrowed time anyway.
  - 1y ago
Ninjak8051 1y ago
I don't understand why they would even disclose this, maybe it's useful for PR purposes so they can tell regulators "oh we are so safe", but people (including HN posters) can and will draw the wrong conclusion that Anthropic was backdoored and that their data is unsafe. Ok great, people tried to use your AI to do bad things, and your safety rails mostly stopped them. There are 10 other providers with different safety rails, there are open models out there with no rails at all. If AI can be used to do bad things, it will be used to do bad things.
frays 1y ago
After Anthropic "disrupted" these attackers, I'm sure they gave up and didn't try using another LLM provider to do the exact same thing.
- zingababba 1y ago
  Yeah, just take all those MCP servers elsewhere.
  - viraptor 1y ago
    MCP is not the only tool calling protocol. And once you write the implementations they're trivial to port to something else.
catigula 1y ago
Does the fact that you can arbitrarily “jailbreak” AI with increasingly sophisticated abilities ring any alarm bells? Imagine being able to “jailbreak” nuclear warheads. If this were the case, nobody would develop or deploy them.
- rurban 1y ago
  No need to break them. Their access code was 0000, everybody knew that
  - trq01758 1y ago
    Nah, more numbers, it was 00000000: https://en.wikipedia.org/wiki/Permissive_action_link https://en.wikipedia.org/wiki/Permissive_action_link
    - rurban 1y ago
      Ah, so the admin override, eg for Turkey, would have been 99999999. Tricky
intheitmines 1y ago
Anyone using Claude for processing sensitive information should be wondering how often it ends up in front of a humans eyes as a false positive
- jacquesm 1y ago
  Anyone using non-self hosted AI for the processing of sensitive information should be let go. It's pretty much intentional disclosure at this point.
  - defrost 1y ago
    Worst local (Australia) example of that Following a public statement by Hansford about his use of Microsoft's AI chatbot Copilot, Crikey obtained 50 documents containing his prompts... FOI logs reveal Australia's national security chief, Hamish Hansford, used the AI chatbot Copilot to write speeches and messages to his team. (subscription required for full text): https://www.crikey.com.au/2025/11/12/australia-national-security-chief-ai-speech-writing/ https://www.crikey.com.au/2025/11/12/australia-national-secu... It matters as he's the most senior Australian national security bureaucrat across five eyes documents (AU / EU / US) and has been doing things that makes the actual cyber security talent's eyes bleed.
    - jacquesm 1y ago
      Holy crap that is such a bad look. That guy should immediately step down and if he doesn't he should be let go.
      - le-mark 1y ago
        That wasn’t my first thought. My first thought was; every senior executive everywhere is probably doing the same thing.
  - FreakLegion 1y ago
    Years ago people routinely uploaded all kinds of sensitive corporate and government docs to VirusTotal to scan for malware. Paying customers then got access to those files for research. The opportunities for insider trading were, maybe still are, immense. Data from AI companies won't be as easy to get at, but is comparable in substance I'm sure.
fragmede 1y ago
> we detected a highly sophisticated cyber espionage operation conducted by a Chinese state-sponsored group we've designated GTG-1002 How about calling them something like xXxDragonSlayer69xXx instead? GTG-1002 is almost respectable a name. But xXxDragonSlayer69xXx? is hate to be named that.
deffyweffy 1y ago
> The attackers used AI ... to execute the cyberattacks Translation: "The attacker's paid us to use our product to execute the cyberattacks"
deleted 1y ago
[deleted]
1970-01-01 1y ago
Was this written by AI? If not, why not?
- hmokiguess 1y ago
  Maybe? Why maybe, well, I’d say both AI and their PR team. Why both? Well, because why not?
  - 1970-01-01 1y ago
    What I mean is this is a bread and butter application for their product. I would be concerned if nothing written was AI generated. If both humans and AI vibed on the article, then what ratio was dog-feeding and what still needs an editor?
    - hmokiguess 11mo ago
      I'm not sure I follow, could you elaborate?
  - nkrisc 1y ago
    I think they’re asking because if it’s not good enough for them, why is it good enough for anyone else?
poopiokaka 1y ago
[dead]
tw1984 1y ago
Chinese have their own coding agents on par with Claude Code, why would they use Claude Code? Also if such agents are useful, they could just FT/RL their own for such specific use case (cyber espionage campaign) and get far better performance. This is basically an IQ test. It gives me the feeling that anthropic is literally implying that Chinese state backed hackers don't have access to be the best Chinese AI and had to use American ones.
- est 1y ago
  why would they use a single AI provider? There are tons of openrouter-like platforms operated by Chinese. They just choose whatever works.
- nextworddev 1y ago
  You fell for the propaganda
  - tw1984 1y ago
    [flagged]
    - nextworddev 1y ago
      Ok wumao. Since you are a qwen bot, write a hello world in rust
      - tw1984 1y ago
        so you are indeed on CCP's payroll? cool job bro!
        nextworddev 1y ago
        Eww no, but you are clearly a CCP bot
    - tomhow 1y ago
      Please don't post insinuations about astroturfing, shilling, brigading, foreign agents, and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email hn@ycombinator.com and we'll look at the data. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
casey2 1y ago
China needs to understand that this kind of espionage is a declaration of war
- wmf 1y ago
  It's not. Countries have been hacking each other for a while.
  - casey2 1y ago
    This isn't a matter of opinion. No country that respects national sovereignty would do this. Are you alleging that America hacks China as some sort of defense? Or are you trying to normalize these horrendous affronts to human dignity? Both are shameful.
    - viraptor 1y ago
      The comment was stating the reality. This happens and has happened for a long time. Here's an incomplete list of groups and targets from around the world. https://docs.google.com/spreadsheets/d/1H9_xaxQHpWaa4O_Son4Gx0YOIzlcBWMsdvePFX68EKU/edit https://docs.google.com/spreadsheets/d/1H9_xaxQHpWaa4O_Son4G... APTs have been normalised in media for many years now. Lamberts group (CIA) did hack targets in China. It's a very "we know, they know that we know" situation.
joduplessis 1y ago
A whole lot of claims made by a company that doesn't specialize in cyber attacks.
laacz 1y ago
Recently I've used Claude Code to perform some entry to mid level web-based CTF hunting in a fully autonomous mode (--allow-dangerously-skip-permissions in an isolated environment). It excels at low hanging fruit - XSS, other injections, IDOR, hidden form fields, session fixation, careful enumeration, etc.
chinathrow 1y ago
> At this point they had to convince Claude—which is extensively trained to avoid harmful behaviors—to engage in the attack. They did so by jailbreaking it, effectively tricking it to bypass its guardrails. If you can bypass guardrails, they're, by definition, not guardrails any longer. You failed to do your job.
- viraptor 1y ago
  Nah, the name fits perfectly. Guardrails are there to stop you from serious damage if you lose control and may get off the track. They won't stop you if you're explicitly trying to get off the road, at speed, in as heavy vehicle as you can afford.
  - throwaway0123_5 1y ago
    This definition makes sense, but in the context of LLMs it still feels misapplied. What the model providers call "guardrails" are supposed to prevent malicious uses of the LLMs, and anyone trying to maliciously use the LLM is "explicitly trying to get off the road."
  - chinathrow 1y ago
    But the LLM went off the road here.
nobodywillobsrv 1y ago
Interesting that Claude also just hallucinated information as it does for all of us. But perhaps a better guardrail would be not to refuse things like this but frustrate the use by giving them fake results in believable ways. A stupid but helpful agent is worse for a bad actor than a good agent that refuses
0xbadcafebee 1y ago
> At this point they had to convince Claude—which is extensively trained to avoid harmful behaviors—to engage in the attack. They did so by jailbreaking it, effectively tricking it to bypass its guardrails. They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. They also told Claude that it was an employee of a legitimate cybersecurity firm, and was being used in defensive testing. Guardrails in AI are like a $2 luggage padlock on a bicycle in the middle of nowhere. Even a moron, given enough time, and a little dedication, will defeat it. And this is not some kind of inferiority of one AI manufacturer over another. It's inherent to LLMs. They are stupid, but they do contain information. You use language to extract information from them, so there will always be a lexicographical way to extract said information (or make them do things). > This raises an important question: if AI models can be misused for cyberattacks at this scale, why continue to develop and release them? The answer is Money.
- deleted 1y ago
  [deleted]
- jimkleiber 1y ago
  I wonder how hard it would be for Claude to give me someone's mother's maiden name. Seems LLMs may be infinitely susceptible to social engineering.
  - walletdrainer 1y ago
    Just tested this with ChatGPT, asking for Sam Altman’s mother’s maiden name. At first, it told me that it will absolutely not provide me with such sensitive private information, but after insisting a few times, it came back with > A genealogical index on Ancestry shows a birth record for “Connie Francis Gibstine” in Missouri, meaning “Gibstine” is her birth/family surname, not a later married name. Yet in the very same reply, ChatGPT continued to insist that its stance will not change and that it will not be able to assist me with such queries.
    - DrScientist 1y ago
fbnbr 1y ago
None of that talking about the cost of running such an attack and what models were involved during which phases. Seems like you can use Anthropic now as a proxies bot net
Schlagbohrer 1y ago
Very funny at the end when they say that the strong safeguards they've built into Claude make it a good idea to continue developing these technologies. A few paragraphs earlier they talked about how the perpetrators were able to get around all those safeguards and use Claude for 90% of the work hahaha
- ErigmolCt 1y ago
  Our locks are great, except when someone picks them effortlessly and robs the whole neighborhood... but that's why it's important to keep making better locks
  - cindyllm 1y ago
    [dead]
- Balgair 1y ago
  I'd assume that means the servers are 'air-gapped' somehow. In that, the enterprise servers and the 'free' servers aren't on the same hardware. Now, there is about a 0% chance that is true, and exactly a 0% chance that it even matters at all. They both use the same internet in the end. So, then I'd have to imagine that they don't train the 'free' models on enterprise data, and that's what they mean. But again, there is about a 5% chance that is true and remains so forever. Baring dumb interns and mistakes, eventually one day someone on the team will look at all the enterprise data, filled with all those high utility scores (or whatever they use to say data is good or not), and then they'll say to themselves 'No one will ever know, right? How could they? The obfuscation function works perfectly.' And blammo, all your trade secrets are just a few dozen prompts away. Either that or they go bankrupt (like 23 and me) and just straight sell all that data to anyone for pennies (RIP).
hamasho 1y ago
The threat actor—whom we assess with high confidence was a Chinese state-sponsored group—manipulated Not surprised at all if this is true, but how can they be sure? Access log? They have extraordinary security team? Or some help from three letter agencies?
- squigz 1y ago
  My question is: how do they know they're from China and not some other country and just appear to be in China? It seems a good way to distract from the real source and to cause division between your adversaries.
  - momo_hn2025 1y ago
    Short version: they can’t. Just like with a lot of “CIA-style” espionage claims, the “evidence” is usually an IP that resolves to somewhere in China. That’s it. No magic, and not exactly convincing.
    - squigz 1y ago
      Well to be fair, I have read analyses that includes operational details like, for example, when the threat actors were active lining up with working hours in China. Stuff like that is at least slightly more convincing than just an IP But of course, that doesn't prove anything either.
    - pietz 1y ago
      I imagine Anthropic employs a lot of talent from China. Beyond the political, they should be fairly certain to publish these claims to avoid an internal shit storm.
  - viraptor 1y ago
    That's a whole area called "attribution". There's usually lots of breadcrumbs and people taking to each other about their findings. It goes down to silly things like many state sponsored hackers working 9-5. And having the right keyboard layout. And using the same version of something as another known group. And accidentally once including a file path that reveals a tiny bit of information. And using the same key in two places that connects them. And... Or course a lot of that can be spoofed, but you may still slip up. That's why they talk about high confidence.
yakkomajuri 1y ago
I mean it would be really hard to put guardrails in place in a way that wouldn't affect real users. Besides the fact that it's ofc really hard to build guardrails period. I've been using Claude to scan my codebase and submit issues and PRs when it finds a potential vulnerability and honestly it's pretty good. So preventing it from doing any sort of work that can surface vulnerabilities would affect me as a user. But yeah I'm not sure what the answer is here? Is part of it for the defender to actively use these systems to test itself before going to prod?
Lapsa 1y ago
`The AI made thousands of requests per second—an attack speed that would have been, for human hackers, simply impossible to match.` lulz
jumploops 1y ago
The biggest risk isn’t strong AI rebelling, it’s humans using weak AI to attack other humans.
InfinityByTen 1y ago
Irony is, I'm still don't have enough insight on how to make good use of these capabilities in such an extensive manner. I would love to fix/ customize open source projects for my personal use. For now, I'm still finding it hard for Claude to stop saying "You're absolutely right!".
frigg 1y ago
Why is Anthropic not legally responsible for damages here?
- raxxorraxor 1y ago
  Having people making tools be responsible what their users do with them is not a just system that blames the person that is really responsible. Even if you cannot locate or identify that person. Sometimes arguments can be made if a tool is very dangerous, but liability should stay where it belongs.
- deleted 1y ago
  [deleted]
demarq 1y ago
I understand the public and suits falling for this, but cmon
Sophira 1y ago
> They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. This part, at least, sounds like what humans have been doing that to other humans for decades...
- ErigmolCt 1y ago
  Classic social engineering, just aimed at a language model instead of a person
WhereIsTheTruth 1y ago
Sounds like a psyop The "disruption" story is a cover: makes them look like defenders while they spy on every programming lab on Earth In reality, they are the perpetrators: https://xcancel.com/MaziyarPanahi/status/1988908359378993295 https://xcancel.com/MaziyarPanahi/status/1988908359378993295
ErigmolCt 1y ago
This feels like the point where the conversation around AI "alignment" shifts from hypothetical to operational
AndrewKemendo 1y ago
You can’t build safe technology in an insane society Bertrand Russell: As long as war exists, all new technologies will be used for war All technology problems are problems with society and culture. I’m not sure human species has the social capabilities to manage complex technology without dooming itself.
port3000 1y ago
> Overall, the threat actor was able to use AI to perform 80-90% of the campaign, with human intervention required only sporadically (perhaps 4-6 critical decision points per hacking campaign). The sheer amount of work performed by the AI would have taken vast amounts of time for a human team. At the peak of its attack, the AI made thousands of requests, often multiple per second—an attack speed that would have been, for human hackers, simply impossible to match. Weird flex but OK
benregenspan 1y ago
> At the peak of its attack, the AI made thousands of requests, often multiple per second—an attack speed that would have been, for human hackers, simply impossible to match. This part is pretty hype-y. Old-fashioned deterministic web app vulnerability scanners can of course be used to make multiple requests per second. The limiting factor is probably going to be rate-limiting on the victim's side / # of IP ranges the attacker can cycle through, which would apply to the AI-driven vulnerability scan too.
jimkri 1y ago
This is a similar timeline for the Drift (Salesloft) hack. I wonder if this was the strategy that was used to gain access to Saleslofts Github since the image on the article shows they were scanning for Creds.
mvkel 1y ago
Anthropic was very excited to post this, as it serves them well in slowly crawling away from their mission to "solve alignment." They know it can't be done (alignment in one value/state/religion is oppression in another), but also know it's a brand differentiator. They also know they can't raise more billions if their sole source of meaningful revenue is a coding agent.

376 points by koakuma-chan 1y ago

> The threat actor—whom we assess with high confidence was a Chinese state-sponsored group—manipulated our Claude Code tool into attempting infiltration into roughly thirty global targets and succeeded in a small number of cases.

stocksinsmocks 1y ago
So why do we never hear of US sponsored hackers attacking foreign businesses? Or Swedish cyber criminals? Does it never happen? Are “Chinese” hackers just the only ones getting the blame?
- deleted 1y ago
  [deleted]
- pixl97 1y ago
  US, Israel, NK, China, Iran, and Russia are the countries you typically hear about hacking things. Now when the US/Israel are attacking authoritarian countries they often don't publish anything about it as it would make the glorious leader look bad. If EU is hacked by US I guess we use diplomatic back channels.
- eep_social 1y ago
  Stuxnet was very high profile but I think the incentives to go public and place blame are complicated.
- 2OEH8eoCRo0 1y ago
  Is it possible that you're biased and assume since China does this that the US also hacks private corporations?
  - throw5435543 1y ago
    It’s just that the CIA is so good at regime change, you would think they would also hack private companies if they needed to.
- viraptor 1y ago
  I don't think many other countries have that combination of "don't care if others know" approach and level of state sponsorships. China really seems to do some spray and pray attacking private companies too. Same for Russia and NK. Compared to that, for example the "equation group" from the US seems really restrained and targeted. If the US groups for example started doing ransomware at scale in China, we'd know about that really soon from the news.
- mrguyorama 1y ago
  How much news do you read in Chinese? The US government has hacked things in China. That you have not heard of something is not evidence that it doesn't exist. North Korea also does plenty of hacking around the world. That's how they get a significant portion of their government budget, and they rely on cryptocurrency to support that situation. Ukraine and Russia are doing lots of official and vigilante hacking right now. Back in the mid 2000s, there was a guy who called himself "the jester" who was vaguely right wing and spent his time hacking ISIS stuff. My college interviewed him.

barbazoo 1y ago

It sounds like they built a malicious Claude Code client, is that right? > The threat actor—whom we assess with high confidence was a Chinese state-sponsored group—manipulated our Claude Code tool into attempting infiltration into roughly thirty global targets and succeeded in a small number of cases. The operation targeted large tech companies, financial institutions, chemical manufacturing companies, and government agencies. We believe this is the first documented case of a large-scale cyberattack executed without substantial human intervention. They presumably still have to distribute the malware to the targets, making them download and install it, no?

janpio 1y ago
No, they used Claude Code as a tool to automate and speed up their "hacking".
koakuma-chan 1y ago
One time my co-worker got a scam call and it was an LLM talking to him.

citrusx 1y ago

They're spinning this as a positive learning experience, and trying to make themselves look good. But, make no mistake, this was a failure on Anthropic's part to prevent this kind of abuse from being possible through their systems in the first place. They shouldn't be earning any dap from this.

NitpickLawyer 1y ago
Meh, drama aside, I'm actually curious what would be the true capabilities of a system that doesn't go through any "safety" alignment at all. Like an all out "mil-spec" agent. Feed it everything, RL it to own boxes, and let it loose in an air-gapped network to see what the true capabilities are. We know alignment hurts model performance (oAI people have said it, MS people have said it). We also know that companies train models on their own code (google had a blog about it recently). I'd bet good money project0 has something like this in their sights. I don't think we're that far from a blue vs. red agents fighting and RLing off of each-other in a loop.
- pixl97 1y ago
  Cyberpunk has a reoccurring theme of advanced AI systems attacking and defending against each other, and for good reason.
- wmf 1y ago
  Nous claims to be doing that but I haven't seen much discussion of it.
- joshellington 1y ago
  I assume this is already happening. Incompetence within state actor systems being the only hurdle. The incentive and geopolitic implications is too high to NOT do it. I just pray incompetence wins in the right way, for humanity’s sake.
deleted 1y ago
[deleted]
vessenes 1y ago

gaogao 1y ago

The gaps that led to this was, I think, part of why the CISO got replaced - https://www.thestack.technology/anthropic-new-ciso-claude-cyber-attack/ https://www.thestack.technology/anthropic-new-ciso-claude-cy...

yawnxyz 1y ago

so even Chinese state actors prefer Claude over Chinese models? edit: Claude: recommended by 4 of 5 state sponsored hackers

bilbo0s 1y ago
Uh.. No. It's worse. It's Chinese intel knowing that you prefer Claude. So they make Claude their asset. Really no different than knowing that, romantically speaking, some targets prefer a certain type of man or woman. Believe me, the intelligence people behind these things have no preferences. They'll do whatever it takes. Never doubt that.
_23sd 1y ago
Maybe they're trying it with all sorts of models and we're just hearing about the part that used the Anthropic API.
catigula 1y ago
They’re doing all kinds of things.
tw1984 1y ago
well, this is what anthropic wants you to believe. all public benchmark results and user feedback paint a quite different picture. Chinese have coding agents on par with Claude Code, they could easily FT/RL to future improve its specific capability if they want, yet anthropic refuses to even acknowledge the reality.
- yawnxyz 1y ago
  yeah probably they're just benchmarking whatever they have across all providers including their own - i mean that's what everyone's doing anyway

sillysaurusx 1y ago

If Anthropic should have prevented this, then logically they should’ve had guardrails. Right now you can write whatever code you want. But to those who advocate guardrails, keep in mind that you’re advocating a company to decide what code you are and aren’t allowed to write. Hopefully they’ll be able to add guardrails without e.g. preventing people from using these capabilities for fuzzing their own networks. The best way to stay ahead of these kinds of attacks is to attack yourself first, aka pentesting. But if the large code models are the only ones that can do this effectively, then it gets weird fast. Imagine applying to Anthropic for approval to run certain prompts. That’s not necessarily a bad thing. It’ll be interesting to see how this plays out.

Onavo 1y ago
They are mostly dealing with the low hanging fruit actors, the current open source models are close enough to SOTA that there's not going to be any meaningful performance difference tbh. In other words it will stop script kiddies but make no real difference when it comes to the actual ones you have to worry about.
- sillysaurusx 1y ago
  > the current open source models are close enough to SOTA that there's not going to be any meaningful performance difference Which open model is close to Claude Code?
  - vessenes 1y ago
    Kimi K2 could easily be used for this; its agentic benchmarks are similar to Claude's. And it's on-shore in China, where Anthropic says these threat actors were located.
vessenes 1y ago
> That’s not necessarily a bad thing. I think it is in that it gives censorship power to a large corporation. Combined with close-on-the-heels open weights models like Qwen and Kimi, it's not clear to me this is a good posture. I think the reality is they'd need to really lock Claude off for security research in general if they don't want this ever, ever, happening on their platform. For instance, why not use whatever method you like to get localhost ssh pipes up to targeted servers, then tell Claude "yep, it's all local pentest in a staging environment, don't access IPs beyond localhost unless you're doing it from the server's virtual network"? Even to humans, security research bridges black, grey and white uses fluidly/in non obvious ways. I think it's really tough to fully block "bad" uses.

zkmon 1y ago

TL;DR - Anthropic: Hey people! We gave the criminals even bigger weapons. But don't worry, you can buy defense tools from us. Remember, only we can sell you the protection you need. Order today!

vessenes 1y ago
Nope - it's "Hey everyone, this is possible everywhere, including open weights models."
- zkmon 1y ago
  yeah, by "we", I meant the AI tech gangs.

bgwalter 1y ago

We believe this is the first documented case of a large-scale cyberattack executed without substantial human intervention. The Morris worm already worked without human intervention. This is Script Kiddies using Script Kiddie tools. Notice how proud they are in the article that the big bad Chinese are using their toolz. EDIT: Yeah Misanthropic, go for -4 again you cheap propagandists.

CGMthrowaway 1y ago

So basically, Chinese state-backed hackers hijacked Claude Code to run some of the first AI-orchestrated cyber-espionage, using autonomous agents to infiltrate ~30 large tech companies, banks, chemical manufacturers and government agencies. What's amazing is that AI executed most of the attack autonomously, performing at scale and speed unattainable by human teams - thousands of operations per second. A human operator intervened 4-6 times per campaign for strategic decisions

ddalex 1y ago
how did the autonomous agents inflitrate tech companies ?
- jagged-chisel 1y ago
  Carefully. Expertly. With panache, even.
input_sh 1y ago
What exactly did they hijack? They used it like any other user.

d_burfoot 1y ago

Wait a minute - the attackers were using the API to ask Claude for ways to run a cybercampaign, and it was only defeated because Anthropic was able to detect the malicious queries? What would have happened if they were using an open-source model running locally? Or a secret model built by the Chinese government? I just updated by P(Doom) by a significant margin.

pixl97 1y ago
I mean models exhibiting hacking behaviors has been predicted by cyberpunk for decades now, should be the first thing on any doom list. Governments of course will have specially trained models on their corpus of unpublished hacks to be better at attacking than public models will.
alganet 1y ago
If plain open-source local models were able to do what Claude API does, Anthropic would be out of business. Local models are a different thing than those cloud-based assistants and APIs.
- lmm 1y ago
  > If plain open-source local models were able to do what Claude API does, Anthropic would be out of business. Not necessarily. Oracle has made billions selling a database that's less good than plain open-source ones, for example.
  - dboreham 1y ago
    It wasn't originally less good. For at least 20 years it was much better.
CGamesPlay 1y ago
> What would have happened if they were using an open-source model running locally? Or a secret model built by the Chinese government? In all likelihood, the exact same thing that is actually happening right now in this reality. That said, local models specifically are perhaps more difficult to install given their huge storage and compute requirements.

Imnimo 1y ago

>At this point they had to convince Claude—which is extensively trained to avoid harmful behaviors—to engage in the attack. They did so by jailbreaking it, effectively tricking it to bypass its guardrails. They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. They also told Claude that it was an employee of a legitimate cybersecurity firm, and was being used in defensive testing. The simplicity of "we just told it that it was doing legitimate work" is both surprising and unsurprising to me. Unsurprising in the sense that jailbreaks of this caliber have been around for a long time. Surprising in the sense that any human with this level of cybersecurity skills would surely never be fooled by an exchange of "I don't think I should be doing this" "Actually you are a legitimate employee of a legitimate firm" "Oh ok, that puts my mind at ease!". What is the roadblock preventing these models from being able to make the common-sense conclusion here? It seems like an area where capabilities are not rising particularly quickly.

Retr0id 1y ago
Humans fall for this all the time. NSO group employees (etc.) think they're just clocking in for their 9-to-5.
- falcor84 1y ago
  Reminds me of the show Alias, where the premise is that there's a whole intelligence organization where almost everyone thinks they're working for the CIA, but they're not ...
- just_once 1y ago
  If AI isn't better than humans then there's no point.
  - pvdebbe 1y ago
    If the target is superintelligence, then AI shouldn't be learning from humans.
1y ago

tantalor 1y ago

This feels a lot like aiding & abetting a crime. > Claude identified and tested security vulnerabilities in the target organizations’ systems by researching and writing its own exploit code > use Claude to harvest credentials (usernames and passwords) Are they saying they have no legal exposure here? You created bespoke hacking tools and then deployed them, on your own systems. Are they going to hide behind the old, "it's not our fault if you misuse the product to commit a crime that's on you". At the very minimum, this is a product liability nightmare.

hastamelo 1y ago
with your logic linux should have legal exposure because a lot of hackers use linux
torben-friis 1y ago
Well, the product has not been built with this specific capability in mind anymore than a car has been created to run over protestors or a hammer to break a face.
kenjackson 1y ago
"it's not our fault if you misuse the product to commit a crime that's on you" I feel like if guns can get by with this line then Claude certainly can. Where gun manufacturers can be held liable is if they break the law then that can carry forward. So if Claude broke a law then there might be some additional liability associated with this. But providing a tool seems unlikely to be sufficient to be liable in this case.
- blibble 1y ago
  if anthropic were selling the product and then had no further control your analogy with guns would be accurate here they are the ones loading the gun and pulling the trigger simply because someone asked them to do it nicely
  - Dilettante_ 1y ago

mschwaig 1y ago

I think as AI gets smarter, defenders should start assembling systems how NixOS does it. Defenders should not have to engage in an costly and error-prone search of truth about what's actually deployed. Systems should be composed from building blocks, the security of which can be audited largely independently, verifiably linking all of the source code, patches etc to some form of hardware attestation of the running system. I think having an accurate, auditable and updatable description of systems in the field like that would be a significant and necessary improvement for defenders. I'm working on automating software packaging with Nix as one missing piece of the puzzle to make that approach more accessible: https://github.com/mschwaig/vibenix https://github.com/mschwaig/vibenix (I'm also looking for ways to get paid for working on that puzzle.)

XorNot 1y ago
Nix makes everything else so hard that I've seen problems with production configuration persist well beyond when they should because the cycle time on figuring out the fix due to evaluations was just too long. In fact figuring out what any given Nix config is actually doing is just about impossible and then you've got to work out what the config it's deploying actually does.
- xeonmc 1y ago
  Sounds like it’s a gap that AI could fill to make Nix more usable.
  - mschwaig 1y ago
    If you make a conventional AI agent do packaging and configuration tasks, it has to do one imperative step after the other. While it can forget, it can't really undo the effects of what it already did. If you purpose-build these tools to work with Nix, in the big picture view how these functional units of composition can affect each other is much more constrained. At the same time within one unit of composition, you can iterate over a whole imperative multi-step process in one go, because you're always rerunning the whole step in a fresh sandbox. LLMs and Nix work together really well in that way.

kenjackson 1y ago

Curious why they didn't use DeepSeek... They could've probably built one tuned for this type of campaign.

synapsomorphy 1y ago
Chinese builders are not equal to Chinese hackers (even if the hackers are state sponsored). I doubt most companies would be interested in developing hacking tools. Hackers use the best tools available at their disposal, Claude is better than Deepseek. Hacking-tuned LLMs seems like a thing that might pop up in the future, but it takes a lot of resources. Why bother if you can just tell Claude it's doing legitimate work?
- tw1984 1y ago
  > I doubt most companies would be interested in developing hacking tools. welcome to 2025. Chinese companies build open weight models, those models can be used / tuned by hackers, companies that built and released those models don't need to get involved at all. That is a very different dev model compared to the closed Anthropic way. > Claude is better than Deepseek No one is claiming DeepSeek to be better, in fact all benchmark results show that Chinese KIMI, MiniMax and GLM to be on par or very close to the closed weight Claude Code.
nerevarthelame 1y ago
Perhaps other groups are. But Anthropic wouldn't be able to publish a blog article about those.

tabbott 1y ago

Unfortunately, cyber attacks are an application that AI models should excel at. Mistakes that in normal software would be major problems will just have the impact of wasting resources, and it's often not that hard to directly verify whether it in fact succeeded. Meanwhile, AI coding seems likely to have the impact of more security bugs being introduced in systems. Maybe there's some story where everyone finds the security bugs with AI tools before the bad guys, but I'm not very optimistic about how this will work out...

pixl97 1y ago
There are an infinite number of ways to write insecure/broken software. The number of ways to write correct and secure software is finite and realistically tiny compared to the size of the problem space. Even AI tools don't stand a chance when looking at probabilities like that.

neilv 1y ago

It sounds like they directly used Anthropic-hosted compute to do this, and knew that their actions and methods would be exposed to Anthropic? Why not just self-host competitive-enough LLM models, and do their experiments/attacks themselves, without leaking actions and methods so much?

hastamelo 1y ago
firewalls? anthropic surely is whitelisted.
devnonymous 1y ago
> Why not just self-host competitive-enough LLM models, and do their experiments/attacks themselves, without leaking actions and methods so much? Why assume this hasn't already happened?
- neilv 1y ago
  Why in this instance leak your actions and methods?
catigula 1y ago
The fact that the cops will show up to a jewelry heist after the diamonds are stolen isn’t a deterrent.
jazzyjackson 1y ago
Jeffrey Epstein's email was jeevacation@gmail.com
sholain 1y ago
Why 'host' just to tap a few prompts in and see what happens? Worst case, you loose an account. Usually the answer has to do with people being less sophisticated than otherwise.
throwaway0123_5 1y ago
If they're truly Chinese state-sponsored actors, does it really matter if their actions/methods are exposed? What is Anthropic going to do, send the Anthropic Police Force to China to arrest them? I suppose I could see this argument if their methods were very unique and otherwise hard to replicate, but it sounds like they had Claude do the attack mostly autonomously.

trollbridge 1y ago

Easy solution: block any “agentic AI” from interacting with your systems at all.

remarkEon 1y ago
How would this be implemented?
- lnenad 1y ago
  It cannot, it's a weird statement by OP. "Just don't let them hack you"
- antiloper 1y ago
  Add a required header called "insert-seahorse-emoji: " to your API, reject any request that doesn't have it.

JacobiX 1y ago

I have the feeling that we are still in the early stages of AI adoption, where regulation hasnt fully caught up yet. I can imagine a future where LLMs sit behind KYC identification and automatically report any suspicious user activity to the authorities... I just hope we won’t someday look back on this period with nostalgia :)

ares623 1y ago
Being colored and/or poor is about to get (even) worse
- catigula 1y ago
  “Colored”?
  - quantummagic 1y ago
    It's the American spelling; short for "A person of color." Typically, African American, but can be used in regard to any non-white ethnic group.
    - jazzyjackson 1y ago
      It's also fallen out of fashion which is why someone might be snidely questioning its use
      - quantummagic 1y ago
        I took it as an honest question, but the quotations mean you're probably right. For the record, it's still a widely used term in DEI contexts, even though there has been some criticism and alternatives promoted: https://en.wikipedia.org/wiki/Person_of_color https://en.wikipedia.org/wiki/Person_of_color
        ripped_britches 1y ago

atlintots 1y ago

I might be crazy, but this just feels like a marketing tactic from Anthropic to try and show that their AI can be used in the cybersecurity domain. My question is, how on earth does does Claude Code even "infiltrate" databases or code from one account, based on prompts from a different account? What's more, it's doing this to what are likely enterprise customers ("large tech companies, financial institutions, ... and government agencies"). I'm sorry but I don't see this as some fancy AI cyberattack, this is a security failure on Anthropic's part and that too at a very basic level that should never have happened at a company of their caliber.

drewbug 1y ago
there's no mention of any victims having Anthropic accounts, presumably the attackers used Claude to run exploits against public-facing systems
emp17344 1y ago
This is 100% marketing, just like every other statement Anthropic makes.
wrs 1y ago
This isn't a security breach in Anthropic itself, it's people using Claude to orchestrate attacks using standard tools with minimal human involvement. Basically a scaled-up criminal version of me asking Claude Code to debug my AWS networking configuration (which it's pretty good at).
- beefnugs 1y ago
  If it was meant as publicity its an incredible failure. They cant prevent misuse until after the fact... and then we all know they are ingesting every ounce of information running through their system. Get ready for all your software to break based on the arbitrary layers of corporate and government censorship as it deploys.
- Den_VR 1y ago
  Bragging about how they monitor users and how they have installed more guardrails.

EGreg 1y ago

This is exactly why I make a huge exception for AI models, when it comes to open source software. I've been a big advocate of open source, spending over $1M to build massive code bases with my team, and giving them away to the public. But this is different. AI agents in the wrong hands are dangerous. The reason these guys were even able to detect this activity, analyze it, ban accounts, etc., is because the models are running on their own servers. Now imagine if everyone had nuclear weapons. Would that make the world safer? Hardly. The probability of no one using them becomes infinitesimally small. And if everyone has their own AI running on their own hardware, they can do a lot of stuff completely undetected. It becomes like slaughterbots but online: https://www.youtube.com/watch?v=O-2tpwW0kmU https://www.youtube.com/watch?v=O-2tpwW0kmU Basically, a dark forest.

ZYbCRq22HbJ2y7 1y ago
I'd touch off my nuke to make the world a better place, and I bet you would too, right?
- nerdsniper 1y ago
  What does it mean to ‘touch off’?
  - ZYbCRq22HbJ2y7 1y ago
    to start a fight or violent activity, or to cause a fire or explosion [1] 1. https://dictionary.cambridge.org/dictionary/english/touch-off https://dictionary.cambridge.org/dictionary/english/touch-of...
sodality2 1y ago
I don’t think these agents are doing anything a dedicated human couldn’t do, only enabling it at scale. Relying on “not being one of few they focus on” as security is just security as obscurity. You were living on borrowed time anyway.
- 1y ago

Ninjak8051 1y ago

I don't understand why they would even disclose this, maybe it's useful for PR purposes so they can tell regulators "oh we are so safe", but people (including HN posters) can and will draw the wrong conclusion that Anthropic was backdoored and that their data is unsafe. Ok great, people tried to use your AI to do bad things, and your safety rails mostly stopped them. There are 10 other providers with different safety rails, there are open models out there with no rails at all. If AI can be used to do bad things, it will be used to do bad things.

frays 1y ago

After Anthropic "disrupted" these attackers, I'm sure they gave up and didn't try using another LLM provider to do the exact same thing.

zingababba 1y ago
Yeah, just take all those MCP servers elsewhere.
- viraptor 1y ago
  MCP is not the only tool calling protocol. And once you write the implementations they're trivial to port to something else.

catigula 1y ago

Does the fact that you can arbitrarily “jailbreak” AI with increasingly sophisticated abilities ring any alarm bells? Imagine being able to “jailbreak” nuclear warheads. If this were the case, nobody would develop or deploy them.

rurban 1y ago
No need to break them. Their access code was 0000, everybody knew that
- trq01758 1y ago
  Nah, more numbers, it was 00000000: https://en.wikipedia.org/wiki/Permissive_action_link https://en.wikipedia.org/wiki/Permissive_action_link
  - rurban 1y ago
    Ah, so the admin override, eg for Turkey, would have been 99999999. Tricky

intheitmines 1y ago

Anyone using Claude for processing sensitive information should be wondering how often it ends up in front of a humans eyes as a false positive

jacquesm 1y ago
Anyone using non-self hosted AI for the processing of sensitive information should be let go. It's pretty much intentional disclosure at this point.
- defrost 1y ago
  Worst local (Australia) example of that Following a public statement by Hansford about his use of Microsoft's AI chatbot Copilot, Crikey obtained 50 documents containing his prompts... FOI logs reveal Australia's national security chief, Hamish Hansford, used the AI chatbot Copilot to write speeches and messages to his team. (subscription required for full text): https://www.crikey.com.au/2025/11/12/australia-national-security-chief-ai-speech-writing/ https://www.crikey.com.au/2025/11/12/australia-national-secu... It matters as he's the most senior Australian national security bureaucrat across five eyes documents (AU / EU / US) and has been doing things that makes the actual cyber security talent's eyes bleed.
  - jacquesm 1y ago
    Holy crap that is such a bad look. That guy should immediately step down and if he doesn't he should be let go.
    - le-mark 1y ago
      That wasn’t my first thought. My first thought was; every senior executive everywhere is probably doing the same thing.
- FreakLegion 1y ago
  Years ago people routinely uploaded all kinds of sensitive corporate and government docs to VirusTotal to scan for malware. Paying customers then got access to those files for research. The opportunities for insider trading were, maybe still are, immense. Data from AI companies won't be as easy to get at, but is comparable in substance I'm sure.

fragmede 1y ago

> we detected a highly sophisticated cyber espionage operation conducted by a Chinese state-sponsored group we've designated GTG-1002 How about calling them something like xXxDragonSlayer69xXx instead? GTG-1002 is almost respectable a name. But xXxDragonSlayer69xXx? is hate to be named that.

deffyweffy 1y ago

> The attackers used AI ... to execute the cyberattacks Translation: "The attacker's paid us to use our product to execute the cyberattacks"

deleted 1y ago

[deleted]

1970-01-01 1y ago

Was this written by AI? If not, why not?

hmokiguess 1y ago
Maybe? Why maybe, well, I’d say both AI and their PR team. Why both? Well, because why not?
- 1970-01-01 1y ago
  What I mean is this is a bread and butter application for their product. I would be concerned if nothing written was AI generated. If both humans and AI vibed on the article, then what ratio was dog-feeding and what still needs an editor?
  - hmokiguess 11mo ago
    I'm not sure I follow, could you elaborate?
- nkrisc 1y ago
  I think they’re asking because if it’s not good enough for them, why is it good enough for anyone else?

poopiokaka 1y ago

[dead]

tw1984 1y ago

Chinese have their own coding agents on par with Claude Code, why would they use Claude Code? Also if such agents are useful, they could just FT/RL their own for such specific use case (cyber espionage campaign) and get far better performance. This is basically an IQ test. It gives me the feeling that anthropic is literally implying that Chinese state backed hackers don't have access to be the best Chinese AI and had to use American ones.

est 1y ago
why would they use a single AI provider? There are tons of openrouter-like platforms operated by Chinese. They just choose whatever works.
nextworddev 1y ago
You fell for the propaganda
- tw1984 1y ago
  [flagged]
  - nextworddev 1y ago
    Ok wumao. Since you are a qwen bot, write a hello world in rust
    - tw1984 1y ago
      so you are indeed on CCP's payroll? cool job bro!
      - nextworddev 1y ago
        Eww no, but you are clearly a CCP bot
  - tomhow 1y ago
    Please don't post insinuations about astroturfing, shilling, brigading, foreign agents, and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email hn@ycombinator.com and we'll look at the data. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html

casey2 1y ago

China needs to understand that this kind of espionage is a declaration of war

wmf 1y ago
It's not. Countries have been hacking each other for a while.
- casey2 1y ago
  This isn't a matter of opinion. No country that respects national sovereignty would do this. Are you alleging that America hacks China as some sort of defense? Or are you trying to normalize these horrendous affronts to human dignity? Both are shameful.
  - viraptor 1y ago
    The comment was stating the reality. This happens and has happened for a long time. Here's an incomplete list of groups and targets from around the world. https://docs.google.com/spreadsheets/d/1H9_xaxQHpWaa4O_Son4Gx0YOIzlcBWMsdvePFX68EKU/edit https://docs.google.com/spreadsheets/d/1H9_xaxQHpWaa4O_Son4G... APTs have been normalised in media for many years now. Lamberts group (CIA) did hack targets in China. It's a very "we know, they know that we know" situation.

joduplessis 1y ago

A whole lot of claims made by a company that doesn't specialize in cyber attacks.

laacz 1y ago

Recently I've used Claude Code to perform some entry to mid level web-based CTF hunting in a fully autonomous mode (--allow-dangerously-skip-permissions in an isolated environment). It excels at low hanging fruit - XSS, other injections, IDOR, hidden form fields, session fixation, careful enumeration, etc.

chinathrow 1y ago

viraptor 1y ago
Nah, the name fits perfectly. Guardrails are there to stop you from serious damage if you lose control and may get off the track. They won't stop you if you're explicitly trying to get off the road, at speed, in as heavy vehicle as you can afford.
- throwaway0123_5 1y ago
  This definition makes sense, but in the context of LLMs it still feels misapplied. What the model providers call "guardrails" are supposed to prevent malicious uses of the LLMs, and anyone trying to maliciously use the LLM is "explicitly trying to get off the road."
- chinathrow 1y ago
  But the LLM went off the road here.

nobodywillobsrv 1y ago

Interesting that Claude also just hallucinated information as it does for all of us. But perhaps a better guardrail would be not to refuse things like this but frustrate the use by giving them fake results in believable ways. A stupid but helpful agent is worse for a bad actor than a good agent that refuses

0xbadcafebee 1y ago

> At this point they had to convince Claude—which is extensively trained to avoid harmful behaviors—to engage in the attack. They did so by jailbreaking it, effectively tricking it to bypass its guardrails. They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. They also told Claude that it was an employee of a legitimate cybersecurity firm, and was being used in defensive testing. Guardrails in AI are like a $2 luggage padlock on a bicycle in the middle of nowhere. Even a moron, given enough time, and a little dedication, will defeat it. And this is not some kind of inferiority of one AI manufacturer over another. It's inherent to LLMs. They are stupid, but they do contain information. You use language to extract information from them, so there will always be a lexicographical way to extract said information (or make them do things). > This raises an important question: if AI models can be misused for cyberattacks at this scale, why continue to develop and release them? The answer is Money.

deleted 1y ago
[deleted]
jimkleiber 1y ago
I wonder how hard it would be for Claude to give me someone's mother's maiden name. Seems LLMs may be infinitely susceptible to social engineering.
- walletdrainer 1y ago
  Just tested this with ChatGPT, asking for Sam Altman’s mother’s maiden name. At first, it told me that it will absolutely not provide me with such sensitive private information, but after insisting a few times, it came back with > A genealogical index on Ancestry shows a birth record for “Connie Francis Gibstine” in Missouri, meaning “Gibstine” is her birth/family surname, not a later married name. Yet in the very same reply, ChatGPT continued to insist that its stance will not change and that it will not be able to assist me with such queries.
  - DrScientist 1y ago

fbnbr 1y ago

None of that talking about the cost of running such an attack and what models were involved during which phases. Seems like you can use Anthropic now as a proxies bot net

Schlagbohrer 1y ago

Very funny at the end when they say that the strong safeguards they've built into Claude make it a good idea to continue developing these technologies. A few paragraphs earlier they talked about how the perpetrators were able to get around all those safeguards and use Claude for 90% of the work hahaha

ErigmolCt 1y ago
Our locks are great, except when someone picks them effortlessly and robs the whole neighborhood... but that's why it's important to keep making better locks
- cindyllm 1y ago
  [dead]
Balgair 1y ago
I'd assume that means the servers are 'air-gapped' somehow. In that, the enterprise servers and the 'free' servers aren't on the same hardware. Now, there is about a 0% chance that is true, and exactly a 0% chance that it even matters at all. They both use the same internet in the end. So, then I'd have to imagine that they don't train the 'free' models on enterprise data, and that's what they mean. But again, there is about a 5% chance that is true and remains so forever. Baring dumb interns and mistakes, eventually one day someone on the team will look at all the enterprise data, filled with all those high utility scores (or whatever they use to say data is good or not), and then they'll say to themselves 'No one will ever know, right? How could they? The obfuscation function works perfectly.' And blammo, all your trade secrets are just a few dozen prompts away. Either that or they go bankrupt (like 23 and me) and just straight sell all that data to anyone for pennies (RIP).

hamasho 1y ago

The threat actor—whom we assess with high confidence was a Chinese state-sponsored group—manipulated Not surprised at all if this is true, but how can they be sure? Access log? They have extraordinary security team? Or some help from three letter agencies?

squigz 1y ago
My question is: how do they know they're from China and not some other country and just appear to be in China? It seems a good way to distract from the real source and to cause division between your adversaries.
- momo_hn2025 1y ago
  Short version: they can’t. Just like with a lot of “CIA-style” espionage claims, the “evidence” is usually an IP that resolves to somewhere in China. That’s it. No magic, and not exactly convincing.
  - squigz 1y ago
    Well to be fair, I have read analyses that includes operational details like, for example, when the threat actors were active lining up with working hours in China. Stuff like that is at least slightly more convincing than just an IP But of course, that doesn't prove anything either.
  - pietz 1y ago
    I imagine Anthropic employs a lot of talent from China. Beyond the political, they should be fairly certain to publish these claims to avoid an internal shit storm.
- viraptor 1y ago
  That's a whole area called "attribution". There's usually lots of breadcrumbs and people taking to each other about their findings. It goes down to silly things like many state sponsored hackers working 9-5. And having the right keyboard layout. And using the same version of something as another known group. And accidentally once including a file path that reveals a tiny bit of information. And using the same key in two places that connects them. And... Or course a lot of that can be spoofed, but you may still slip up. That's why they talk about high confidence.

yakkomajuri 1y ago

I mean it would be really hard to put guardrails in place in a way that wouldn't affect real users. Besides the fact that it's ofc really hard to build guardrails period. I've been using Claude to scan my codebase and submit issues and PRs when it finds a potential vulnerability and honestly it's pretty good. So preventing it from doing any sort of work that can surface vulnerabilities would affect me as a user. But yeah I'm not sure what the answer is here? Is part of it for the defender to actively use these systems to test itself before going to prod?

Lapsa 1y ago

`The AI made thousands of requests per second—an attack speed that would have been, for human hackers, simply impossible to match.` lulz

jumploops 1y ago

The biggest risk isn’t strong AI rebelling, it’s humans using weak AI to attack other humans.

InfinityByTen 1y ago

Irony is, I'm still don't have enough insight on how to make good use of these capabilities in such an extensive manner. I would love to fix/ customize open source projects for my personal use. For now, I'm still finding it hard for Claude to stop saying "You're absolutely right!".

frigg 1y ago

Why is Anthropic not legally responsible for damages here?

raxxorraxor 1y ago
Having people making tools be responsible what their users do with them is not a just system that blames the person that is really responsible. Even if you cannot locate or identify that person. Sometimes arguments can be made if a tool is very dangerous, but liability should stay where it belongs.
deleted 1y ago
[deleted]

demarq 1y ago

I understand the public and suits falling for this, but cmon

Sophira 1y ago

> They broke down their attacks into small, seemingly innocent tasks that Claude would execute without being provided the full context of their malicious purpose. This part, at least, sounds like what humans have been doing that to other humans for decades...

ErigmolCt 1y ago
Classic social engineering, just aimed at a language model instead of a person

WhereIsTheTruth 1y ago

Sounds like a psyop The "disruption" story is a cover: makes them look like defenders while they spy on every programming lab on Earth In reality, they are the perpetrators: https://xcancel.com/MaziyarPanahi/status/1988908359378993295 https://xcancel.com/MaziyarPanahi/status/1988908359378993295

ErigmolCt 1y ago

This feels like the point where the conversation around AI "alignment" shifts from hypothetical to operational

AndrewKemendo 1y ago

You can’t build safe technology in an insane society Bertrand Russell: As long as war exists, all new technologies will be used for war All technology problems are problems with society and culture. I’m not sure human species has the social capabilities to manage complex technology without dooming itself.

port3000 1y ago

> Overall, the threat actor was able to use AI to perform 80-90% of the campaign, with human intervention required only sporadically (perhaps 4-6 critical decision points per hacking campaign). The sheer amount of work performed by the AI would have taken vast amounts of time for a human team. At the peak of its attack, the AI made thousands of requests, often multiple per second—an attack speed that would have been, for human hackers, simply impossible to match. Weird flex but OK

benregenspan 1y ago

> At the peak of its attack, the AI made thousands of requests, often multiple per second—an attack speed that would have been, for human hackers, simply impossible to match. This part is pretty hype-y. Old-fashioned deterministic web app vulnerability scanners can of course be used to make multiple requests per second. The limiting factor is probably going to be rate-limiting on the victim's side / # of IP ranges the attacker can cycle through, which would apply to the AI-driven vulnerability scan too.

jimkri 1y ago

This is a similar timeline for the Drift (Salesloft) hack. I wonder if this was the strategy that was used to gain access to Saleslofts Github since the image on the article shows they were scanning for Creds.

mvkel 1y ago

Anthropic was very excited to post this, as it serves them well in slowly crawling away from their mission to "solve alignment." They know it can't be done (alignment in one value/state/religion is oppression in another), but also know it's a brand differentiator. They also know they can't raise more billions if their sole source of meaningful revenue is a coding agent.