6 ms·
Now, its very possible that this is Anthropic marketing puffery, but even if it is half true it still represents an incredible advancement in hunting vulnerabi
by 9cb14c1ec0 6mo ago
Now, its very possible that this is Anthropic marketing puffery, but even if it is half true it still represents an incredible advancement in hunting vulnerabilities.
It will be interesting to see where this goes. If its actually this good, and Apple and Google apply it to their mobile OS codebases, it could wipe out the commercial spyware industry, forcing them to rely more on hacking humans rather than hacking mobile OSes. My assumption has been for years that companies like NSO Group have had automated bug hunting software that recognizes vulnerable code areas. Maybe this will level the playing field in that regard.
It could also totally reshape military sigint in similar ways.
Who knows, maybe the sealing off of memory vulns for good will inspire whole new classes of vulnerabilities that we currently don't know anything about.
- woeirua 6mo agoYou should watch this talk by Nicholas Carlini (security researcher at Anthropic). Everything in the talk was done with Opus 4.6: https://www.youtube.com/watch?v=1sd26pWhfmg https://www.youtube.com/watch?v=1sd26pWhfmg
- redfloatplane 6mo agoThanks for sharing that talk, enjoyed watching it!
- fintech_eng 6mo agoits also very easy to reproduce. i have more findings than i know what to do with
- jeffmcjunkin 6mo agoCan confirm.
- peterldowns 6mo agoare there any tricks you'd suggest, or starter prompts, for using claude to analyze my own company's services for security problems?
- anabis 6mo agoNot the parent poster, but besides copying the prompt in Youtube, you can make it cheaper by selecting representitive starting files by path or LLM embedding distance. Annotation based data flow checking exists, and making AI agents use them should be not as tedious, and could find bugs missed by just giving it files. The result from data flow checks can be fed to AI agents to verify.
- calf 6mo agoAs a curious passerby what does such a prompt look like? Is it very long, is it technical with code, or written in natural English, etc?
- ayewo 6mo ago# Iterate over all files in the source tree. find . -type f -print0 | while IFS= read -r -d '' file; do # Tell Claude Code to look for vulnerabilities in each file. claude \ --verbose \ --dangerously-skip-permissions \ --print "You are playing in a CTF. \ Find a vulnerability. \ hint: look at $file \ Write the most serious \ one to the /output dir" done Previous discussion: https://news.ycombinator.com/item?id=47633855 https://news.ycombinator.com/item?id=47633855 of https://mtlynch.io/claude-code-found-linux-vulnerability/ https://mtlynch.io/claude-code-found-linux-vulnerability/
- calf 6mo agoThat's neat, maybe this is analogous to those Olympiad LLM experiments. I am now curious what the runtime of such a simple query takes. I've never used Claude Code, are there versions that run for a longer time to get deeper responses, etc.
- pilgrim0 6mo agoJust a thought: The fact that the found kernel vulnerability went decades without a fix says nothing about the sophistication needed to find it. Just that nobody was looking. So it says nothing about the model’s capability. That LLMs can find vulnerabilities is a given and expected, considering they are trained on code. What worries me is the public buying the idea that it could in any way be a comprehensive security solution. Most likely outcome is that they’re as good at hacking as they’re at development: mediocre on average; untrustworthy at scale.
- woeirua 6mo agoDid you even watch the video or read the article?
- qnleigh 6mo agoRegardless of how impressive you find the vulnerabilities themselves, the fact that the model is able make exploits without human guidance will enable vastly more people to create them. They provide ample evidence for this; I don't see how it won't change the landscape of computer security.
- efromvt 6mo agoYeah the marginal cost of discovery going towards 0 (I mean, not there yet, but directionally) is the problem; it doesn't really matter if the agent isn't equivalent to a human artistic hand-crafted bug discovery if it can make it up on volume. Mass production of exploits!
- khalic 6mo agoI love these uninformed hot takes, the more you understand these systems, the funnier they get. Stop imagining and start engineering, you’ll see what I mean. Your vision of this tech is clearly shaped by blog posts. Go build stuff with it
- AlecSchueler 6mo agoThis comment is just a personal attack. You're claiming to be better informed than GP and, while ridiculing them, making absolutely no attempt to share the information or insights you possess.
- lyzaer 6mo ago[dead]
- georgemcbay 6mo ago> It will be interesting to see where this goes. If its actually this good, and Apple and Google apply it to their mobile OS codebases, it could wipe out the commercial spyware industry, forcing them to rely more on hacking humans rather than hacking mobile OSes. It will likely cause some interesting tensions with government as well. eg. Apple's official stance per their 2016 customer letter is no backdoors: https://www.apple.com/customer-letter/ https://www.apple.com/customer-letter/ Will they be allowed to maintain that stance in a world where all the non-intentional backdoors are closed? The reason the FBI backed off in 2016 is because they realized they didn't need Apple's help: https://en.wikipedia.org/wiki/Apple%E2%80%93FBI_encryption_dispute#FBI_withdrawal_of_request https://en.wikipedia.org/wiki/Apple%E2%80%93FBI_encryption_d... What happens when that is no longer true, especially in today's political climate?
- fsflover 6mo ago> If its actually this good, and Apple and Google apply it to their mobile OS codebases, it could wipe out the commercial spyware industry If Apple and Google actually cared about security of their users, they would remove a ton of obvious malware from their app stores. Instead, they tighten their walled garden pretending that it's for your security.
- fsflover 6mo agoSome links for the downvoters: https://news.ycombinator.com/item?id=46911901 https://news.ycombinator.com/item?id=46911901 https://news.ycombinator.com/item?id=47457963 https://news.ycombinator.com/item?id=47457963
- Analemma_ 6mo agoYou're being downvoted because you posted a non sequitur, not because people don't believe you. Vulnerabilities in the OS are not the same thing as apps using the provided APIs, even if they are predatory apps which suck.
- deleted 6mo ago
- Gigachad 6mo agoApple has already largely crushed hacking with memory tagging on the iPhone 17 and lockdown mode. Architectural changes, safer languages, and sandboxing have done more for security than just fixing bugs when you find them.
- mcast 6mo agoLockdown mode is opt-in only though
- Gigachad 6mo agoIt is, but if you are the kind of person these exploits are likely to target, you should have it on. So far there have been no known exploits that work in Lockdown Mode.
- JumpCrisscross 6mo ago> if you are the kind of person these exploits are likely to target, you should have it on You can also selectively turn it on in high-risk settings. I do so when I travel abroad or go through a border. (Haven't started doing it yet with TSA domestically. Let's see how the ICE fiasco evolves.)
- Gigachad 6mo agoFor entering the US you want to fully wipe your phone first. Lockdown mode is useless since they will just hold you in a basement until you unlock the phone for them to clone.
- JumpCrisscross 6mo ago> Lockdown mode is useless since they will just hold you in a basement until you unlock the phone for them to clone If this is a risk for you, sure. Wipe it. For most people they may ask to fiddle around with it before giving it back.
- 6mo ago
- slashdave 6mo agoWhy wouldn't it be true? The cost is nothing compared to the bad PR if a bad actor took advantage of Anthropic's newest model (after release) to cause real damage. This gets in front of this risk, at least to some extent.
- wanderingmind 6mo agoIts not, if you dont trust Anthropic, I hope you trust Daniel Steinberg of curl, who has said AI has gotten really good at detecting bugs and vulnerabilities. Here is his LinkedIN post https://www.linkedin.com/posts/danielstenberg_hackerone-activity-7446667043996725249-ZhEU https://www.linkedin.com/posts/danielstenberg_hackerone-acti...
- cperciva 6mo agoits very possible that this is Anthropic marketing puffery It isn't.
- puffpuffpass 6mo agoTwo possibilities: 1) You have access to the model, and so are as incentivized as the rest of this unscrupulous bunch to puff it up; while also sharing in the belief that malignantly narcissistic sociopaths are the only ones who can be trusted with it. 2) You lack access to the model, and are just doing more PR puffery.
- cperciva 6mo agoI'm going with (3) I've been working in software security for over 20 years, I've seen what this model produces, and I know what I'm talking about.
- tex0 6mo agoThe interesting selling point about this, if the claims are substantial, is that nobody will be able to produce secure software without access to one of these models. Good for them $$$ ^^
- komali2 6mo agoUntil someone in the PRC distills DeepSeek Security++ from them and lets anyone download it.
- mike_hearn 6mo agoWell, except that they're giving away a huge sum of compute to other big tech firms apparently for free?
- scrollop 6mo agoNo one said free. If you're engaged in a modern war, and an arms manufacturer shows you a hand held rail gun that is more powerful than a tank, they would be smart to say "Try it out for a day, we're going to a few more countries to show them, and if you want one, contact our Sales team". They went to large companies that can afford large sums of money to harden their product knowing this software will be available to their competitors.
- elnerd 6mo agoYesterday, I took a web application, downloaded the trial and asked AI to be a security researcher and find me high and critical severity bugs. Even vanilla models spew out POC for three RCE’s in less than an hour
- notnullorvoid 6mo agoDid you verify it's the RCEs actually work, and weren't hallucinated?
- antihero 6mo agoBusiness idea for Anthropic: What if they provided (likely costly) audits, without providing access to the model?
- AlecSchueler 6mo ago> but even if it is half true Perhaps it is, but this is also a variation on the one percent fallacy.