3 ms·

> We are releasing Opus 4.7 with safeguards that automatically detect and block requests that indicate prohibited or high-risk cybersecurity uses. Fucking hell

• zb3 6mo ago
  It appears we're learning the hard way that we can't rely on capabilities of models that aren't open weights. These can be taken from us at any time, so expect it to get much worse..
  • hootz 6mo ago
    Can't wait for a random chinese company to train a model on Mythos by breaking Anthropic's ToS just to release it for free and with open weights.
• Havoc 6mo ago
  Claude code had safeguards like that hardcoded into the software. You could see it if you intercept the prompts with a proxy
• methodical 6mo ago
  To be fair, delineating between benevolent and malevolent pen-testing and cybersecurity purposes is practically impossible since the only difference is the user's intentions. I am entirely unsurprised (and would expect) that as models improve the amount to which widely available models will be prohibited from cybersecurity purposes will only increase. Not to say I see this as the right approach, in theory the two forces would balance each other out as both white hats and black hats would have access to the same technology, but I can understand the hesitancy from Anthropic and others.
  • ACCount37 6mo ago
    Yes, and the previous approach Anthropic took was "allow anything that looks remotely benign". The only thing that would get a refusal would be a downright "write an exploit for me". Which is why I favored Anthropic's models. It remains to be seen whether Anthropic's models are still usable now. I know just how much of a clusterfuck their "CBRN filter" is, so I'm dreading the worst.
  • trinix912 6mo ago
    But this technology is now out there, the cat's out of the bag, there's no going back to a world where people can't ask AI to write malware for them. I'd argue that black hats will find a way to get uncensored models and use them to write malware either way, and that further restricting generally available LLMs for cybersec usage would end up hurting white hats and programmers pentesting their own code way more (which would once again help the black hats, as they would have an advantage at finding unpatched exploits).
  • ninjagoo 6mo ago
    > since the only difference is the user's intentions Have these been banned yet: dual-use kitchen items, actual weapons of war for consumer use, dual-use garden chemicals, dual-use household chemicals etc. etc? Has human cybersecurity research stopped? Have malware authors stopped research? No? then this sounds more like hype than real reasons. There's also the possibility that there's a singular anthropic individual who's gained a substantial amount of internal power and is driving user-hostile changes in the product under the guise of cybersecurity.
• senko 6mo ago
  From the article: > Security professionals who wish to use Opus 4.7 for legitimate cybersecurity purposes (such as vulnerability research, penetration testing, and red-teaming) are invited to join our new Cyber Verification Program.
  • ACCount37 6mo ago
    Yeah no. They can fuck right off with KYC humiliation rituals.
  • atonse 6mo ago
    This seems reasonable to me. The legit security firms won't have a problem doing this, just like other vendors (like Apple, who can give you special iOS builds for security analysis). If anyone has a better idea on how to _pragmatically_ do this, I'm all ears.
    • adrian_b 6mo ago
      If the vendors of programs do not want bugs to be found in their programs, they should search for them themselves and ensure that there are no such bugs. The "legit security firms" have no right to be considered more "legit" than any other human for the purpose of finding bugs or vulnerabilities in programs. If I buy and use a program, I certainly do not want it to have any bug or vulnerability, so it is my right to search for them. If the program is not commercial, but free, then it is also my right to search for bugs and vulnerabilities in it. I might find acceptable to not search for bugs or vulnerabilities in a program only if the authors of that program would assume full liability in perpetuity for any kind of damage that would ever be caused by their program, in any circumstances, which is the opposite of what almost any software company currently does, by disclaiming all liabilities. There exists absolutely no scenario where Anthropic has any right to decide who deserves to search for bugs and vulnerabilities and who does not. If someone uses tools or services provided by Anthropic to perform some illegal action, then such an action is punishable by the existing laws and that does not concern Anthropic any more than a vendor of screwdrivers should be concerned if someone used one as a tool during some illegal activity. I am really astonished by how much younger people are willing to put up with the behaviors of modern companies that would have been considered absolutely unacceptable by anyone, a few decades ago.
• johnmlussier 6mo ago
  Incredible - in one fell swoop killing my entire use case for Claude. I have about 15 submissions that I now need to work with Codex on cause this "smarter" model refuses to read program guidelines and take them seriously.
• brynnbee 6mo ago
  I'm currently testing 4.7 with some reverse engineering stuff/Ghidra scripting and it hasn't refused anything so far, but I'm also doing it on a 20 year old video game, so maybe it doesn't think that's problematic.
  • ACCount37 6mo ago
    I really hope it's that way for my use cases too, also Ghidra and decompiler outputs, but I'm not optimistic.