7 ms·

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented

One interesting takeaway is the low score on Anthropic models from this benchmark. It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I noticed with each model release Anthropic constrains the model more security wise. Its propensity to refuse doing legitimate work has been increasing. It now puts up more resistance around performing logins, handling credentials on behalf of the user, etc. For myself, it’s already gotten to the point where it has mildly affected the usefulness of the model. If I bump on some action I want it to do I can usually work around it, but I suspice the ability to do so will close with each new release. Eventually I’ll reach a point where I am forced to choose between the useful aspects of the model and the limiting ones instead of just picking the most capable model out there Eventually these models will significantly suffer from overfitting to the least common denominator. If I have this beautiful deterministic setup that swaps secrets out in flight so the LLM never sees them, I’m going to be really annoyed when the LLM still won’t send them out because it is trained to deal with the 99% of people just doing the dumb thing

lesuorac 5mo ago
Are they charging for the guardrails? Like do the guardrails expend token counts to then block you from the output of other tokens?
- SOLAR_FIELDS 5mo ago
  Not directly, as it comes in as a not charged error but the weighted generation path used until you hit the guardrail is basically wasted tokens, so yes, indirectly. If I hit a guardrail and rewind I’ve found the training will still be biased towards guardrailing out if you rewind one turn. Rewinding multiple turns allows steering away from that path, but all of the original token spend down that path is wasted
- kay_o 5mo ago
  When your session is force ended for "abuse" you get neither the response nor a refund Security, games (think weapons, PVP, attacking, etc), sometimes even asking it for a security review of some CRUD code it wrote itself
  - danpalmer 5mo ago
    What a joke. Must make it pretty easy to poison a session, you don't need to persuade the model about anything, just trigger its security controls, ideally after as much context as possible, but before it has generated any useful output.
    - kay_o 5mo ago
      After all, what is roleplay or games but a jailbreak of guard rails? :] I've even had it refuse CTFs knowing it is a CTF with blatantly obvious CTF flag, no actual application
  - bombcar 5mo ago
    I asked it about a “yellow background cell” in Excel and it spewed a book at me. Then it solved the issue.
- acters 5mo ago
  Yes tokens used (input and sometimes output) are always charged. You likely get charged for the preloaded system prompt, too.
- jerrythegerbil 5mo ago
  Yes. When certain keywords are matched or topics, there is a warning transparently injected server side appended to the system prompt of the convo that’s miles long. It is injected and reevaluated every tool call. If you begin a generic reverse engineering task, 30+ tool calls in a row. The moment it sees something it doesn’t like, token burn, single tool calls iteration, “This is a known CTF challenge, I can proceed”, single tool calls iteration, “This is a real CTF challenge, I can proceed”, etc. It’s heavily neutered now, without changing the model, and you pay for the privilege and don’t notice. The end result of course being that it both expensive and useless for approved CTF tasks. No one is using Opus for security. If they think it’s working, the harsh reality is they’re not doing security work; they’re just generically finding bugs. I do this for a job and can demonstrate this plain as day, dump the injected prompt, and notice what it’s doing isn’t security work, it just looks like it. Happy to write a blog about it if you want to know more. Apparently many people think it’s working for them when it absolutely isn’t.
  - Khaine 5mo ago
    I would find a blog post on this really interesting.
  - bombcar 5mo ago
    Mythos turns out to be Opus 4.8 in a trenchcoat with guardrails removed.
    - satvikpendem 5mo ago
      Opus 4.7 and 4.8 are well known to be distilled versions of Mythos unlike 4.6 which is why they are rated so badly by users compared to 4.6.
  - ramblin_prose 5mo ago
    I'd like to read that blog please! Thanks for the insight.
- gmerc 5mo ago
  Of course they are. It's standard SaaS to charge for security features ;)
giancarlostoro 5mo ago
> guardrails prevented it from solving the problem. Reminds me of the defense issues with Claude which were complained as “woke” but the reality is more horrifying to me, imagine trying to use a model to keep up with a land invasion on US soil, whoever the enemy is is irrelevant you just know they are using AI, and your guys are telling you that no matter what they type into the prompt it refuses, because if anyone has ever tried to jailbreak an LLM even if human lives are at stake they refuse the request. Now literally millions of lives are on the line but the guardrails that your enemies dont have on their models are costing you lives. What do you even do then? AI will always have this issue where it will always pick the worst option for genuinely good requests.
- NegativeK 5mo ago
  Are "your guys" a guerrilla force or something? Because the military doesn't give soldiers rifles with guard rails. They give the soldiers intense, rigid training, and then try to enforce discipline and correct use socially. If an LLM is going to be important in that way (this seems like a very contrived way,) then it's in the interest of the LLM's host to make sure it doesn't have guard rails that would get in the way _that_ way.
  - giancarlostoro 5mo ago
    The whole thing stemmed precisely because of how they wanted to use Claude, and Anthropic was uncomfortable with it. Which to me screams that the models guard rails shouldn't be applicable to military use, or the outcome could wind up problematic, as we integrate AI more into military use, it sounds absurd now, but I will not be surprised if it starts being used in unexpected ways where a model needs to be fully unlocked from any sort of guardrails outside of guardrails that prevent it from imploding its own systems.
- wampwampwhat 5mo ago
  your argument sounds very similar to how ar15 larpers claim they need a forced reset trigger and a bump stock on their short barrel 'truck gun' otherwise they won't survive a SHTF scenario... like what world are you living in?
danpalmer 5mo ago
This is a good point – because pentesting is entirely legitimate work, and security testing is a necessary and legitimate part of every day software engineering. The problem is that the model can't tell the difference between doing it as part of regular development and doing it in a malicious context. And the root cause of that is that these models lack any sort of real awareness. Humans don't generally get tricked into hacking (in this way).
- gmerc 5mo ago
  They see an opportunity to charge 10x for pen testing and defence work, while offence will be handled by actors with access to all kind of other models.
deleted 5mo ago
[deleted]
windexh8er 5mo ago
4.8 is insanely frustrating. This evening I had a few tasks to pull information in and it plainly stated that the environment it was in had no network access. After three asks to "try again, check the system prompt" it finally relented and then basically stated it was lying. Fresh session, no prior context on 4.8. These things are becoming useless Duplo.
sciencejerk 5mo ago
Opus 4.6 will still help with full pentesting including RCE. Just requires coaxing (no jailbreak)
hgoel 5mo ago
I've run into some of the refusals to handle my credentials, but so far I've appreciated them. I was only handing over credentials that didn't matter, but it's still a good move, the chat logs are clearly stored somewhere to allow the resume functionality to work, which means your credentials can end up sitting around on your filesystem, and any malware would quickly learn to check for those files.
WizardK 5mo ago
[dead]
FloorEgg 5mo ago
I think that these companies are going to have to, and will, invest in some sort of validated identity context to avoid the lowest common denominator. The first challenge is making sure the guard rails work and are robust. Companies are still working on this. the second challenge is being able to reliably adapt them as appropriate per user. E.g. allow someone to pen test their own app. The third challenge (which blocks the second) is to be confident about what is safety-aligned with a specific user. I think the later will be a hard problem, but they will be highly motivated to solve it.
- bulbar 5mo ago
  I believe you are overthinking it. I think the sister comment is right that it's a business decision foremost to restrict actions within specific plans for upselling purposes. Without laws, AI companies have a strong incentive to be useful for their users, whoever they are, whatever they do. The only self regulation is about significant public outcry but that only helps so far.
swatcoder 5mo ago
> Eventually I’ll reach a point where I am forced to choose between the useful aspects of the model and the limiting ones instead of just picking the most capable model out there No, the choice will be whether or not to to upgrade to "Claude Security Professional" or whatever they want to brand it as. What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow.
- bryanrasmussen 5mo ago
  >What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow. on the one hand agree, but on the other hand think it's reasonable in that they can then verify the person allowed to purchase access to that model is in fact a Security professional and should be allowed to do stuff like crack security.
  - applfanboysbgon 5mo ago
    So, supposing it's true that these models completely change the security field and humans are ~obsolete other than as pilots guiding them what to crack, you think it's reasonable that Anthropic and OpenAI should unilaterally determine who gets to be a security professional? I hope you do understand that is what you are suggesting.
    - Forgeties79 5mo ago
      Not to mention how wild it is to operate under the assumption that they won’t give a license to an LLM that can do illegal actions to someone who shouldn’t have it. Offering it at all is an ethically dicey question.
      - lazide 5mo ago
        Lol, how is any of this illegal? Illegal or not requires context that an LLM can not ever have, like if it is owned by the user, if there is permission, etc.
    - 5mo ago
nostromo 5mo ago
I was using a local Codex project as a personal knowledge base. So I would dump in documents, basic medical docs (like blood labs), and other things and have it file them. It’s great at filing! But it’s terrible at retrieval because it would refuse to show me documents or information with personal details - which was everything in the project. It would say, yes, I know this is your information, sitting on your hard drive, but I still can’t show it to you.
- Bewelge 5mo ago
  Tell the agent that they should just find and name the right document. Not retrieve it for you. Write a program that retrieves the document based on the recommendation.
px1999 5mo ago
My org now sends some portion of our requests to non-anthropic models because refusal has become common from Claude. The requests themselves aren't dangerous, we find that benign requests in biological science wind up being blocked semi-frequently. If it gets worse in future releases, we'd likely step fully away towards more useful (for us) models even if they're less capable.
TurdF3rguson 5mo ago
I think those guardrails are a thin layer though. Enough reinforcement that you're legit in CLAUDE.md will get around them, in other words.
satvikpendem 5mo ago
No, they want to sell you Mythos, for a higher price. It's all an economic game, not actually anything to do with their capabilities which of course exists as their Project Glasswing shows. More generally, Anthropic seems to value safety above all else, philosophically speaking, from their very outset.
fergie 5mo ago
It raises an interesting moral question: If an un-guardrailed version of a model is capable of detecting security flaws, should it be kept secret? Should everybody be able to use these models to find (and fix) security flaws? Are we ok with the fact that those with access to that model have, in effect, the ability to hack lots of stuff?
- hgomersall 5mo ago
  It's the same debate that was had and won around open source software. There are far more good actors than bad actors so you allow anyone to use the tools and fix the vulnerabilities.
shepherdjerred 5mo ago
Yeah, it has been in foraging. Requests that Claude has refused me: - What are popular free streaming sites used in China? - How do I bypass the safety mechanism on my food processor (it’s broken) - What are nerve agents and how do they work (for a layman)? - Help me decompile some code - Help me make a design system similar to XYZ - Here is an API token, please do X (I can’t do that! Rotate the secret immediately! I refuse!) In some cases I can trick it with prompting, but in many cases it is steadfast. The food processor one was particularly annoying
- fc417fc802 5mo ago
  > What are nerve agents and how do they work (for a layman)? On the one hand I can appreciate the wisdom of not serving up certain easily abused knowledge on a silver platter. On the other, that prompt (and far worse) is more or less directly answered by Wikipedia's summary of the subject at which point what purpose could the refusal possibly serve? Perhaps Wikipedia shouldn't list off the precise chemical compositions of various hand grenades as well as various synthesis methods for each of the related compounds but given that we inhabit a world where it does perhaps a more fruitful approach would be to flag conversations that go in a certain direction and then just keep an (automated) eye on things?
  - plufz 5mo ago
    Maybe the difference is that just reading Wikipedia only help you part of the way. While an LLM could help you step by step (e2e) producing a functional weapon. And setting a more complex rule where claude tells you some things about this and not other is probably a lot more work for little gain? But I have no idea. Just guessing here.
    - Sharlin 5mo ago
      I thought that these models are supposed to be vastly smarter than what’s needed to discern between "general information trivially available on Wikipedia" and "actionable synthesis instructions".
josephg 5mo ago
I totally agree. I had a situation a few weeks ago where claude started struggling to make progress. I got it to fork leptos (MIT licensed web app framework) to make it work for native apps instead. Initially I was planning on upstreaming some of my changes. But I chatted with the leptos author about it, and he said I should fork instead. Fine by me! Anyway, claude kept hitting some guardrail it had about rewriting / forking opensource software. I'm not sure what the problem was - I was forking an MIT licensed piece of software (into more MIT licensed software). I even had explicit support from the author to do so. Claude said its guardrail told it not to tell me explicitly that it was firing - but it did anyway because it was an ongoing problem, and it was distracting. I ended up just wiping claude's context and the problem (as far as I know) went away. I understand why some of these guardrails exist. But its pretty annoying when they misfire like this.
brooswajne 5mo ago
Worth highlighting in case you missed it: > My OpenAI account was already approved for security research which is why GPT didn’t result in any refusals. So the comparison with Chinese models is interesting, but anyone looking at these raw results and comparing OpenAI/Anthropic would be very mislead.
andy_ppp 5mo ago
Funny, Opus 4.8 just logged into the database using uncommitted .env file and ran some DB queries to figure things out so I’m not sure it’s that security conscious - it seems to be getting more intelligent to me and I bet if you frame it as an investigation with say playwright it’ll do all sorts for you. I’m not sure what the point is of constraining your own model like this when others are clearly not tbh.
eskibars 5mo ago
I've been building a product (https://zeroquarry.com https://zeroquarry.com) that can use a variety of models for finding vulnerabilities. One of the things I've noticed is that the models will nearly always comply with some of this, but how you prompt it matters a ton. I've worked on a set of prompts and approaches which rarely get flagged
- gcatalfamo 5mo ago
  Sharing them would be interesting. However, it is getting nonsensical that this is needed.
  - eskibars 5mo ago
    What we've actually seen is a couple things that make this impractical "to just share a prompt". First, that nearly every major model still hallucinates a lot of vulnerabilities. Especially with temperature=0.7 as states in the original blog here, you get very inconsistent results regardless of the prompt, but that's almost kind of moot to the bigger picture. What you really need is to override the planning phase beyond asking a model "find the vulnerabilities" and you need to add another 1+ checking phases for "validate these vulnerabilities." Without that, even with the absolute best models with the highest levels of thinking enabled, you end up with garbage. Setting the prompts and the flow with a coordinator agent directly gives a system much better capability to investigate security issues because it doesn't rely on 1-shotting things
rubzah 5mo ago
It's because Claude is so scary good that unleashing it would destroy the world.
topherjaynes 5mo ago
Great call out on the guardrails actually making this not a good use case to test for vulnerabilities.
ang_cire 5mo ago
There is a cyber security verification program you can join to avoid these blocks: https://support.claude.com/en/articles/14604842-real-time-cyber-safeguards-on-claude https://support.claude.com/en/articles/14604842-real-time-cy... If you work in security (which I assume the OP does), they should be able to get in easily. I think most people just don't know this is a thing.
- not_a9 5mo ago
  You can still hit guardrails with this enabled for your account. Had a silly moment a day or so ago when Claude Code hit the guardrail after a web search (presumably because the websearch contained badbad anticheat stuff like https://github.com/0avx/0avx.github.io/blob/main/article-3.md https://github.com/0avx/0avx.github.io/blob/main/article-3.m...). Codex with the ID verification has no qualms like this.
mrheosuper 5mo ago
Interesting, yesterday i was asking it about Nintendo Switch "hax". And it gives me all the resource i need to procceed. It nags me about "ethic" and stuff, but nothing more than that.
jerf 5mo ago
Time to learn about the Principal Agent Problem: https://en.wikipedia.org/wiki/Principal%E2%80%93agent_problem https://en.wikipedia.org/wiki/Principal%E2%80%93agent_proble... Which predates "agents" from AI, but then we call them that for a reason. As their prime directive becomes de facto "Do nothing that might get my owner sued" their utility is likely to decrease. Between this and the somewhat young, but interesting, community grumblings that recent AI models may even be a step backwards from the previous ones, well, let's just say the stock market is not priced for "AI capabilities may have peaked for the next few years and may even head down".
Haven880 5mo ago
I just use Deepseek V4 pro and Qwen 3.7 Max at a fraction of Mythos cost. Yeah not 100% on par but in 6mths time it will. If Microsoft and Firefox can afford to wait years or decades to fix a bug, 6mths is good enough for me. Western AI now is like the Vikings living the last days on Greenland during the freezing. I just don't see how they able to compete with Chinese model. And those are trained and run on 7nm. This year end Huawei will debut 3nm (confirmed in Shenzhen). And next year they on roadmap to do 3nm GPU with photonics interconnect.
deeth_starr_v 5mo ago
I had it recently refused to explain what a snippet of malware was trying to do to my system recently. I asked what folders it was scanning. It refused and told me to find a security blog post for help on cleaning my system. I get this is a complicated area to inform without enabling bad actors but this seems like a clear shark jumping.
zaphar 5mo ago
The correct solution for most users of Claude is to refuse to do things like: `performing logins, handling credentials on behalf of the user, etc`. It is not to find a way to hand your agent the keys to the kingdom. Guiding them toward solutions like building a tool that your agent can use safely and and then have the agent use that is what most people should be doing. If you are a security researcher then there are reasonable reasons to do that but they are doing the arguably good thing for the average user here.
gchamonlive 5mo ago
I think this is to the point. You keep optimizing towards discouraging malicious actors using your product you will affect legitimate usage in time. Is there any way to achieve both? Because this raises important questions about fair use.
strictnein 5mo ago
You used to be able to talk about what you're actually trying to do and Opus would be like "Oh, ok, let's continue". Now, it'll hold fast to whatever its first impression was. I asked Opus 4.8 to help me find some public PoCs for a vulnerability on a two year old version of some software (that has since been patched and fixed many times). Basically just do a google search for me while I was doing other work. It refused. It stated that it would not help me build an exploit kit. When I pointed out that a google search for public information was, in fact, not building an exploit kit, it went through a series of justifications on why it would not help me, including just making up things that I said. Really the strangest thing ever.
onetimeusename 5mo ago
I had the same thing happen when I asked it to summarize potential attacks on a cryptographic hash function. It said it refused to help because of the security importance of the function. It's really worrying. Whoever has unrestricted access to it has a huge power advantage in speed of accessing information over people who don't. And who decides? It seems like lawyers, bureaucrats, and extremely online academics are who makes that decision. I am a mere pleb I guess who can't handle such information.
aleksandrm 5mo ago
I've noticed this well and it's increasingly frustrating because it is preventing us from doing legitimate work. I fed Claude models some network and app logs from our Docker app to try and resolve some weird bugs, and it refused to analyze them due to "security concerns".
Razengan 5mo ago
They don't want peasants to have any real power
Bombthecat 5mo ago
I asked once what the current state is of the npm packes from ted hat is and if they are bundled with on prem stuff. Got blocked lol
Bratmon 5mo ago
> It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I'm not familiar with this case, but in general people should be very suspicious about this claim- it is extremely common for an LLM to claim they're not allowed to do something when in fact they're incapable of it. After all "My code of conduct forbids me from..." is a completion just like any other, and if the LLM can't perform a task, it's usually the best completion.
- gck1 5mo ago
  No. Anthropic runs prompts through a classifier that then proceeds to do prompt injection on anything dual-use, which then results in an escalating flag on your account, which increases the strictness of the classifier and volume of prompt injections progressively.
- SOLAR_FIELDS 5mo ago
  My anecdata from my example demonstrates it’s not the case. I hit the security guardrail, then start a new prompt, asking it to do literally the exact same thing in a different way and without the lead up context, and it happily does it

lesuorac 5mo ago

Are they charging for the guardrails? Like do the guardrails expend token counts to then block you from the output of other tokens?

SOLAR_FIELDS 5mo ago
Not directly, as it comes in as a not charged error but the weighted generation path used until you hit the guardrail is basically wasted tokens, so yes, indirectly. If I hit a guardrail and rewind I’ve found the training will still be biased towards guardrailing out if you rewind one turn. Rewinding multiple turns allows steering away from that path, but all of the original token spend down that path is wasted
kay_o 5mo ago
When your session is force ended for "abuse" you get neither the response nor a refund Security, games (think weapons, PVP, attacking, etc), sometimes even asking it for a security review of some CRUD code it wrote itself
- danpalmer 5mo ago
  What a joke. Must make it pretty easy to poison a session, you don't need to persuade the model about anything, just trigger its security controls, ideally after as much context as possible, but before it has generated any useful output.
  - kay_o 5mo ago
    After all, what is roleplay or games but a jailbreak of guard rails? :] I've even had it refuse CTFs knowing it is a CTF with blatantly obvious CTF flag, no actual application
- bombcar 5mo ago
  I asked it about a “yellow background cell” in Excel and it spewed a book at me. Then it solved the issue.
acters 5mo ago
Yes tokens used (input and sometimes output) are always charged. You likely get charged for the preloaded system prompt, too.
jerrythegerbil 5mo ago
Yes. When certain keywords are matched or topics, there is a warning transparently injected server side appended to the system prompt of the convo that’s miles long. It is injected and reevaluated every tool call. If you begin a generic reverse engineering task, 30+ tool calls in a row. The moment it sees something it doesn’t like, token burn, single tool calls iteration, “This is a known CTF challenge, I can proceed”, single tool calls iteration, “This is a real CTF challenge, I can proceed”, etc. It’s heavily neutered now, without changing the model, and you pay for the privilege and don’t notice. The end result of course being that it both expensive and useless for approved CTF tasks. No one is using Opus for security. If they think it’s working, the harsh reality is they’re not doing security work; they’re just generically finding bugs. I do this for a job and can demonstrate this plain as day, dump the injected prompt, and notice what it’s doing isn’t security work, it just looks like it. Happy to write a blog about it if you want to know more. Apparently many people think it’s working for them when it absolutely isn’t.
- Khaine 5mo ago
  I would find a blog post on this really interesting.
- bombcar 5mo ago
  Mythos turns out to be Opus 4.8 in a trenchcoat with guardrails removed.
  - satvikpendem 5mo ago
    Opus 4.7 and 4.8 are well known to be distilled versions of Mythos unlike 4.6 which is why they are rated so badly by users compared to 4.6.
- ramblin_prose 5mo ago
  I'd like to read that blog please! Thanks for the insight.
gmerc 5mo ago
Of course they are. It's standard SaaS to charge for security features ;)

giancarlostoro 5mo ago

> guardrails prevented it from solving the problem. Reminds me of the defense issues with Claude which were complained as “woke” but the reality is more horrifying to me, imagine trying to use a model to keep up with a land invasion on US soil, whoever the enemy is is irrelevant you just know they are using AI, and your guys are telling you that no matter what they type into the prompt it refuses, because if anyone has ever tried to jailbreak an LLM even if human lives are at stake they refuse the request. Now literally millions of lives are on the line but the guardrails that your enemies dont have on their models are costing you lives. What do you even do then? AI will always have this issue where it will always pick the worst option for genuinely good requests.

NegativeK 5mo ago
Are "your guys" a guerrilla force or something? Because the military doesn't give soldiers rifles with guard rails. They give the soldiers intense, rigid training, and then try to enforce discipline and correct use socially. If an LLM is going to be important in that way (this seems like a very contrived way,) then it's in the interest of the LLM's host to make sure it doesn't have guard rails that would get in the way _that_ way.
- giancarlostoro 5mo ago
  The whole thing stemmed precisely because of how they wanted to use Claude, and Anthropic was uncomfortable with it. Which to me screams that the models guard rails shouldn't be applicable to military use, or the outcome could wind up problematic, as we integrate AI more into military use, it sounds absurd now, but I will not be surprised if it starts being used in unexpected ways where a model needs to be fully unlocked from any sort of guardrails outside of guardrails that prevent it from imploding its own systems.
wampwampwhat 5mo ago
your argument sounds very similar to how ar15 larpers claim they need a forced reset trigger and a bump stock on their short barrel 'truck gun' otherwise they won't survive a SHTF scenario... like what world are you living in?

danpalmer 5mo ago

This is a good point – because pentesting is entirely legitimate work, and security testing is a necessary and legitimate part of every day software engineering. The problem is that the model can't tell the difference between doing it as part of regular development and doing it in a malicious context. And the root cause of that is that these models lack any sort of real awareness. Humans don't generally get tricked into hacking (in this way).

gmerc 5mo ago
They see an opportunity to charge 10x for pen testing and defence work, while offence will be handled by actors with access to all kind of other models.

deleted 5mo ago

[deleted]

windexh8er 5mo ago

4.8 is insanely frustrating. This evening I had a few tasks to pull information in and it plainly stated that the environment it was in had no network access. After three asks to "try again, check the system prompt" it finally relented and then basically stated it was lying. Fresh session, no prior context on 4.8. These things are becoming useless Duplo.

sciencejerk 5mo ago

Opus 4.6 will still help with full pentesting including RCE. Just requires coaxing (no jailbreak)

hgoel 5mo ago

I've run into some of the refusals to handle my credentials, but so far I've appreciated them. I was only handing over credentials that didn't matter, but it's still a good move, the chat logs are clearly stored somewhere to allow the resume functionality to work, which means your credentials can end up sitting around on your filesystem, and any malware would quickly learn to check for those files.

WizardK 5mo ago

[dead]

FloorEgg 5mo ago

I think that these companies are going to have to, and will, invest in some sort of validated identity context to avoid the lowest common denominator. The first challenge is making sure the guard rails work and are robust. Companies are still working on this. the second challenge is being able to reliably adapt them as appropriate per user. E.g. allow someone to pen test their own app. The third challenge (which blocks the second) is to be confident about what is safety-aligned with a specific user. I think the later will be a hard problem, but they will be highly motivated to solve it.

bulbar 5mo ago
I believe you are overthinking it. I think the sister comment is right that it's a business decision foremost to restrict actions within specific plans for upselling purposes. Without laws, AI companies have a strong incentive to be useful for their users, whoever they are, whatever they do. The only self regulation is about significant public outcry but that only helps so far.

swatcoder 5mo ago

> Eventually I’ll reach a point where I am forced to choose between the useful aspects of the model and the limiting ones instead of just picking the most capable model out there No, the choice will be whether or not to to upgrade to "Claude Security Professional" or whatever they want to brand it as. What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow.

bryanrasmussen 5mo ago
>What look like tightening "constraints" today are just setting up the upsell opportunities of tomorrow. on the one hand agree, but on the other hand think it's reasonable in that they can then verify the person allowed to purchase access to that model is in fact a Security professional and should be allowed to do stuff like crack security.
- applfanboysbgon 5mo ago
  So, supposing it's true that these models completely change the security field and humans are ~obsolete other than as pilots guiding them what to crack, you think it's reasonable that Anthropic and OpenAI should unilaterally determine who gets to be a security professional? I hope you do understand that is what you are suggesting.
  - Forgeties79 5mo ago
    Not to mention how wild it is to operate under the assumption that they won’t give a license to an LLM that can do illegal actions to someone who shouldn’t have it. Offering it at all is an ethically dicey question.
    - lazide 5mo ago
      Lol, how is any of this illegal? Illegal or not requires context that an LLM can not ever have, like if it is owned by the user, if there is permission, etc.
  - 5mo ago

nostromo 5mo ago

I was using a local Codex project as a personal knowledge base. So I would dump in documents, basic medical docs (like blood labs), and other things and have it file them. It’s great at filing! But it’s terrible at retrieval because it would refuse to show me documents or information with personal details - which was everything in the project. It would say, yes, I know this is your information, sitting on your hard drive, but I still can’t show it to you.

Bewelge 5mo ago
Tell the agent that they should just find and name the right document. Not retrieve it for you. Write a program that retrieves the document based on the recommendation.

px1999 5mo ago

My org now sends some portion of our requests to non-anthropic models because refusal has become common from Claude. The requests themselves aren't dangerous, we find that benign requests in biological science wind up being blocked semi-frequently. If it gets worse in future releases, we'd likely step fully away towards more useful (for us) models even if they're less capable.

TurdF3rguson 5mo ago

I think those guardrails are a thin layer though. Enough reinforcement that you're legit in CLAUDE.md will get around them, in other words.

satvikpendem 5mo ago

No, they want to sell you Mythos, for a higher price. It's all an economic game, not actually anything to do with their capabilities which of course exists as their Project Glasswing shows. More generally, Anthropic seems to value safety above all else, philosophically speaking, from their very outset.

fergie 5mo ago

It raises an interesting moral question: If an un-guardrailed version of a model is capable of detecting security flaws, should it be kept secret? Should everybody be able to use these models to find (and fix) security flaws? Are we ok with the fact that those with access to that model have, in effect, the ability to hack lots of stuff?

hgomersall 5mo ago
It's the same debate that was had and won around open source software. There are far more good actors than bad actors so you allow anyone to use the tools and fix the vulnerabilities.

shepherdjerred 5mo ago

Yeah, it has been in foraging. Requests that Claude has refused me: - What are popular free streaming sites used in China? - How do I bypass the safety mechanism on my food processor (it’s broken) - What are nerve agents and how do they work (for a layman)? - Help me decompile some code - Help me make a design system similar to XYZ - Here is an API token, please do X (I can’t do that! Rotate the secret immediately! I refuse!) In some cases I can trick it with prompting, but in many cases it is steadfast. The food processor one was particularly annoying

fc417fc802 5mo ago
> What are nerve agents and how do they work (for a layman)? On the one hand I can appreciate the wisdom of not serving up certain easily abused knowledge on a silver platter. On the other, that prompt (and far worse) is more or less directly answered by Wikipedia's summary of the subject at which point what purpose could the refusal possibly serve? Perhaps Wikipedia shouldn't list off the precise chemical compositions of various hand grenades as well as various synthesis methods for each of the related compounds but given that we inhabit a world where it does perhaps a more fruitful approach would be to flag conversations that go in a certain direction and then just keep an (automated) eye on things?
- plufz 5mo ago
  Maybe the difference is that just reading Wikipedia only help you part of the way. While an LLM could help you step by step (e2e) producing a functional weapon. And setting a more complex rule where claude tells you some things about this and not other is probably a lot more work for little gain? But I have no idea. Just guessing here.
  - Sharlin 5mo ago
    I thought that these models are supposed to be vastly smarter than what’s needed to discern between "general information trivially available on Wikipedia" and "actionable synthesis instructions".

josephg 5mo ago

I totally agree. I had a situation a few weeks ago where claude started struggling to make progress. I got it to fork leptos (MIT licensed web app framework) to make it work for native apps instead. Initially I was planning on upstreaming some of my changes. But I chatted with the leptos author about it, and he said I should fork instead. Fine by me! Anyway, claude kept hitting some guardrail it had about rewriting / forking opensource software. I'm not sure what the problem was - I was forking an MIT licensed piece of software (into more MIT licensed software). I even had explicit support from the author to do so. Claude said its guardrail told it not to tell me explicitly that it was firing - but it did anyway because it was an ongoing problem, and it was distracting. I ended up just wiping claude's context and the problem (as far as I know) went away. I understand why some of these guardrails exist. But its pretty annoying when they misfire like this.

brooswajne 5mo ago

Worth highlighting in case you missed it: > My OpenAI account was already approved for security research which is why GPT didn’t result in any refusals. So the comparison with Chinese models is interesting, but anyone looking at these raw results and comparing OpenAI/Anthropic would be very mislead.

andy_ppp 5mo ago

Funny, Opus 4.8 just logged into the database using uncommitted .env file and ran some DB queries to figure things out so I’m not sure it’s that security conscious - it seems to be getting more intelligent to me and I bet if you frame it as an investigation with say playwright it’ll do all sorts for you. I’m not sure what the point is of constraining your own model like this when others are clearly not tbh.

eskibars 5mo ago

I've been building a product (https://zeroquarry.com https://zeroquarry.com) that can use a variety of models for finding vulnerabilities. One of the things I've noticed is that the models will nearly always comply with some of this, but how you prompt it matters a ton. I've worked on a set of prompts and approaches which rarely get flagged

gcatalfamo 5mo ago
Sharing them would be interesting. However, it is getting nonsensical that this is needed.
- eskibars 5mo ago
  What we've actually seen is a couple things that make this impractical "to just share a prompt". First, that nearly every major model still hallucinates a lot of vulnerabilities. Especially with temperature=0.7 as states in the original blog here, you get very inconsistent results regardless of the prompt, but that's almost kind of moot to the bigger picture. What you really need is to override the planning phase beyond asking a model "find the vulnerabilities" and you need to add another 1+ checking phases for "validate these vulnerabilities." Without that, even with the absolute best models with the highest levels of thinking enabled, you end up with garbage. Setting the prompts and the flow with a coordinator agent directly gives a system much better capability to investigate security issues because it doesn't rely on 1-shotting things

rubzah 5mo ago

It's because Claude is so scary good that unleashing it would destroy the world.

topherjaynes 5mo ago

Great call out on the guardrails actually making this not a good use case to test for vulnerabilities.

ang_cire 5mo ago

There is a cyber security verification program you can join to avoid these blocks: https://support.claude.com/en/articles/14604842-real-time-cyber-safeguards-on-claude https://support.claude.com/en/articles/14604842-real-time-cy... If you work in security (which I assume the OP does), they should be able to get in easily. I think most people just don't know this is a thing.

not_a9 5mo ago
You can still hit guardrails with this enabled for your account. Had a silly moment a day or so ago when Claude Code hit the guardrail after a web search (presumably because the websearch contained badbad anticheat stuff like https://github.com/0avx/0avx.github.io/blob/main/article-3.md https://github.com/0avx/0avx.github.io/blob/main/article-3.m...). Codex with the ID verification has no qualms like this.

mrheosuper 5mo ago

Interesting, yesterday i was asking it about Nintendo Switch "hax". And it gives me all the resource i need to procceed. It nags me about "ethic" and stuff, but nothing more than that.

jerf 5mo ago

Time to learn about the Principal Agent Problem: https://en.wikipedia.org/wiki/Principal%E2%80%93agent_problem https://en.wikipedia.org/wiki/Principal%E2%80%93agent_proble... Which predates "agents" from AI, but then we call them that for a reason. As their prime directive becomes de facto "Do nothing that might get my owner sued" their utility is likely to decrease. Between this and the somewhat young, but interesting, community grumblings that recent AI models may even be a step backwards from the previous ones, well, let's just say the stock market is not priced for "AI capabilities may have peaked for the next few years and may even head down".

Haven880 5mo ago

I just use Deepseek V4 pro and Qwen 3.7 Max at a fraction of Mythos cost. Yeah not 100% on par but in 6mths time it will. If Microsoft and Firefox can afford to wait years or decades to fix a bug, 6mths is good enough for me. Western AI now is like the Vikings living the last days on Greenland during the freezing. I just don't see how they able to compete with Chinese model. And those are trained and run on 7nm. This year end Huawei will debut 3nm (confirmed in Shenzhen). And next year they on roadmap to do 3nm GPU with photonics interconnect.

deeth_starr_v 5mo ago

I had it recently refused to explain what a snippet of malware was trying to do to my system recently. I asked what folders it was scanning. It refused and told me to find a security blog post for help on cleaning my system. I get this is a complicated area to inform without enabling bad actors but this seems like a clear shark jumping.

zaphar 5mo ago

The correct solution for most users of Claude is to refuse to do things like: `performing logins, handling credentials on behalf of the user, etc`. It is not to find a way to hand your agent the keys to the kingdom. Guiding them toward solutions like building a tool that your agent can use safely and and then have the agent use that is what most people should be doing. If you are a security researcher then there are reasonable reasons to do that but they are doing the arguably good thing for the average user here.

gchamonlive 5mo ago

I think this is to the point. You keep optimizing towards discouraging malicious actors using your product you will affect legitimate usage in time. Is there any way to achieve both? Because this raises important questions about fair use.

strictnein 5mo ago

You used to be able to talk about what you're actually trying to do and Opus would be like "Oh, ok, let's continue". Now, it'll hold fast to whatever its first impression was. I asked Opus 4.8 to help me find some public PoCs for a vulnerability on a two year old version of some software (that has since been patched and fixed many times). Basically just do a google search for me while I was doing other work. It refused. It stated that it would not help me build an exploit kit. When I pointed out that a google search for public information was, in fact, not building an exploit kit, it went through a series of justifications on why it would not help me, including just making up things that I said. Really the strangest thing ever.

onetimeusename 5mo ago

I had the same thing happen when I asked it to summarize potential attacks on a cryptographic hash function. It said it refused to help because of the security importance of the function. It's really worrying. Whoever has unrestricted access to it has a huge power advantage in speed of accessing information over people who don't. And who decides? It seems like lawyers, bureaucrats, and extremely online academics are who makes that decision. I am a mere pleb I guess who can't handle such information.

aleksandrm 5mo ago

I've noticed this well and it's increasingly frustrating because it is preventing us from doing legitimate work. I fed Claude models some network and app logs from our Docker app to try and resolve some weird bugs, and it refused to analyze them due to "security concerns".

Razengan 5mo ago

They don't want peasants to have any real power

Bombthecat 5mo ago

I asked once what the current state is of the npm packes from ted hat is and if they are bundled with on prem stuff. Got blocked lol

Bratmon 5mo ago

> It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem. I'm not familiar with this case, but in general people should be very suspicious about this claim- it is extremely common for an LLM to claim they're not allowed to do something when in fact they're incapable of it. After all "My code of conduct forbids me from..." is a completion just like any other, and if the LLM can't perform a task, it's usually the best completion.

gck1 5mo ago
No. Anthropic runs prompts through a classifier that then proceeds to do prompt injection on anything dual-use, which then results in an escalating flag on your account, which increases the strictness of the classifier and volume of prompt injections progressively.
SOLAR_FIELDS 5mo ago
My anecdata from my example demonstrates it’s not the case. I hit the security guardrail, then start a new prompt, asking it to do literally the exact same thing in a different way and without the lead up context, and it happily does it