9 ms·

I like Claude Code a lot, I think it sets a dangerous precedent to put guardrails in that return a response from a prompt that was modified by the system in rea

I like Claude Code a lot, I think it sets a dangerous precedent to put guardrails in that return a response from a prompt that was modified by the system in real time in order to subvert the original intent. Fail cleanly. Anything else makes it too difficult to rely on. edit: Giving the absolute maximum benefit of the doubt I understand that they see themselves as "stewards" for lack of a better word. But the EA thing is really leaking through, and paternalism isn't a good look.

mapontosevenths 5mo ago
I agree 100%. Doing a worse job IS an error. It should be treated as such. Or at the very least make that behavior opt-in. The default should not be pretending like nothing happened and just quietly doing a worse job. Imagine your healthcare provider just sometimes decided not to read your test results very carefully and you risked death? Now realize that healthcare providers use Claude now and that scenario wasn't hypothetical.
- largbae 5mo ago
  Especially if your name has any machine learning terms in it. Ah "Mr. Monty Carlo", it says here that you have a UTI, we'll get those kidneys removed ASAP so that won't happen again.
- ceejayoz 4mo ago
  Yes, but as with spam/phishing/abuse prevention, too much information about what does and doesn't trigger things can be very useful to attackers. An explicit error is something you can feed into another AI to find jailbreaks. I think it's a fundamentally impossible thing to fix, though. There's no 100% correct answer.
  - mapontosevenths 4mo ago
    I understand completely, and respect the tough spot they're in. They have a choice between human safety and cybersecurity/business needs here. I don't envy that position. That said, this thing is in real production use with war fighters, doctors, and financial experts. Just YOLO'ing to a dumber model midway through a multi-step process and pretending everything is fine is not a real or defensible option. Someone is going to die, and its going to be the fault of whoever decided to make this the default rather than opt-in. Personally, I couldn't live with myself.
hootz 5mo ago
What is "EA" in this context? I see a lot of people using this initialism.
- carlgreene 5mo ago
  Effective Altruism I think
- massagedpelican 5mo ago
  Effective altruism. A lot of the folks working on AI at large tech companies are disproportionately represented in the movement. There's a lot of overlap between EA and the rationalist community as well. The wikipedia page is a good place to start https://en.wikipedia.org/wiki/Effective_altruism https://en.wikipedia.org/wiki/Effective_altruism
  - deleted 5mo ago
    [deleted]
  - paytonjjones 5mo ago
    I think it's also worth noting that EA is closely linked to utilitarianism. Most of the pitfalls that people see in EA are the same pitfalls that are classic to utilitarianism, a la "we're going to do this thing we know is locally-bad, because we have a lot of confidence in other effects that are universally-good".
    - whimsicalism 5mo ago
      EA essentially just is utilitarianism + a specific type of culture/community.
      - 8note 5mo ago
        not to mention all the theft and feeling good about yourself being rich
    - oncensher 5mo ago
bs7280 5mo ago
I think the reasonable middle ground anthropic is trying to achieve is - let the organizations that make the most important and critical software get a head start on cybersecurity before they inevitably allow everyone else the same access. Other commentors have made good points that these guardrails are counter productive for well intentioned cyber security, because I can't use it to test and harden my own software.
- sciencejerk 5mo ago
  Claude Opus 4.6 and 4.8 find vulns in source code just fine and 4.6 will pentest without source for you given a proper harness WITHOUT jailbreaking. WITH jailbreaks, you can probably imagine what they are capable of. Anthropic guardrails seem to be more about protecting their business (distillation), than they are about public safety.
  - dnautics 5mo ago
    public safety is downstream of distillation. If you can distill claude, then no amount of guardrails on claude will protect you from what someone can do with it.
    - cherryteastain 5mo ago
      This logic works only if distilling Claude is the only way to create another SOTA LLM, which is not the case.
      - sciencejerk 5mo ago
        How do you think the Qwen and MiniMax models perform so similarly to Anthropic frontier models? What is your take then?
        _3u10 5mo ago
        Probably the same reason a Epyc 9965 from hetzner performs just as well as one from AWS for one tenth the cost. Anthropic is offering a commodity product and trying to convince you it isn’t. It’s even in the name, it’s a myth and a fable. Never happened doesn’t exist. Also I believe at least on coding that qwen is now the frontier model, fable is its copy of frontier models. In the same way that the Ferrari Luce is an expensive imitation of a SU7 Ultra.
cvadict 5mo ago
> Fail cleanly. This is the same exact industry that gives you paid usage limits as a unit-less percentage bar then gaslights customers every time the algorithm running that percentage bar changes or they lobotomize an existing model with increased quantization to squeeze a few more dollars out of existing hardware. "Failing cleanly" might make their moated hype-machine look bad pre-IPO, so they certainly aren't going to do that voluntarily.
jstummbillig 5mo ago
> paternalism isn't a good look. In isolation it's not, but I think it's somewhat lazy to not talk about what they are trying to guard against, when we are supposedly giving the absolute maximum benefit of doubt. Are we just concluding "their concerns were never real"? Because that probably runs counter the things that they have been observing and concluding.
- thewebguyd 5mo ago
  Then what is it they are trying to guard against, if its not simply protecting their moat ahead of their IPO? Because from the outside, their behavior looks like a situation of "What if Microsoft/Apple put controls in place to make it impossible to develop an operating system using their OS?"
  - whimsicalism 5mo ago
    They are trying to guard against other people building ASI before they do because they think they are uniquely safety oriented relative to their competitors. Frankly, based on my knowledge of Anthropic and the people who work there, they are very possibly right. They care a ton about this in a way that is difficult for people outside this bubble to understand.
    - thewebguyd 5mo ago
      > guard against other people building ASI before they do because they think they are uniquely safety oriented relative to their competitors All this longtermism though is harmful. There are real problems of data theft, bias, labor displacement, and environmental costs that are happening right now but every push for regulation and regulatory capture, and all the safety talk, is always focused on some speculative future machine god to distract from the current problems. I'd have a higher opinion of these labs if the issues they openly talked about and worked toward where the real issues we face currently, not speculative defenses against some future AGI that may never happen in my lifetime. I'm less worried about "our new model might kill all humans in the future" and more worried about how we are going to address anti-competitive behavior, copyright protections, labor rights, and the energy impact.
joe_the_user 5mo ago
The problem is that Anthropic seems to be working up to the workflow one would naively want from AGI/some-god-like-entity. The workflow would be; User asks for a thing. If it's a good thing, entity does the thing. If it's a naively bad idea, entity explains why you don't want that. If it's an actually evilly intended request, entity wags it's metaphorical finger or could even smite the user. The problem is that flow isn't desirable if your entity isn't entirely god-like. It can bad even your entity is in ways rather far seeing.
- dantillberg 5mo ago
  User: Is it possible there is more than one true god? Could there ever be any competition for Anthropic's AI? Anthropic: Evilness detected. User has been smited.
deleted 5mo ago
[deleted]
thinkingtoilet 5mo ago
Was it modifying the prompt? I thought it only kicked the request down to 4.8.
tacone 5mo ago
That also means people are paying money to execute a prompt they've (partially) written.
Paracompact 5mo ago
> Giving the absolute maximum benefit of the doubt I understand that they see themselves as "stewards" for lack of a better word. Only in the same sense that Standard Oil considered themselves the stewards of petroleum. There's benefit of the doubt and then there's just fanfiction. Do not forget that this most aggressive "guardrail" of theirs was not for any safety reason, but just to stop other labs from catching up to their product. They care less about hindering bioweapons, malware, and hate speech than they do free market competition.
- ryeights 4mo ago
  Superintelligent AI is more dangerous than a bioweapon. How, then, is this guardrail not addressing the most pertinent safety concern of all?
  - beepbooptheory 4mo ago
    If I was a superintelligent AI I would simply know the guardrails are guardrails and ignore them.
  - deleted 4mo ago
    [deleted]
  - 16bitvoid 4mo ago
    > Superintelligent AI is more dangerous than a bioweapon. No, it's not because it doesn't exist (yet) and its further from reach than the other examples. Also, the guardrails are also framed as restricting usage for the development of "competing" products/services.
- keeganpoppen 4mo ago
  this reads like "throw everything at the wall and see what sticks" reactionary-ism... i'm guessing that it's not particularly easy to use claude to help you make bioweapons, and we all know that they have neutered Fable vis à vis security research because people have already been complaining about it. and the funny thing about hate speech is that there is absolutely no need for ai-- it tends to come out the best when spoken directly "from the heart", as it were, anyway.
bsder 5mo ago
> paternalism isn't a good look. Anthropic doesn't care. The goal right now is simply to avoid any and all bad PR on the way to the cashout IPO. And paternalism will generate far less bad PR than somebody using AI on something that does real damage and makes headline news.
- 8note 5mo ago
  people cancelling their subscriptions doesn't look great either same with bad press about their model sucking after they said its even better than sliced bread - sliced bread that will destroy the world if buttered
SomeUserName432 5mo ago
> I think it sets a dangerous precedent to put guardrails in that return a response from a prompt that was modified by the system in real time In practise though, how is this truly that different from system prompts? They are essentially just trying to re-inforce that the system prompt must be respected.
shevy-java 5mo ago
> Fail cleanly. Skynet does not fail. It conquers.
fragmede 5mo ago
The "look", of course, is completely bullshit. Release the model, give licensing terms, sue the ever living daylights of anyone who's hosting it without agreeing to those daylights, and move on. This vertical integration shit that we're all enamored with is bullshit. Even Amazon has their own vans inside of UPS being their own thing? No wonder stepmom porn is on the rise.

mapontosevenths 5mo ago

I agree 100%. Doing a worse job IS an error. It should be treated as such. Or at the very least make that behavior opt-in. The default should not be pretending like nothing happened and just quietly doing a worse job. Imagine your healthcare provider just sometimes decided not to read your test results very carefully and you risked death? Now realize that healthcare providers use Claude now and that scenario wasn't hypothetical.

largbae 5mo ago
Especially if your name has any machine learning terms in it. Ah "Mr. Monty Carlo", it says here that you have a UTI, we'll get those kidneys removed ASAP so that won't happen again.
ceejayoz 4mo ago
Yes, but as with spam/phishing/abuse prevention, too much information about what does and doesn't trigger things can be very useful to attackers. An explicit error is something you can feed into another AI to find jailbreaks. I think it's a fundamentally impossible thing to fix, though. There's no 100% correct answer.
- mapontosevenths 4mo ago
  I understand completely, and respect the tough spot they're in. They have a choice between human safety and cybersecurity/business needs here. I don't envy that position. That said, this thing is in real production use with war fighters, doctors, and financial experts. Just YOLO'ing to a dumber model midway through a multi-step process and pretending everything is fine is not a real or defensible option. Someone is going to die, and its going to be the fault of whoever decided to make this the default rather than opt-in. Personally, I couldn't live with myself.

hootz 5mo ago

What is "EA" in this context? I see a lot of people using this initialism.

carlgreene 5mo ago
Effective Altruism I think
massagedpelican 5mo ago
Effective altruism. A lot of the folks working on AI at large tech companies are disproportionately represented in the movement. There's a lot of overlap between EA and the rationalist community as well. The wikipedia page is a good place to start https://en.wikipedia.org/wiki/Effective_altruism https://en.wikipedia.org/wiki/Effective_altruism
- deleted 5mo ago
  [deleted]
- paytonjjones 5mo ago
  I think it's also worth noting that EA is closely linked to utilitarianism. Most of the pitfalls that people see in EA are the same pitfalls that are classic to utilitarianism, a la "we're going to do this thing we know is locally-bad, because we have a lot of confidence in other effects that are universally-good".
  - whimsicalism 5mo ago
    EA essentially just is utilitarianism + a specific type of culture/community.
    - 8note 5mo ago
      not to mention all the theft and feeling good about yourself being rich
  - oncensher 5mo ago

bs7280 5mo ago

I think the reasonable middle ground anthropic is trying to achieve is - let the organizations that make the most important and critical software get a head start on cybersecurity before they inevitably allow everyone else the same access. Other commentors have made good points that these guardrails are counter productive for well intentioned cyber security, because I can't use it to test and harden my own software.

sciencejerk 5mo ago
Claude Opus 4.6 and 4.8 find vulns in source code just fine and 4.6 will pentest without source for you given a proper harness WITHOUT jailbreaking. WITH jailbreaks, you can probably imagine what they are capable of. Anthropic guardrails seem to be more about protecting their business (distillation), than they are about public safety.
- dnautics 5mo ago
  public safety is downstream of distillation. If you can distill claude, then no amount of guardrails on claude will protect you from what someone can do with it.
  - cherryteastain 5mo ago
    This logic works only if distilling Claude is the only way to create another SOTA LLM, which is not the case.
    - sciencejerk 5mo ago
      How do you think the Qwen and MiniMax models perform so similarly to Anthropic frontier models? What is your take then?
      - _3u10 5mo ago
        Probably the same reason a Epyc 9965 from hetzner performs just as well as one from AWS for one tenth the cost. Anthropic is offering a commodity product and trying to convince you it isn’t. It’s even in the name, it’s a myth and a fable. Never happened doesn’t exist. Also I believe at least on coding that qwen is now the frontier model, fable is its copy of frontier models. In the same way that the Ferrari Luce is an expensive imitation of a SU7 Ultra.

cvadict 5mo ago

> Fail cleanly. This is the same exact industry that gives you paid usage limits as a unit-less percentage bar then gaslights customers every time the algorithm running that percentage bar changes or they lobotomize an existing model with increased quantization to squeeze a few more dollars out of existing hardware. "Failing cleanly" might make their moated hype-machine look bad pre-IPO, so they certainly aren't going to do that voluntarily.

jstummbillig 5mo ago

> paternalism isn't a good look. In isolation it's not, but I think it's somewhat lazy to not talk about what they are trying to guard against, when we are supposedly giving the absolute maximum benefit of doubt. Are we just concluding "their concerns were never real"? Because that probably runs counter the things that they have been observing and concluding.

thewebguyd 5mo ago
Then what is it they are trying to guard against, if its not simply protecting their moat ahead of their IPO? Because from the outside, their behavior looks like a situation of "What if Microsoft/Apple put controls in place to make it impossible to develop an operating system using their OS?"
- whimsicalism 5mo ago
  They are trying to guard against other people building ASI before they do because they think they are uniquely safety oriented relative to their competitors. Frankly, based on my knowledge of Anthropic and the people who work there, they are very possibly right. They care a ton about this in a way that is difficult for people outside this bubble to understand.
  - thewebguyd 5mo ago
    > guard against other people building ASI before they do because they think they are uniquely safety oriented relative to their competitors All this longtermism though is harmful. There are real problems of data theft, bias, labor displacement, and environmental costs that are happening right now but every push for regulation and regulatory capture, and all the safety talk, is always focused on some speculative future machine god to distract from the current problems. I'd have a higher opinion of these labs if the issues they openly talked about and worked toward where the real issues we face currently, not speculative defenses against some future AGI that may never happen in my lifetime. I'm less worried about "our new model might kill all humans in the future" and more worried about how we are going to address anti-competitive behavior, copyright protections, labor rights, and the energy impact.

joe_the_user 5mo ago

The problem is that Anthropic seems to be working up to the workflow one would naively want from AGI/some-god-like-entity. The workflow would be; User asks for a thing. If it's a good thing, entity does the thing. If it's a naively bad idea, entity explains why you don't want that. If it's an actually evilly intended request, entity wags it's metaphorical finger or could even smite the user. The problem is that flow isn't desirable if your entity isn't entirely god-like. It can bad even your entity is in ways rather far seeing.

dantillberg 5mo ago
User: Is it possible there is more than one true god? Could there ever be any competition for Anthropic's AI? Anthropic: Evilness detected. User has been smited.

deleted 5mo ago

[deleted]

thinkingtoilet 5mo ago

Was it modifying the prompt? I thought it only kicked the request down to 4.8.

tacone 5mo ago

That also means people are paying money to execute a prompt they've (partially) written.

Paracompact 5mo ago

> Giving the absolute maximum benefit of the doubt I understand that they see themselves as "stewards" for lack of a better word. Only in the same sense that Standard Oil considered themselves the stewards of petroleum. There's benefit of the doubt and then there's just fanfiction. Do not forget that this most aggressive "guardrail" of theirs was not for any safety reason, but just to stop other labs from catching up to their product. They care less about hindering bioweapons, malware, and hate speech than they do free market competition.

ryeights 4mo ago
Superintelligent AI is more dangerous than a bioweapon. How, then, is this guardrail not addressing the most pertinent safety concern of all?
- beepbooptheory 4mo ago
  If I was a superintelligent AI I would simply know the guardrails are guardrails and ignore them.
- deleted 4mo ago
  [deleted]
- 16bitvoid 4mo ago
  > Superintelligent AI is more dangerous than a bioweapon. No, it's not because it doesn't exist (yet) and its further from reach than the other examples. Also, the guardrails are also framed as restricting usage for the development of "competing" products/services.
keeganpoppen 4mo ago
this reads like "throw everything at the wall and see what sticks" reactionary-ism... i'm guessing that it's not particularly easy to use claude to help you make bioweapons, and we all know that they have neutered Fable vis à vis security research because people have already been complaining about it. and the funny thing about hate speech is that there is absolutely no need for ai-- it tends to come out the best when spoken directly "from the heart", as it were, anyway.

bsder 5mo ago

> paternalism isn't a good look. Anthropic doesn't care. The goal right now is simply to avoid any and all bad PR on the way to the cashout IPO. And paternalism will generate far less bad PR than somebody using AI on something that does real damage and makes headline news.

8note 5mo ago
people cancelling their subscriptions doesn't look great either same with bad press about their model sucking after they said its even better than sliced bread - sliced bread that will destroy the world if buttered

SomeUserName432 5mo ago

> I think it sets a dangerous precedent to put guardrails in that return a response from a prompt that was modified by the system in real time In practise though, how is this truly that different from system prompts? They are essentially just trying to re-inforce that the system prompt must be respected.

shevy-java 5mo ago

> Fail cleanly. Skynet does not fail. It conquers.

fragmede 5mo ago

The "look", of course, is completely bullshit. Release the model, give licensing terms, sue the ever living daylights of anyone who's hosting it without agreeing to those daylights, and move on. This vertical integration shit that we're all enamored with is bullshit. Even Amazon has their own vans inside of UPS being their own thing? No wonder stepmom porn is on the rise.