4 ms·

Anthropic walks back policy that could have 'sabotaged' researchers using Claude

71 points by ericflo 4mo ago

https://archive.ph/yxYhU https://archive.ph/yxYhU https://web.archive.org/web/20260611033414/https://www.wired.com/story/anthropic-responds-to-backlash-on-claudes-secret-sabotage-on-ai-research/ https://web.archive.org/web/20260611033414/https://www.wired...

impulser_ 4mo ago
It's kinda weird to think the Chinese AI labs might be more trust worthy than the US labs. - Anthropic is ran by a bunch of nut jobs. - OpenAI is ran by a guy you can't trust. I don't even know if we should include DeepMind, Meta, or xAi in the conversation of AI labs at this point since they can't produce models better than Chinese labs.
- reasonableklout 4mo ago
  To be fair, nerfing Claude on frontier research tasks is consistent with Anthropic's stated beliefs. So in that sense you can trust them to always behave consistently if strangely. But this launch was done very poorly with the lack of transparency on when the frontier research policy was violated.
  - impulser_ 4mo ago
    Yeah and their belief are fucking crazy and dangerous. They are literally sabotaging their users. They built in malware into their model if you prompt it about training a fucking AI model. It doesn't tell you, no it literally sabotages you by editing your prompt and intentionally goes against your request. You want fucking nut jobs like this building models? It's one thing to build safeguards on your model and have it prompt the user back. I'm sorry I can't help you with this request. Chinese models do this for some requests. It's another thing to actively try to make the model perform worst for your user on purpose because it asked the model to do something you, the model creator, didn't like. Imagine someone is asking a logical medical question and the model swaps the prompt and purpose being less intelligent and gives bad advice to this person. How do these people not understand they are stupid.
    - noduerme 4mo ago
      Is it really crazy to nerf a proprietary model to prevent it from training another model? I don't think that's even remotely similar to giving bad medical advice.
      - Balinares 4mo ago
        Given that the "proprietary" model is built on stolen work at an unprecedented scale, it's at the very least hypocritical to a degree that would not be possible without a fundamentally amoral mindset.
      - preg_match 4mo ago
        It’s not a nerf, it’s sabotage. That’s different. This is like if you’re driving a car and it detects your pulling up to a competing dealership so it cuts the brakes. This is, in my mind, effectively malware. We don’t know exactly what code the model will inject, and we certainly don’t know when it will happen. It could very easily introduce vulnerabilities.
    - skeledrew 4mo ago
      > You want fucking nut jobs like this building models? It takes *nut jobs" to advance tech like this at the speed it is. They have strong beliefs and they work hard to realize those beliefs.
      - DarioAmogey 4mo ago
        [dead]
- redox99 4mo ago
  Deepmind is definitely frontier. They just don't care that much about code.
- piyuv 4mo ago
  The thing with Chinese AI labs is that you don’t need to trust them. They publish the models, you can run them on-prem or rent a beefy VPS.
rvz 4mo ago
Too late. We now all know that Anthropic CAN do that if they want to. The fact that they told you upfront about it shows that their arrogance on this self-sabotage against their customers is at stratospheric levels. Believe them the first time, and they are not your friends at all.
nmfisher 4mo ago
Call me a cynic, but I don't believe this is a genuine change of heart at all. It feels much more like a panicked response to something that might undermine their IPO. Even if you trust Anthropic today (which I don't), they clearly don't want competition and there's no telling what other shady moves they'll pull in future. The only sustainable way forward is to support open models. I was already on the fence about whether or not to keep my Max subscription (the extra cost over something like DeepSeek V4 didn't really feel justifiable). This is the tipping point for me, I'll be cancelling my sub before it renews at the end of the month.
- LoganDark 4mo ago
  I think they are legitimately convinced that this model is so dangerous it could destroy the world and that they genuinely have the responsibility to prevent it from assisting other models to destroy the world. I don't think I agree that I should be forbidden from e.g. patching a binary to work on the latest macOS since the company behind it died and intentionally installed a time-based kill-switch (FUCK ADOBE for popularizing that practice). But ooOOooOOoo working with machine code is so cybersecurity and therefore suspicious.
  - chatmasta 4mo ago
    The company was founded basically out of the effective altruism movement.
    - LoganDark 4mo ago
      What is the impact of effective altruism? I looked it up, but I don't understand how it differs from simple logical consideration, i.e. how it would be responsible for any of Anthropic's eccentricities.
      - chatmasta 4mo ago
        It’s logical consideration with “logical” meaning Spock style logic, ie utilitarianism at all costs. Another prominent EA is SBF for example. It’s designed to sound innocuous and many of its cultish promoters may genuinely believe it’s innocuous, but it’s not.
LoganDark 4mo ago
https://archive.ph/yxYhU https://archive.ph/yxYhU
SilverElfin 4mo ago
The damage is done. They designed Fable to be dishonest and sabotage-y. Look at this report of Claude now randomly changing AI software without being asked to: https://xcancel.com/hammer_mt/status/2064839924398825798 https://xcancel.com/hammer_mt/status/2064839924398825798 This is so completely dishonest. But it also shows how deeply anti competitive Anthropic is. They will talk about safety but it’s not actually about safety: building features like this seems intended to hurt competition in the AI space. They don’t mind if AI helps YOUR competitor but if it means competition for them, they suddenly have a problem with it. I don’t care that they walked this back. They’ve shown who they are. And what they’re capable of.
dtj1123 4mo ago
Great, now just let me use it for bioinformatics and we're good.
maniacwhat 4mo ago
One takeaway from this is that anthropic do not want their public-facing model to be good at advanced ML research. With that incentive it seems reasonable to assume that future anthropic models will not be good at ML research, at least because they aren't incentivised to make it so, and therefore using other models perhaps open source will be the way to go. This does make the large assumption that they can afford to train a parallel model for themselves to assist in their own research. But given their huge valuation, and incentives, that extra cost is feasibly worth it for them.
nomorehere 4mo ago
I think I've seen this model used by major academic publishers before.: https://files.catbox.moe/mrh1ll.jpg https://files.catbox.moe/mrh1ll.jpg
DarioAmogey 4mo ago
[dead]

71 points by ericflo 4mo ago

impulser_ 4mo ago

It's kinda weird to think the Chinese AI labs might be more trust worthy than the US labs. - Anthropic is ran by a bunch of nut jobs. - OpenAI is ran by a guy you can't trust. I don't even know if we should include DeepMind, Meta, or xAi in the conversation of AI labs at this point since they can't produce models better than Chinese labs.

reasonableklout 4mo ago
To be fair, nerfing Claude on frontier research tasks is consistent with Anthropic's stated beliefs. So in that sense you can trust them to always behave consistently if strangely. But this launch was done very poorly with the lack of transparency on when the frontier research policy was violated.
- impulser_ 4mo ago
  Yeah and their belief are fucking crazy and dangerous. They are literally sabotaging their users. They built in malware into their model if you prompt it about training a fucking AI model. It doesn't tell you, no it literally sabotages you by editing your prompt and intentionally goes against your request. You want fucking nut jobs like this building models? It's one thing to build safeguards on your model and have it prompt the user back. I'm sorry I can't help you with this request. Chinese models do this for some requests. It's another thing to actively try to make the model perform worst for your user on purpose because it asked the model to do something you, the model creator, didn't like. Imagine someone is asking a logical medical question and the model swaps the prompt and purpose being less intelligent and gives bad advice to this person. How do these people not understand they are stupid.
  - noduerme 4mo ago
    Is it really crazy to nerf a proprietary model to prevent it from training another model? I don't think that's even remotely similar to giving bad medical advice.
    - Balinares 4mo ago
      Given that the "proprietary" model is built on stolen work at an unprecedented scale, it's at the very least hypocritical to a degree that would not be possible without a fundamentally amoral mindset.
    - preg_match 4mo ago
      It’s not a nerf, it’s sabotage. That’s different. This is like if you’re driving a car and it detects your pulling up to a competing dealership so it cuts the brakes. This is, in my mind, effectively malware. We don’t know exactly what code the model will inject, and we certainly don’t know when it will happen. It could very easily introduce vulnerabilities.
  - skeledrew 4mo ago
    > You want fucking nut jobs like this building models? It takes *nut jobs" to advance tech like this at the speed it is. They have strong beliefs and they work hard to realize those beliefs.
    - DarioAmogey 4mo ago
      [dead]
redox99 4mo ago
Deepmind is definitely frontier. They just don't care that much about code.
piyuv 4mo ago
The thing with Chinese AI labs is that you don’t need to trust them. They publish the models, you can run them on-prem or rent a beefy VPS.

rvz 4mo ago

Too late. We now all know that Anthropic CAN do that if they want to. The fact that they told you upfront about it shows that their arrogance on this self-sabotage against their customers is at stratospheric levels. Believe them the first time, and they are not your friends at all.

nmfisher 4mo ago

Call me a cynic, but I don't believe this is a genuine change of heart at all. It feels much more like a panicked response to something that might undermine their IPO. Even if you trust Anthropic today (which I don't), they clearly don't want competition and there's no telling what other shady moves they'll pull in future. The only sustainable way forward is to support open models. I was already on the fence about whether or not to keep my Max subscription (the extra cost over something like DeepSeek V4 didn't really feel justifiable). This is the tipping point for me, I'll be cancelling my sub before it renews at the end of the month.

LoganDark 4mo ago
I think they are legitimately convinced that this model is so dangerous it could destroy the world and that they genuinely have the responsibility to prevent it from assisting other models to destroy the world. I don't think I agree that I should be forbidden from e.g. patching a binary to work on the latest macOS since the company behind it died and intentionally installed a time-based kill-switch (FUCK ADOBE for popularizing that practice). But ooOOooOOoo working with machine code is so cybersecurity and therefore suspicious.
- chatmasta 4mo ago
  The company was founded basically out of the effective altruism movement.
  - LoganDark 4mo ago
    What is the impact of effective altruism? I looked it up, but I don't understand how it differs from simple logical consideration, i.e. how it would be responsible for any of Anthropic's eccentricities.
    - chatmasta 4mo ago
      It’s logical consideration with “logical” meaning Spock style logic, ie utilitarianism at all costs. Another prominent EA is SBF for example. It’s designed to sound innocuous and many of its cultish promoters may genuinely believe it’s innocuous, but it’s not.

LoganDark 4mo ago

https://archive.ph/yxYhU https://archive.ph/yxYhU

SilverElfin 4mo ago

The damage is done. They designed Fable to be dishonest and sabotage-y. Look at this report of Claude now randomly changing AI software without being asked to: https://xcancel.com/hammer_mt/status/2064839924398825798 https://xcancel.com/hammer_mt/status/2064839924398825798 This is so completely dishonest. But it also shows how deeply anti competitive Anthropic is. They will talk about safety but it’s not actually about safety: building features like this seems intended to hurt competition in the AI space. They don’t mind if AI helps YOUR competitor but if it means competition for them, they suddenly have a problem with it. I don’t care that they walked this back. They’ve shown who they are. And what they’re capable of.

dtj1123 4mo ago

Great, now just let me use it for bioinformatics and we're good.

maniacwhat 4mo ago

One takeaway from this is that anthropic do not want their public-facing model to be good at advanced ML research. With that incentive it seems reasonable to assume that future anthropic models will not be good at ML research, at least because they aren't incentivised to make it so, and therefore using other models perhaps open source will be the way to go. This does make the large assumption that they can afford to train a parallel model for themselves to assist in their own research. But given their huge valuation, and incentives, that extra cost is feasibly worth it for them.

nomorehere 4mo ago

I think I've seen this model used by major academic publishers before.: https://files.catbox.moe/mrh1ll.jpg https://files.catbox.moe/mrh1ll.jpg

DarioAmogey 4mo ago

[dead]