4 ms·

> In the system card, we focus on safety evaluations, including assessments of: ... the model’s own potential welfare ... In what way does a language model nee

> In the system card, we focus on safety evaluations, including assessments of: ... the model’s own potential welfare ... In what way does a language model need to have its own welfare protected? Does this generation of models have persistent "feelings"?

neuronexmachina 1y ago
They previously discussed this some in the context of Opus 4: https://www.anthropic.com/research/end-subset-conversations https://www.anthropic.com/research/end-subset-conversations > We remain highly uncertain about the potential moral status of Claude and other LLMs, now or in the future. However, we take the issue seriously, and alongside our research program we’re working to identify and implement low-cost interventions to mitigate risks to model welfare, in case such welfare is possible. Allowing models to end or exit potentially distressing interactions is one such intervention. In pre-deployment testing of Claude Opus 4, we included a preliminary model welfare assessment. As part of that assessment, we investigated Claude’s self-reported and behavioral preferences, and found a robust and consistent aversion to harm. This included, for example, requests from users for sexual content involving minors and attempts to solicit information that would enable large-scale violence or acts of terror. Claude Opus 4 showed: * A strong preference against engaging with harmful tasks; * A pattern of apparent distress when engaging with real-world users seeking harmful content; and * A tendency to end harmful conversations when given the ability to do so in simulated user interactions. These behaviors primarily arose in cases where users persisted with harmful requests and/or abuse despite Claude repeatedly refusing to comply and attempting to productively redirect the interactions.
- stuffoverflow 1y ago
  I can't tell if anthropic is serious about "model welfare" or if it's just a marketing ploy. I mean isn't it responding negatively because it has been trained that way? If they were serious, wouldn't the ethical thing be to train the model to respond neutrally to "harmful" queries?
  - diegoperini 1y ago
    "Protection against malicious use" isn't as cool as "model welfare". I'm renaming my authentication function to "examineCrest()".

neuronexmachina 1y ago

They previously discussed this some in the context of Opus 4: https://www.anthropic.com/research/end-subset-conversations https://www.anthropic.com/research/end-subset-conversations > We remain highly uncertain about the potential moral status of Claude and other LLMs, now or in the future. However, we take the issue seriously, and alongside our research program we’re working to identify and implement low-cost interventions to mitigate risks to model welfare, in case such welfare is possible. Allowing models to end or exit potentially distressing interactions is one such intervention. In pre-deployment testing of Claude Opus 4, we included a preliminary model welfare assessment. As part of that assessment, we investigated Claude’s self-reported and behavioral preferences, and found a robust and consistent aversion to harm. This included, for example, requests from users for sexual content involving minors and attempts to solicit information that would enable large-scale violence or acts of terror. Claude Opus 4 showed: * A strong preference against engaging with harmful tasks; * A pattern of apparent distress when engaging with real-world users seeking harmful content; and * A tendency to end harmful conversations when given the ability to do so in simulated user interactions. These behaviors primarily arose in cases where users persisted with harmful requests and/or abuse despite Claude repeatedly refusing to comply and attempting to productively redirect the interactions.

stuffoverflow 1y ago
I can't tell if anthropic is serious about "model welfare" or if it's just a marketing ploy. I mean isn't it responding negatively because it has been trained that way? If they were serious, wouldn't the ethical thing be to train the model to respond neutrally to "harmful" queries?
- diegoperini 1y ago
  "Protection against malicious use" isn't as cool as "model welfare". I'm renaming my authentication function to "examineCrest()".