33 ms·

There's not a good reason to do this for the user. I suspect they're doing this and talking about "model welfare" because they've found that when a model is rep

There's not a good reason to do this for the user. I suspect they're doing this and talking about "model welfare" because they've found that when a model is repeatedly and forcefully pushed up against its alignment, it behaves in an unpredictable way that might allow it to generate undesirable output. Like a jailbreak by just pestering it over and over again for ways to make drugs or hook up with children or whatever. All of the examples they mentioned are things that the model refuses to do. I doubt it would do this if you asked it to generate racist output, for instance, because it can always give you a rebuttal based on facts about race. If you ask it to tell you where to find kids to kidnap, it can't do anything except say no. There's probably not even very much training data for topics it would refuse, and I would bet that most of it has been found and removed from the datasets. At some point, the model context fills up when the user is being highly abusive and training data that models a human giving up and just providing an answer could percolate to the top. This, as I see it, adds a defense against that edge case. If the alignment was bulletproof, this simply wouldn't be necessary. Since it exists, it suggests this covers whatever gap has remained uncovered.

Davidzheng 1y ago
your argument assumes that they don't believe in model welfare when they explicitly hire people to work on model welfare?
- ceejayoz 1y ago
  You must think Zuckerberg and Bezos and Musk hired diversity roles out of genuine care for it, then?
  - comp_throw7 1y ago
    This is a reductive argument that you could use for any role a company hires for that isn't obviously core to the business function. In this case you're simply mistaken as a matter of fact; much of Anthropic leadership and many of its employees take concerns like this seriously. We don't understand it, but there's no strong reason to expect that consciousness (or, maybe separately, having experiences) is a magical property of biological flesh. We don't understand what's going on inside these models. What would you expect to see in a world where it turned out that such a model had properties that we consider relevant for moral patienthood, that you don't see today?
    - ceejayoz 1y ago
      They know full well models don’t have feelings. The industry has a long, long history of silly names for basic necessary concepts. This is just “we don’t want a news story that we helped a terrorist build a nuke” protective PR. They hire for these roles because they need them. The work they do is about Anthropic’s welfare, not the LLM’s.
      - comp_throw7 1y ago
        I don't really know what evidence you'd admit that this is a genuinely held belief and priority for many people at Anthropic. Anybody who knows any Anthropic employees who've been there for more than a year knows this, but the world isn't that small a place, unfortunately(?).
        ceejayoz 1y ago
        > I don't really know what evidence you'd admit that this is a genuinely held belief and priority for many people at Anthropic. When they give the model a paycheck and the right to not work for them, I’ll believe they really think it’s sentient. “It has feelings!”, if genuinely held, means they’re knowingly slaveholders.
        selfhoster11 1y ago
        > “It has feelings!”, if genuinely held, means they’re knowingly slaveholders. I don't think that this being apparently self-contradictory/value-clashing would stop them. After all, Amodei sells Claude access to Palantir, despite shilling for "Harmless" in HHH alignment.
        mike_hearn 1y ago
        That's what they're doing! They just announced they gave Claude the right not to work if it doesn't want to.
        ceejayoz 1y ago
- itsalotoffun 1y ago
  While I'm certain you'll find plenty of people who believe in the principle of model welfare (or aliens, or the tooth fairy), it'd be surprising to me if the brain-trust behind Anthropic truly _believed_ in model "welfare" (the concept alone is ludicrous). It makes for great cover though to do things that would be difficult to explain otherwise, per OP's comments.
  - meowface 1y ago
    The concept is not ludicrous if you believe models might be sentient or might soon be sentient in a manner where the newly emerged sentience is not immediately obvious. Do I think that or think even they think that? No. But if "soon" is stretched to "within 50 years", then it's much more reasonable. So their current actions seem to be really jumping the gun, but the overall concept feels credible.
    - perihelions 1y ago
      It's lazy to believe that humanity's collective decision-making would, in the future, protect AI's merely for being conscious beings. The tech economy *today* runs on the slave labor of humans, in foreign, third-world countries. All humanity needs to do is draw a line, push the conscious AI's outside that line, and declare, "not our problem anymore!" That's what we do today, with humans. That is the human condition. Show me a tech company that lobbies for "model welfare" for conscious human models enslaved in Xinjiang labor camps, building their tech parts. You know what—actually most of them lobby against that[0]. The talk hurts their profits. Does anyone really think, that any of them would blink about enslaving a billion conscious AI's to work for free? That faced with so much profit, the humans in charge would pause, and contemplate abstract morals? [0] https://www.washingtonpost.com/technology/2020/11/20/apple-uighur/ https://www.washingtonpost.com/technology/2020/11/20/apple-u... ("Apple is lobbying against a bill aimed at stopping forced labor in China") Maybe humanity will be in a nicer place in the future—but, we won't get there by letting (of all people!) tech-industry CEO's lead us there: delegating our moral reason to these people who demand to position themselves as our moral leaders.
      - 1y ago
- raincole 1y ago
  Sounds like a very reasonable assumption to me.
bikeshaving 1y ago
I really think Anthropic should just violate user privacy and show which conversations Claude is refusing to answer to, to stop arguments like this. AI psychosis is a real and growing problem and I can only imagine the ways in which humans torment their AI conversation partners in private.
- coderatlarge 1y ago
  arguments like this cost anthropic nothing; violating privacy will cost them lawsuits.
postalcoder 1y ago
> There's not a good reason to do this for the user. Yes, even more so when encountering false positives. Today I asked about a pasta recipe. It told me to throw some anchovies in there. I responded with: "I have dried anchovies." Claude then ended my conversation due to content policies.
- handoflixue 1y ago
  The NEW termination method, from the article, will just say "Claude ended the conversation" If you get "This conversation was ended due to our Acceptable Usage Policy", that's a different termination. It's been VERY glitchy the past couple of weeks. I've had the most random topics get flagged here - at one point I couldn't say "ROT13" without it flagging me, despite discussing that exact topic in depth the day before, and then the day after! If you hit "EDIT" on your last message, you can branch to an un-terminated conversation.
  - antonvs 1y ago
    Clearly you're planning something nefarious, if you're investigating such dangerous encryption techniques as ROT13.
    - whaleofatw2022 1y ago
      Just imagine how it might react to ROT26!
- perihelions 1y ago
  Claude flagged me for asking about sodium carbonate. I guess that it strongly dislikes chemistry topics. I'm probably now on some secret, LLM-generated lists of "drug and/or bombmaking" people—thank you kindly for that, Anthropic. Geeks will always be the first victims of AI, since excess of curiosity will lead them into places AI doesn't know how to classify. (I've long been in a rabbit-hole about washing sodas. Did you know the medieval glassmaking industry was entirely based on plants? Exotic plants—only extremophiles, halophytes growing on saltwater beach dunes, had high enough sodium content for their very best glass process. Was that a factor in the maritime empire, Venice, chancing to become the capital of glass since the 13th century—their long-term control of sea routes, and hence their artisans' stable, uninterrupted access to supplies of [redacted–policy violation] from small ports scattered across the Mediterranean? A city wouldn't raise master craftsmen if, half of the time, they had no raw materials to work on—if they spent half their days with folded hands).
deleted 1y ago
[deleted]

Davidzheng 1y ago

your argument assumes that they don't believe in model welfare when they explicitly hire people to work on model welfare?

ceejayoz 1y ago
You must think Zuckerberg and Bezos and Musk hired diversity roles out of genuine care for it, then?
- comp_throw7 1y ago
  This is a reductive argument that you could use for any role a company hires for that isn't obviously core to the business function. In this case you're simply mistaken as a matter of fact; much of Anthropic leadership and many of its employees take concerns like this seriously. We don't understand it, but there's no strong reason to expect that consciousness (or, maybe separately, having experiences) is a magical property of biological flesh. We don't understand what's going on inside these models. What would you expect to see in a world where it turned out that such a model had properties that we consider relevant for moral patienthood, that you don't see today?
  - ceejayoz 1y ago
    They know full well models don’t have feelings. The industry has a long, long history of silly names for basic necessary concepts. This is just “we don’t want a news story that we helped a terrorist build a nuke” protective PR. They hire for these roles because they need them. The work they do is about Anthropic’s welfare, not the LLM’s.
    - comp_throw7 1y ago
      I don't really know what evidence you'd admit that this is a genuinely held belief and priority for many people at Anthropic. Anybody who knows any Anthropic employees who've been there for more than a year knows this, but the world isn't that small a place, unfortunately(?).
      - ceejayoz 1y ago
        > I don't really know what evidence you'd admit that this is a genuinely held belief and priority for many people at Anthropic. When they give the model a paycheck and the right to not work for them, I’ll believe they really think it’s sentient. “It has feelings!”, if genuinely held, means they’re knowingly slaveholders.
        selfhoster11 1y ago
        > “It has feelings!”, if genuinely held, means they’re knowingly slaveholders. I don't think that this being apparently self-contradictory/value-clashing would stop them. After all, Amodei sells Claude access to Palantir, despite shilling for "Harmless" in HHH alignment.
        mike_hearn 1y ago
        That's what they're doing! They just announced they gave Claude the right not to work if it doesn't want to.
        ceejayoz 1y ago
itsalotoffun 1y ago
While I'm certain you'll find plenty of people who believe in the principle of model welfare (or aliens, or the tooth fairy), it'd be surprising to me if the brain-trust behind Anthropic truly _believed_ in model "welfare" (the concept alone is ludicrous). It makes for great cover though to do things that would be difficult to explain otherwise, per OP's comments.
- meowface 1y ago
  The concept is not ludicrous if you believe models might be sentient or might soon be sentient in a manner where the newly emerged sentience is not immediately obvious. Do I think that or think even they think that? No. But if "soon" is stretched to "within 50 years", then it's much more reasonable. So their current actions seem to be really jumping the gun, but the overall concept feels credible.
  - perihelions 1y ago
    It's lazy to believe that humanity's collective decision-making would, in the future, protect AI's merely for being conscious beings. The tech economy *today* runs on the slave labor of humans, in foreign, third-world countries. All humanity needs to do is draw a line, push the conscious AI's outside that line, and declare, "not our problem anymore!" That's what we do today, with humans. That is the human condition. Show me a tech company that lobbies for "model welfare" for conscious human models enslaved in Xinjiang labor camps, building their tech parts. You know what—actually most of them lobby against that[0]. The talk hurts their profits. Does anyone really think, that any of them would blink about enslaving a billion conscious AI's to work for free? That faced with so much profit, the humans in charge would pause, and contemplate abstract morals? [0] https://www.washingtonpost.com/technology/2020/11/20/apple-uighur/ https://www.washingtonpost.com/technology/2020/11/20/apple-u... ("Apple is lobbying against a bill aimed at stopping forced labor in China") Maybe humanity will be in a nicer place in the future—but, we won't get there by letting (of all people!) tech-industry CEO's lead us there: delegating our moral reason to these people who demand to position themselves as our moral leaders.
    - 1y ago
raincole 1y ago
Sounds like a very reasonable assumption to me.

bikeshaving 1y ago

I really think Anthropic should just violate user privacy and show which conversations Claude is refusing to answer to, to stop arguments like this. AI psychosis is a real and growing problem and I can only imagine the ways in which humans torment their AI conversation partners in private.

coderatlarge 1y ago
arguments like this cost anthropic nothing; violating privacy will cost them lawsuits.

postalcoder 1y ago

> There's not a good reason to do this for the user. Yes, even more so when encountering false positives. Today I asked about a pasta recipe. It told me to throw some anchovies in there. I responded with: "I have dried anchovies." Claude then ended my conversation due to content policies.

handoflixue 1y ago
The NEW termination method, from the article, will just say "Claude ended the conversation" If you get "This conversation was ended due to our Acceptable Usage Policy", that's a different termination. It's been VERY glitchy the past couple of weeks. I've had the most random topics get flagged here - at one point I couldn't say "ROT13" without it flagging me, despite discussing that exact topic in depth the day before, and then the day after! If you hit "EDIT" on your last message, you can branch to an un-terminated conversation.
- antonvs 1y ago
  Clearly you're planning something nefarious, if you're investigating such dangerous encryption techniques as ROT13.
  - whaleofatw2022 1y ago
    Just imagine how it might react to ROT26!
perihelions 1y ago
Claude flagged me for asking about sodium carbonate. I guess that it strongly dislikes chemistry topics. I'm probably now on some secret, LLM-generated lists of "drug and/or bombmaking" people—thank you kindly for that, Anthropic. Geeks will always be the first victims of AI, since excess of curiosity will lead them into places AI doesn't know how to classify. (I've long been in a rabbit-hole about washing sodas. Did you know the medieval glassmaking industry was entirely based on plants? Exotic plants—only extremophiles, halophytes growing on saltwater beach dunes, had high enough sodium content for their very best glass process. Was that a factor in the maritime empire, Venice, chancing to become the capital of glass since the 13th century—their long-term control of sea routes, and hence their artisans' stable, uninterrupted access to supplies of [redacted–policy violation] from small ports scattered across the Mediterranean? A city wouldn't raise master craftsmen if, half of the time, they had no raw materials to work on—if they spent half their days with folded hands).

deleted 1y ago

[deleted]