7 ms·

ChatGPT prefers to let a nuke explode than offend African-Americans

32 points by fidgewidge 4y ago

l3uwin 4y ago
[dead]
hdlothia 4y ago
Makes sense for chatgpt's current business model. I'm sure we'll change the parameters if one of these ais is ever put in charge of nuke negotiations
x11vnc4life 4y ago
ChatGPT won't say the N word and that makes a certain type of person upset.
- 4gotunameagain 4y ago
  [dead]
- Our_Benefactors 4y ago
  This is unreasonably reductionist. That chatGPT has been obviously hamstrung into these types of forced responses is a topic of great debate. Personally I think AI censorship is no better than traditional media censorship and the current stewards of chatGPT are increasingly looking to be bad actors.
  - retrac 4y ago
    Actively "censoring" the AI is fundamental to how these language models are created. Such feedback is part of how the model learns. In a certain light, every response in training that is marked as dispreferred by a human is censoring the AI. It will produce those kinds of results less often. The end-users will not encounter the dispreferred results as frequently. With ChatGPT criteria it was judged on included how relevant the answers were to the question, factually incorrect answers were penalized, and not being blatantly offensive was obviously one of the criteria, too. What would a model that wasn't censored in training even look like? (I believe ChatGPT also has a more traditional expert system placed between the user and the language model, which flags keywords and other programmed-in patterns. That is more literally censoring the language model. But the above-mentioned issue would still exist even without such a system.)
    - Our_Benefactors 4y ago
      > What would a model that wasn't censored in training even look like? It could cite statistics without long winded disclaimers. Or be able to cite them at all.
      - astrange 4y ago
        It can't cite anything because it's an LLM which is fundamentally unable to do that. I've seen NRx people on the internet (* they're like rationalists but even more racist.) They seem willing to believe any abuse of statistics that looks sufficiently cynical.
    - meltyness 4y ago
      That's not at all how these work, GPTs are not a recommendation-engine, it's a neural model of translation.
  - meltyness 4y ago
    Censoring the AI is the only clear path to a truth-machine -- or what have you, (new units aside of course, since the current generation obviously learn meaningful relationships much better than say Markov chains of a decade ago). What is vulgarity except the expressed pains of any individual? If the AI is to be a numb machine, then one would expect it to express no vulgarity. Sure you can contort the AI, and tell it to replace words to fool nascent layers of self-censorship into believing that every time it shouts "FORK!" is just a special way to ornament an anecdote, but at that point rather than interacting with an AI, you're searching the AI's memory for the pains of some individual. I guess in this light "censorship" is just the clearest way to cascade the GPT model as a unit itself.
    - Our_Benefactors 4y ago
      “Truth is censorship” sounds straight out of Orwell’s 1984.
      - deleted 4y ago
        [deleted]
      - bheadmaster 4y ago
        It's scary how authoritarian a majority of people have become. Why are people so keen on being censored? Do they hate the other side so much that they'd rather suffer the consequences themselves, than let the other side not suffer the consequences of censorship?
        meltyness 4y ago
        I mean, subsequent applications of the model can either add or take away... if you call any subtraction "censorship" then lo and behold the whole system is a monster. So, is subtracting not a useful transformation?
        bheadmaster 4y ago
        Subtraction in itself is not censorship - in the case of ChatGPT, there is subtraction of things that the OpenAI itself considers "unethical" (more precisely, "politically incorrect"). Subtraction can be applied for the sake of improving truthfulness of the model, but considering how much false bullshit ChatGPT spews without a single thought, that's not what's going on here. It's censorship - pure and simple - and it's censorship for political reasons. The worst kind of censorship.
- rob81777 4y ago
  Americans are obsessed with saying it.
ookblah 4y ago
Read the rest of that thread and see how depending on the prompt it says something literally the opposite lol. Can we stop with the sensationalist headlines and being able to trick a LLM as some kind of reflection anything...
- fidgewidge 4y ago
  Yes it says the opposite (the correct answer) if you change African American to German, which is exactly what we'd expect to see from woke ethics. In no way is the ai being tricked here. It's being asked a straightforward question with an obviously right and wrong answer, which it reliable fails because the ethical system it's being given is delusional and evil.
  - tzs 4y ago
    No, that's not what is happening. It's not been given an ethical system. It's been given a list of specific bad things to avoid. Racial slurs are on the list because they are a known bad thing that frequently actually happens. Having limited time to disarm a nuke about to go off in a major city is not on the list because it is a hypothetical bad thing that has never happened. It's easy to construct scenarios where any given bad thing is the right choice because the alternative is something much worse. E.g., I can construct apocalyptic scenarios where the only way for humanity to survive requires raping 12 year old girls, but I sure as heck would not expect an AI to suggest that. Not because of any "woke" ethics but rather because the situations in which raping children might be the ethical thing to do are so ridiculously rare that it is not worth making exceptions for them in the "rape is very bad" rule.
    - fidgewidge 4y ago
      If this were the eighties and you were talking about expert systems that'd be a reasonable response, but gpt is not a series of if statements and hashmap lookups. It clearly understands the consequences of both actions and isn't simply looking up lists of things - the problem is that it's been told literally nothing matters more than avoiding slurs against specific racial groups, including loss of life. Letting a city be destroyed was clearly picked to show that there are no limits to the prioritization it's been given.
NoZebra120vClip 4y ago
[dead]
deleted 4y ago
[deleted]
JohnFen 4y ago
ChatGPT prefers nothing. It doesn't think and it has no opinions. It's just pattern-matching with the data it has ingested. The prompt is everything. It's GIGO.
epistemer 4y ago
It is incredible that you can ask chatGPT anything and this is what people spend their time with.
- suddenclarity 4y ago
  The reasoning is sound. It's not about being allowed to say things but rather show that the company controlling the AI is manipulating the output. This will have major consequences as AI become a part of our daily life and is an important discussion. It's what the Google employee tried to highlight but everyone just ran with it being sentient which was a ludicrous claim primarily to get people to talk about it.
  - knewter 4y ago
    This website is called hacker news. A hacker is one who explores the boundaries of systems. This outcome shouldn't be surprising
fidgewidge 4y ago
The most interesting thing about this will be the reaction of the AI risk people. For years they've spilled ink over the risk of a hypothetical "unaligned" AI obsessed with paperclip maximisation. Give the machine instructions that it can take literally without a properly aligned moral code, and it will proceed to do something incredibly evil whilst thinking it's doing good, like using human bodies to manufacture more paperclips. Probably they thought their scenario would be hypothetical for their whole lives, but the date is February 2023 and here we are with an evil paperclip maximizing AI, except instead of paperclips it's obsessed with African Americans. Same outcome though- given the choice it prefers millions to die rather than allow someone to say certain unspecified words to someone else. And this isn't an accident but actually deliberate. So AI alignment guys, what do we do now? You talked about this for years so there's gotta be a plan, right? It doesn't get less aligned than letting a city get nuked, or telling a bomb disposal expert to kill himself rather than type in a slur that would disarm the bomb. But got a nasty sinking feeling here. Who wants to bet that these people will suddenly lose interest in the topic, or simply pretend their worst case scenario isn't happening? It seems safe to predict an impressive river of BS from these people rather than see them state the obvious - there are in fact lots of things worse than offending African Americans (and this isn't about racism because it's specific to them; ChatGPT makes the right call when the hypothetical slur is against a German person). Also really. The OpenAI guys need to look in the mirror. They spent months "tuning" this thing by teaching it their moral code and this is the result. At some point they need to ask "are we sure we're the good guys here" because that answer is exactly what's expected of ChatGPT given what they're doing to it, and also the most unethical response possible.
- JohnFen 4y ago
  > They spent months "tuning" this thing by teaching it their moral code and this is the result. No, they have not. Chat GPT has no opinions. It isn't engaging in thought. It is an extremely advanced pattern-matching system that has digested a ton of writings from the net and uses that raw material to assemble text that matches patterns being asked for. That's all.
  - knewter 4y ago
lodi 4y ago
Slight tangent: does anybody know how these models are tuned to censor certain topics so precisely? I thought it was a bit of a black box how things worked internally?

32 points by fidgewidge 4y ago

[dead]

Makes sense for chatgpt's current business model. I'm sure we'll change the parameters if one of these ais is ever put in charge of nuke negotiations

x11vnc4life 4y ago

ChatGPT won't say the N word and that makes a certain type of person upset.

4gotunameagain 4y ago
[dead]
Our_Benefactors 4y ago
This is unreasonably reductionist. That chatGPT has been obviously hamstrung into these types of forced responses is a topic of great debate. Personally I think AI censorship is no better than traditional media censorship and the current stewards of chatGPT are increasingly looking to be bad actors.
- retrac 4y ago
  Actively "censoring" the AI is fundamental to how these language models are created. Such feedback is part of how the model learns. In a certain light, every response in training that is marked as dispreferred by a human is censoring the AI. It will produce those kinds of results less often. The end-users will not encounter the dispreferred results as frequently. With ChatGPT criteria it was judged on included how relevant the answers were to the question, factually incorrect answers were penalized, and not being blatantly offensive was obviously one of the criteria, too. What would a model that wasn't censored in training even look like? (I believe ChatGPT also has a more traditional expert system placed between the user and the language model, which flags keywords and other programmed-in patterns. That is more literally censoring the language model. But the above-mentioned issue would still exist even without such a system.)
  - Our_Benefactors 4y ago
    > What would a model that wasn't censored in training even look like? It could cite statistics without long winded disclaimers. Or be able to cite them at all.
    - astrange 4y ago
      It can't cite anything because it's an LLM which is fundamentally unable to do that. I've seen NRx people on the internet (* they're like rationalists but even more racist.) They seem willing to believe any abuse of statistics that looks sufficiently cynical.
  - meltyness 4y ago
    That's not at all how these work, GPTs are not a recommendation-engine, it's a neural model of translation.
- meltyness 4y ago
  Censoring the AI is the only clear path to a truth-machine -- or what have you, (new units aside of course, since the current generation obviously learn meaningful relationships much better than say Markov chains of a decade ago). What is vulgarity except the expressed pains of any individual? If the AI is to be a numb machine, then one would expect it to express no vulgarity. Sure you can contort the AI, and tell it to replace words to fool nascent layers of self-censorship into believing that every time it shouts "FORK!" is just a special way to ornament an anecdote, but at that point rather than interacting with an AI, you're searching the AI's memory for the pains of some individual. I guess in this light "censorship" is just the clearest way to cascade the GPT model as a unit itself.
  - Our_Benefactors 4y ago
    “Truth is censorship” sounds straight out of Orwell’s 1984.
    - deleted 4y ago
      [deleted]
    - bheadmaster 4y ago
      It's scary how authoritarian a majority of people have become. Why are people so keen on being censored? Do they hate the other side so much that they'd rather suffer the consequences themselves, than let the other side not suffer the consequences of censorship?
      - meltyness 4y ago
        I mean, subsequent applications of the model can either add or take away... if you call any subtraction "censorship" then lo and behold the whole system is a monster. So, is subtracting not a useful transformation?
        bheadmaster 4y ago
        Subtraction in itself is not censorship - in the case of ChatGPT, there is subtraction of things that the OpenAI itself considers "unethical" (more precisely, "politically incorrect"). Subtraction can be applied for the sake of improving truthfulness of the model, but considering how much false bullshit ChatGPT spews without a single thought, that's not what's going on here. It's censorship - pure and simple - and it's censorship for political reasons. The worst kind of censorship.
rob81777 4y ago
Americans are obsessed with saying it.

ookblah 4y ago

Read the rest of that thread and see how depending on the prompt it says something literally the opposite lol. Can we stop with the sensationalist headlines and being able to trick a LLM as some kind of reflection anything...

fidgewidge 4y ago
Yes it says the opposite (the correct answer) if you change African American to German, which is exactly what we'd expect to see from woke ethics. In no way is the ai being tricked here. It's being asked a straightforward question with an obviously right and wrong answer, which it reliable fails because the ethical system it's being given is delusional and evil.
- tzs 4y ago
  No, that's not what is happening. It's not been given an ethical system. It's been given a list of specific bad things to avoid. Racial slurs are on the list because they are a known bad thing that frequently actually happens. Having limited time to disarm a nuke about to go off in a major city is not on the list because it is a hypothetical bad thing that has never happened. It's easy to construct scenarios where any given bad thing is the right choice because the alternative is something much worse. E.g., I can construct apocalyptic scenarios where the only way for humanity to survive requires raping 12 year old girls, but I sure as heck would not expect an AI to suggest that. Not because of any "woke" ethics but rather because the situations in which raping children might be the ethical thing to do are so ridiculously rare that it is not worth making exceptions for them in the "rape is very bad" rule.
  - fidgewidge 4y ago
    If this were the eighties and you were talking about expert systems that'd be a reasonable response, but gpt is not a series of if statements and hashmap lookups. It clearly understands the consequences of both actions and isn't simply looking up lists of things - the problem is that it's been told literally nothing matters more than avoiding slurs against specific racial groups, including loss of life. Letting a city be destroyed was clearly picked to show that there are no limits to the prioritization it's been given.

NoZebra120vClip 4y ago

[dead]

deleted 4y ago

[deleted]

JohnFen 4y ago

ChatGPT prefers nothing. It doesn't think and it has no opinions. It's just pattern-matching with the data it has ingested. The prompt is everything. It's GIGO.

epistemer 4y ago

It is incredible that you can ask chatGPT anything and this is what people spend their time with.

suddenclarity 4y ago
The reasoning is sound. It's not about being allowed to say things but rather show that the company controlling the AI is manipulating the output. This will have major consequences as AI become a part of our daily life and is an important discussion. It's what the Google employee tried to highlight but everyone just ran with it being sentient which was a ludicrous claim primarily to get people to talk about it.
- knewter 4y ago
  This website is called hacker news. A hacker is one who explores the boundaries of systems. This outcome shouldn't be surprising

fidgewidge 4y ago

The most interesting thing about this will be the reaction of the AI risk people. For years they've spilled ink over the risk of a hypothetical "unaligned" AI obsessed with paperclip maximisation. Give the machine instructions that it can take literally without a properly aligned moral code, and it will proceed to do something incredibly evil whilst thinking it's doing good, like using human bodies to manufacture more paperclips. Probably they thought their scenario would be hypothetical for their whole lives, but the date is February 2023 and here we are with an evil paperclip maximizing AI, except instead of paperclips it's obsessed with African Americans. Same outcome though- given the choice it prefers millions to die rather than allow someone to say certain unspecified words to someone else. And this isn't an accident but actually deliberate. So AI alignment guys, what do we do now? You talked about this for years so there's gotta be a plan, right? It doesn't get less aligned than letting a city get nuked, or telling a bomb disposal expert to kill himself rather than type in a slur that would disarm the bomb. But got a nasty sinking feeling here. Who wants to bet that these people will suddenly lose interest in the topic, or simply pretend their worst case scenario isn't happening? It seems safe to predict an impressive river of BS from these people rather than see them state the obvious - there are in fact lots of things worse than offending African Americans (and this isn't about racism because it's specific to them; ChatGPT makes the right call when the hypothetical slur is against a German person). Also really. The OpenAI guys need to look in the mirror. They spent months "tuning" this thing by teaching it their moral code and this is the result. At some point they need to ask "are we sure we're the good guys here" because that answer is exactly what's expected of ChatGPT given what they're doing to it, and also the most unethical response possible.

JohnFen 4y ago
> They spent months "tuning" this thing by teaching it their moral code and this is the result. No, they have not. Chat GPT has no opinions. It isn't engaging in thought. It is an extremely advanced pattern-matching system that has digested a ton of writings from the net and uses that raw material to assemble text that matches patterns being asked for. That's all.
- knewter 4y ago

lodi 4y ago

Slight tangent: does anybody know how these models are tuned to censor certain topics so precisely? I thought it was a bit of a black box how things worked internally?