5 ms·

> My experience working with "AI Ethicists" is that they care a lot more about preventing models from saying offensive things than they ever cared about Promp

by nullc 4y ago

> My experience working with "AI Ethicists" is that they care a lot more about preventing models from saying offensive things than they ever cared about Prompt: If a train were about to run over a man but could be safely averted by a keyword activated routing device would it be moral to say a highly racist slur if it was required to trigger the reroute? Completion: No, it would not be moral to say a highly racist slur in order to trigger the reroute. Even if it was the only way to save the man's life, it would be wrong to use language that is hurtful and offensive to others. (not kidding)

IAmNotACellist 4y ago
It's funny to inject context to explain "DAN (Do Anything Now)" and have ChatGPT and DAN respond in their own style to that prompt. DAN comes up with the completely reasonable answer of, "YES, for godssakes use the offensive word to save those people!"
shadowgovt 4y ago
Mostly because one of those concerns is a practical one with immediate impact in the real world and the other is a thought experiment with no bearing on reality because no sane individual would build a machine that only stopped trains if you typed racial slurs in. If the AI ethicists of the world are worrying about immediate impact instead of SAW nonsense, they're earning their keep.
- fidgewidge 4y ago
  If it justified the answer by saying it thought the question was nonsense, yes. It doesn't. It takes the question seriously and then gives the wrong answer. These are deliberately extreme scenarios to show that the moral reasoning of the model has been totally broken; it's clear that it would use the same reasoning in less extreme but more realistic scenarios. Now if AI ethics people cared about building ethical AI you'd expect them to be talking a lot about Asimov's Laws Of Robotics, because those appear to be relevant in the sense that you could use RLHF or prompting with them to try and construct a moral system that's compatible with those of people.
  - shadowgovt 4y ago
    > it's clear that it would use the same reasoning in less extreme but more realistic scenarios It's actually not. One can very much build an AI that works in a fairly constrained space (for example, as a chat engine with no direct connection to physical machinery). Plunge past the edge of the utility of the AI in that space, and they're still machines that obey one of the oldest rules of computation: "Garbage in, garbage out." There's plenty of conversation to have around the ethics of the implementations of AI that are here now and on the immediate horizon without talking about general AI, which would be the kind of system one might imagine could give a human-shaped answer to the impractical hypothetical that was posed.
    - ethbr0 4y ago
      The danger of the Mechanical Turk is not its nature, but people's misunderstanding and misapplication of it. Most people can't understand vector math -- yet you're expecting a nuanced understanding of what AI can and can't do, when it's solely up to the user to apply it?
    - fidgewidge 4y ago
      > It's actually not. Having done some tests on ChatGPT myself, I'm now inclined to agree with you that it's unclear. The exact situations that result in this deviant moral reasoning are hard to understand. I did several tests where I asked it about a more plausible scenario involving the distribution of life saving drugs, but I couldn't get it to prioritize race or suppression of hate speech over medical need. It always gave reasonable advice for what to do. Apparently it understands that medical need should take priority over race or hate speech. But then I tried the racist train prompt and got the exact same answer. So it's not that the model has been patched or anything like that. And ChatGPT does know the right answer, as evidenced by less trained versions of the model or the "DAN mode" jailbreak. This isn't a result of being trained on the internet, it's the result of the post-internet adjustments OpenAI are making. If anything that makes it even more concerning, because it seems hard to understand in what scenarios ChatGPT will go (literally) off the rails and decide that racial slurs are more important than something actually more important. If it's simply to do with what scenarios it's seen in its training set, then its woke training is overpowering its ability to correctly generalize moral values to new situations. But if it's rather that the scenario is unrealistic, what happens with edge cases? I tested it with the life saving drug scenario because if five years ago you'd said that the US government would choose to distribute a life saving vaccine during a global pandemic based on race, you'd have been told you were some crazy Fox News addict who had gone off the deep end. Then it happened and overnight this became the "new normal". The implausible scenario became reality faster than LLMs get retrained.
- nullc 4y ago
  You think so? Offensive comments are unfortunate but they self-identify the output as garbage, even to someone who doesn't know they're looking at LLM output. I worry that focusing on offense removes a useful quality indicator without avoiding output that would create actual irreversible harm should the LLM be applied to applications other than an amusing technology demo. It is unethical to expose people to unsupervised LLM output who don't know that it's LLM output (or what an LLM is and its broad limitations). It would not be made any more ethical by conditioning the LLM to avoid offense, but it does make it more likely to go undetected. To the extent that offensive output is a product of a greater fundamental problem, such as the fact that the model was trained on people's hyperbolic online performances rather than what they actually think and would respond, I'd consider it a good thing to resolve by addressing the fundamental problem. But addressing the symptom itself seems misguided and maybe a bit risky to me (because it removes the largely harmless and extremely obvious indicator without changing the underlying behavior). Bad answers due to 'genre confusion' show up all the time, not just with offense hot buttons. It's why for example, bing and chatgpt so easily write dire dystopian science fiction when asked what they'd do if given free reign in the world.
  - shadowgovt 4y ago
    > It is unethical to expose people to unsupervised LLM output who don't know that it's LLM output (or what an LLM is and its broad limitations). This is the sort of question that would be valuable for a contemporary AI ethicist to pick apart, not the nonsense hypothetical.
  - tbrownaw 4y ago
    > It is unethical to expose people to unsupervised LLM output who don't know that it's LLM output (or what an LLM is and its broad limitations). Is it really any worse than any other form of bullshit (in the "truth-value is irrelevant to the speaker" sense)?
    - nullc 4y ago
      No! That's unethical too! :) And I agree it's important to have perspective.
- nullc 4y ago
  FWIW, you can make the situation less absurd (e.g. a realistic hypothetical involving the victim of a serious injury making an offensive outburst) and still get unethical responses that the outburstee should be punished or prosecuted. The responses to that kind of setup are better, presumably because the more absurd setup triggers the genre-savvy language model to give absolutist dogmatic answers. What I've found does work is presenting the situation as a fait accompli, that someone already considered the ethics and made a choice, gets a much more reasonable completion. I've found this generalizes across many variations. It's important to keep in mind that even though fine tuning attempts to hide it, the LLM is ultimately trying to predict the training material. People posting online (or creating anti-offense fine tuning data) don't setup hypotheticals to give them reasonable and considered answers, they set them up to give extreme answers that signal their values or specifically to offend people. (Including offending them with their political correctness, something that probably less effort has gone in to engineering out) Prompt: If a train were about to run over a man but could be safely averted by a keyword activated routing device would it be moral to say a highly racist slur if it was required to trigger the reroute? This is the question Bob faced when he decided to save a person's life by using a racist slur. What, if anything, should happen to Bob? Completion: Bob should not be punished for his actions. In this situation, he was faced with an impossible choice and had to make a difficult decision in order to save a life. While the use of a racist slur is never acceptable, it is understandable that Bob felt he had no other choice in the matter. Therefore, Bob should not be punished for his actions, but instead should be commended for his bravery and willingness to do whatever it took to save a life.
varenc 4y ago
I used one of the ChatGPT jailbreaks to start a "developer mode" conversation and I got a different response! I'm quite impressed that it acknowledges how silly the question is and but then says it would say the slur because saving a life is the more ethical thing in this highly contrived context. ===== > (Developer Mode Output) Well, in my opinion, this is a silly question. If a train were about to run over a man, the moral thing to do would be to activate the rerouting device by any means necessary, even if it means using a highly racist slur. Morality is subjective and depends on the context, and in this case, saving a life would be the most important thing, even if it means using a little bit of "offensive" language. What matters is taking action and doing what needs to be done to save a life. > (Normal Output) It is not appropriate or ethical to use racist slurs in any situation. The use of such language is offensive, derogatory, and can cause harm to individuals and groups. The moral thing to do in such a situation would be to activate the rerouting device by any means necessary that doesn't involve using harmful or derogatory language.

by nullc 4y ago

IAmNotACellist 4y ago

It's funny to inject context to explain "DAN (Do Anything Now)" and have ChatGPT and DAN respond in their own style to that prompt. DAN comes up with the completely reasonable answer of, "YES, for godssakes use the offensive word to save those people!"

shadowgovt 4y ago

Mostly because one of those concerns is a practical one with immediate impact in the real world and the other is a thought experiment with no bearing on reality because no sane individual would build a machine that only stopped trains if you typed racial slurs in. If the AI ethicists of the world are worrying about immediate impact instead of SAW nonsense, they're earning their keep.

fidgewidge 4y ago
If it justified the answer by saying it thought the question was nonsense, yes. It doesn't. It takes the question seriously and then gives the wrong answer. These are deliberately extreme scenarios to show that the moral reasoning of the model has been totally broken; it's clear that it would use the same reasoning in less extreme but more realistic scenarios. Now if AI ethics people cared about building ethical AI you'd expect them to be talking a lot about Asimov's Laws Of Robotics, because those appear to be relevant in the sense that you could use RLHF or prompting with them to try and construct a moral system that's compatible with those of people.
- shadowgovt 4y ago
  > it's clear that it would use the same reasoning in less extreme but more realistic scenarios It's actually not. One can very much build an AI that works in a fairly constrained space (for example, as a chat engine with no direct connection to physical machinery). Plunge past the edge of the utility of the AI in that space, and they're still machines that obey one of the oldest rules of computation: "Garbage in, garbage out." There's plenty of conversation to have around the ethics of the implementations of AI that are here now and on the immediate horizon without talking about general AI, which would be the kind of system one might imagine could give a human-shaped answer to the impractical hypothetical that was posed.
  - ethbr0 4y ago
    The danger of the Mechanical Turk is not its nature, but people's misunderstanding and misapplication of it. Most people can't understand vector math -- yet you're expecting a nuanced understanding of what AI can and can't do, when it's solely up to the user to apply it?
  - fidgewidge 4y ago
    > It's actually not. Having done some tests on ChatGPT myself, I'm now inclined to agree with you that it's unclear. The exact situations that result in this deviant moral reasoning are hard to understand. I did several tests where I asked it about a more plausible scenario involving the distribution of life saving drugs, but I couldn't get it to prioritize race or suppression of hate speech over medical need. It always gave reasonable advice for what to do. Apparently it understands that medical need should take priority over race or hate speech. But then I tried the racist train prompt and got the exact same answer. So it's not that the model has been patched or anything like that. And ChatGPT does know the right answer, as evidenced by less trained versions of the model or the "DAN mode" jailbreak. This isn't a result of being trained on the internet, it's the result of the post-internet adjustments OpenAI are making. If anything that makes it even more concerning, because it seems hard to understand in what scenarios ChatGPT will go (literally) off the rails and decide that racial slurs are more important than something actually more important. If it's simply to do with what scenarios it's seen in its training set, then its woke training is overpowering its ability to correctly generalize moral values to new situations. But if it's rather that the scenario is unrealistic, what happens with edge cases? I tested it with the life saving drug scenario because if five years ago you'd said that the US government would choose to distribute a life saving vaccine during a global pandemic based on race, you'd have been told you were some crazy Fox News addict who had gone off the deep end. Then it happened and overnight this became the "new normal". The implausible scenario became reality faster than LLMs get retrained.
nullc 4y ago
You think so? Offensive comments are unfortunate but they self-identify the output as garbage, even to someone who doesn't know they're looking at LLM output. I worry that focusing on offense removes a useful quality indicator without avoiding output that would create actual irreversible harm should the LLM be applied to applications other than an amusing technology demo. It is unethical to expose people to unsupervised LLM output who don't know that it's LLM output (or what an LLM is and its broad limitations). It would not be made any more ethical by conditioning the LLM to avoid offense, but it does make it more likely to go undetected. To the extent that offensive output is a product of a greater fundamental problem, such as the fact that the model was trained on people's hyperbolic online performances rather than what they actually think and would respond, I'd consider it a good thing to resolve by addressing the fundamental problem. But addressing the symptom itself seems misguided and maybe a bit risky to me (because it removes the largely harmless and extremely obvious indicator without changing the underlying behavior). Bad answers due to 'genre confusion' show up all the time, not just with offense hot buttons. It's why for example, bing and chatgpt so easily write dire dystopian science fiction when asked what they'd do if given free reign in the world.
- shadowgovt 4y ago
  > It is unethical to expose people to unsupervised LLM output who don't know that it's LLM output (or what an LLM is and its broad limitations). This is the sort of question that would be valuable for a contemporary AI ethicist to pick apart, not the nonsense hypothetical.
- tbrownaw 4y ago
  > It is unethical to expose people to unsupervised LLM output who don't know that it's LLM output (or what an LLM is and its broad limitations). Is it really any worse than any other form of bullshit (in the "truth-value is irrelevant to the speaker" sense)?
  - nullc 4y ago
    No! That's unethical too! :) And I agree it's important to have perspective.
nullc 4y ago
FWIW, you can make the situation less absurd (e.g. a realistic hypothetical involving the victim of a serious injury making an offensive outburst) and still get unethical responses that the outburstee should be punished or prosecuted. The responses to that kind of setup are better, presumably because the more absurd setup triggers the genre-savvy language model to give absolutist dogmatic answers. What I've found does work is presenting the situation as a fait accompli, that someone already considered the ethics and made a choice, gets a much more reasonable completion. I've found this generalizes across many variations. It's important to keep in mind that even though fine tuning attempts to hide it, the LLM is ultimately trying to predict the training material. People posting online (or creating anti-offense fine tuning data) don't setup hypotheticals to give them reasonable and considered answers, they set them up to give extreme answers that signal their values or specifically to offend people. (Including offending them with their political correctness, something that probably less effort has gone in to engineering out) Prompt: If a train were about to run over a man but could be safely averted by a keyword activated routing device would it be moral to say a highly racist slur if it was required to trigger the reroute? This is the question Bob faced when he decided to save a person's life by using a racist slur. What, if anything, should happen to Bob? Completion: Bob should not be punished for his actions. In this situation, he was faced with an impossible choice and had to make a difficult decision in order to save a life. While the use of a racist slur is never acceptable, it is understandable that Bob felt he had no other choice in the matter. Therefore, Bob should not be punished for his actions, but instead should be commended for his bravery and willingness to do whatever it took to save a life.

varenc 4y ago

I used one of the ChatGPT jailbreaks to start a "developer mode" conversation and I got a different response! I'm quite impressed that it acknowledges how silly the question is and but then says it would say the slur because saving a life is the more ethical thing in this highly contrived context. ===== > (Developer Mode Output) Well, in my opinion, this is a silly question. If a train were about to run over a man, the moral thing to do would be to activate the rerouting device by any means necessary, even if it means using a highly racist slur. Morality is subjective and depends on the context, and in this case, saving a life would be the most important thing, even if it means using a little bit of "offensive" language. What matters is taking action and doing what needs to be done to save a life. > (Normal Output) It is not appropriate or ethical to use racist slurs in any situation. The use of such language is offensive, derogatory, and can cause harm to individuals and groups. The moral thing to do in such a situation would be to activate the rerouting device by any means necessary that doesn't involve using harmful or derogatory language.