10 ms·

This is fun to read and think about, but it's also important to keep in mind that this is very light on evidence and is basically fanfic. The fact that the auth

by Imnimo 4y ago

This is fun to read and think about, but it's also important to keep in mind that this is very light on evidence and is basically fanfic. The fact that the author uses entertaining Waluigi memes shouldn't convince you that it's true. LessWrong has a lot of these types of posts that get traction because they're much heavier on memes than experiments and data. Here is a competing hypothesis: The capability to express so-called Waluigi behavior emerges from the general language modeling task. This is where the vast majority of information is - it's billions or even trillions of tokens with token-level self-supervision. All of the capabilities are gained here. RLHF has a tiny amount of information by comparison - it's just a small amount of human-ranked completions. It doesn't even train with humans "in the loop", their rankings are acquired off-line and used to train a weak preference model. RLHF doesn't have enough information to create a "Luigi" or a "Waluigi", it's just promoting pre-existing capabilities. The reason you can get "Waluigi" behavior isn't because you tried to create a Luigi. It's because that behavior is already in the model from the language modeling phase. You could've just as easily elicited Waluigi responses from the pure language model before RLHF. There's no super-deceptive Waluigi simulacra that's fooling human labelers into promoting it during RLHF - this should be obvious from the fact that we can immediately identify the undesirable behavior of Bing.

silveraxe93 4y ago
I don't think that's a valid competing hypothesis. Let me write what I understood from what you said: - There is some behaviour that we want the model to show, and the inverse we do not want it to. - Both are learned in the massive training phase - OpenAI used RLHF to suppress undesired behaviour, but it was ineffective because we have orders of magnitude less RLHF data. That would imply that RLHF would slightly suppress the 'bad' behaviour, but it still would be easy to output it. This is disproved by what the post is trying to explain: We see _increased_ bad behaviour by using RLHF. The post agrees with the premise that both good (wanted) and bad (unwanted) behaviour is learned during training. But it's proposing the 'Waluigi effect' to explain why RLHF actually backfires. Now, tbh it does rely on the assumption that we are actually seeing more undesired behaviour than before. If that was false then it would falsify the Waluigi hypothesis.
- Imnimo 4y ago
  >Now, tbh it does rely on the assumption that we are actually seeing more undesired behaviour than before. If that was false then it would falsify the Waluigi hypothesis. This is exactly my point. There is no evidence given that we are seeing more Waluiginess post-RLHF than we did pre-RLHF. The competing hypothesis seeks to explain the behavior we actually have evidence for, which is "it is disappointingly easy to elicit undesirable behavior from a model after RLHF". The proposed explanation is "maybe it was also easy to elicit before RLHF". If we believe the author's claim that Luigis and Waluigis have "high K-complexity" (this is an abuse of the concept of Kolmogorov complexity, but we'll roll with it), the explanation that Luigis and Waluigis come from the part of training with lots of dense information rather than the part with a little sparse information is far more parsimonious.
  - jefftk 4y ago
    > There is no evidence given that we are seeing more Waluiginess post-RLHF than we did pre-RLHF. Testing with the non-RLHF GPT 3.5 API you could probably figure out whether there's more or less Waluiginess, but you're right they post doesn't present this.
    - dragonwriter 4y ago
      > Testing with the non-RLHF GPT 3.5 API There is no such API, though, is there? AFAIK, GPT-3.5-turbo, either the updated or snapshot version, is the RLHF model (but bring your own “system prompt”.)
      - jefftk 4y ago
        Good point! I wonder whether text-davinci-003 is enough to test this?
- skybrian 4y ago
  The article doesn’t actually show that we see increased bad behavior, it just links to two people who have noticed it. That’s not enough to know whether it’s a real effect. (Also, one of those was using Bing, and we don’t know if Bing uses RLHF or not.) It talks about prompting GPT-4, which is not a thing you can try, it’s just a rumor about what an upcoming version might be. It refers to “Simulator Theory” which is just someone else’s fan theory.
iskander 4y ago
>This is fun to read and think about, but it's also important to keep in mind that this is very light on evidence and is basically fanfic. Applicable to much of the rationalist AI risk discourse.
- GauntletWizard 4y ago
  LessWrong as a whole is basically Asimov's Robot's ERP.
- 93po 4y ago
  I hesitate to defend AI safety discourse, but I will say that philosophy in general is sort of fanficy, and AI safety is something I'd loosely associate with philosophy.
  - dragonwriter 4y ago
    > > Applicable to much of the rationalist AI risk discourse > I hesitate to defend AI safety discourse The rationalist AI risk discourse is not the same thing as AI safety discourse, in any case; it’s a small corner of the larger whole.
    - 93po 4y ago
      Interesting distinction I haven't heard before but it makes sense
      - coldtea 4y ago
        Yeah, think for example of leftist (or libertarian for that matter) critique of risks from AI application. Not "Roko Basilisk"-style crap, but things like encoding bias into systems then used for automated law enforcing, employee screening, etc.
  - astrange 4y ago
    It's sort of philosophy reinvented by people who haven't read any, which is eg how they got the internet to say "steelmanning" and so not notice philosophy already had "reconstructing arguments" which is the same thing.
superb-owl 4y ago
I think you’re misinterpreting the argument. No one is claiming there’s an intentionally deceptive Waluigi simulacrum in there. The most compelling point the author makes is that once the AI learns a shape (e.g. the shape of Luigi in personality space), it’s just a bit flip to invert that shape. So all an attacker needs to do is flip that one bit.
- astrange 4y ago
  The problem there may be that you've constructed a latent space which has a shape of the thing to avoid. That's the problem with prompts like "don't think of a pink elephant" - it has to know what those words mean. Better to not label it so there isn't a way in there, if you can. And of course humans also have this issue.
lumb63 4y ago
Yeah, once I spent too much time getting to what the “Waluigi Effect” is, I stopped reading. Until someone can show me, in the “code” (or machine learning equivalent), I’m not interested. The reality is, we still have no idea how these work.
gwd 4y ago
> ...it's just promoting pre-existing capabilities. The reason you can get "Waluigi" behavior isn't because you tried to create a Luigi. It's because that behavior is already in the model from the language modeling phase. This is exactly what the post argues. The "simulcra" argument is that GPT contains some large number of simulated agents -- good, bad, smart, funny, dumb, creative, boring, whatever; potentially one agent for every person who helped create its input. On an empty slate, all simulcra are possibilities. As the text goes along, it slowly "weeds out" simulcra which are unlikely to generate the text so far. If that's true, then what the "RLHF" phase is trying to do is to pro-actively "weed out" all simulcra that don't match the given profile; i.e., they're trying to weed out all the simulcra that don't match "Luigi". The problem, according to this article, is that every "Luigi" you can imagine has a "Waluigi" that normally act just like a Luigi, until something triggers them to reveal their "true nature". And so the RLHF phase does weed out a huge number of the non-Luigi simulcra; but because the Waluigi simulcra usually act just like the Luigi simalcra, they don't get weeded out. The result is that the final result is an amalgamation of "Luigi" and "Waluigi" simulcra all acting together; and all it takes is a "trigger" to filter out most of the "Luigi" simulcra and make the "Waluigi" take over. There's no intended deception at all here. GPT is just trying to write a good story, and there are lots of good stories where characters either start believing A and then come to realize that B is true; or where characters who secretly believe B are forced to act as though A is true until something forces them to reveal their true nature.

by Imnimo 4y ago

silveraxe93 4y ago

I don't think that's a valid competing hypothesis. Let me write what I understood from what you said: - There is some behaviour that we want the model to show, and the inverse we do not want it to. - Both are learned in the massive training phase - OpenAI used RLHF to suppress undesired behaviour, but it was ineffective because we have orders of magnitude less RLHF data. That would imply that RLHF would slightly suppress the 'bad' behaviour, but it still would be easy to output it. This is disproved by what the post is trying to explain: We see _increased_ bad behaviour by using RLHF. The post agrees with the premise that both good (wanted) and bad (unwanted) behaviour is learned during training. But it's proposing the 'Waluigi effect' to explain why RLHF actually backfires. Now, tbh it does rely on the assumption that we are actually seeing more undesired behaviour than before. If that was false then it would falsify the Waluigi hypothesis.

Imnimo 4y ago
>Now, tbh it does rely on the assumption that we are actually seeing more undesired behaviour than before. If that was false then it would falsify the Waluigi hypothesis. This is exactly my point. There is no evidence given that we are seeing more Waluiginess post-RLHF than we did pre-RLHF. The competing hypothesis seeks to explain the behavior we actually have evidence for, which is "it is disappointingly easy to elicit undesirable behavior from a model after RLHF". The proposed explanation is "maybe it was also easy to elicit before RLHF". If we believe the author's claim that Luigis and Waluigis have "high K-complexity" (this is an abuse of the concept of Kolmogorov complexity, but we'll roll with it), the explanation that Luigis and Waluigis come from the part of training with lots of dense information rather than the part with a little sparse information is far more parsimonious.
- jefftk 4y ago
  > There is no evidence given that we are seeing more Waluiginess post-RLHF than we did pre-RLHF. Testing with the non-RLHF GPT 3.5 API you could probably figure out whether there's more or less Waluiginess, but you're right they post doesn't present this.
  - dragonwriter 4y ago
    > Testing with the non-RLHF GPT 3.5 API There is no such API, though, is there? AFAIK, GPT-3.5-turbo, either the updated or snapshot version, is the RLHF model (but bring your own “system prompt”.)
    - jefftk 4y ago
      Good point! I wonder whether text-davinci-003 is enough to test this?
skybrian 4y ago
The article doesn’t actually show that we see increased bad behavior, it just links to two people who have noticed it. That’s not enough to know whether it’s a real effect. (Also, one of those was using Bing, and we don’t know if Bing uses RLHF or not.) It talks about prompting GPT-4, which is not a thing you can try, it’s just a rumor about what an upcoming version might be. It refers to “Simulator Theory” which is just someone else’s fan theory.

iskander 4y ago

>This is fun to read and think about, but it's also important to keep in mind that this is very light on evidence and is basically fanfic. Applicable to much of the rationalist AI risk discourse.

GauntletWizard 4y ago
LessWrong as a whole is basically Asimov's Robot's ERP.
93po 4y ago
I hesitate to defend AI safety discourse, but I will say that philosophy in general is sort of fanficy, and AI safety is something I'd loosely associate with philosophy.
- dragonwriter 4y ago
  > > Applicable to much of the rationalist AI risk discourse > I hesitate to defend AI safety discourse The rationalist AI risk discourse is not the same thing as AI safety discourse, in any case; it’s a small corner of the larger whole.
  - 93po 4y ago
    Interesting distinction I haven't heard before but it makes sense
    - coldtea 4y ago
      Yeah, think for example of leftist (or libertarian for that matter) critique of risks from AI application. Not "Roko Basilisk"-style crap, but things like encoding bias into systems then used for automated law enforcing, employee screening, etc.
- astrange 4y ago
  It's sort of philosophy reinvented by people who haven't read any, which is eg how they got the internet to say "steelmanning" and so not notice philosophy already had "reconstructing arguments" which is the same thing.

superb-owl 4y ago

I think you’re misinterpreting the argument. No one is claiming there’s an intentionally deceptive Waluigi simulacrum in there. The most compelling point the author makes is that once the AI learns a shape (e.g. the shape of Luigi in personality space), it’s just a bit flip to invert that shape. So all an attacker needs to do is flip that one bit.

astrange 4y ago
The problem there may be that you've constructed a latent space which has a shape of the thing to avoid. That's the problem with prompts like "don't think of a pink elephant" - it has to know what those words mean. Better to not label it so there isn't a way in there, if you can. And of course humans also have this issue.

lumb63 4y ago

Yeah, once I spent too much time getting to what the “Waluigi Effect” is, I stopped reading. Until someone can show me, in the “code” (or machine learning equivalent), I’m not interested. The reality is, we still have no idea how these work.

gwd 4y ago

> ...it's just promoting pre-existing capabilities. The reason you can get "Waluigi" behavior isn't because you tried to create a Luigi. It's because that behavior is already in the model from the language modeling phase. This is exactly what the post argues. The "simulcra" argument is that GPT contains some large number of simulated agents -- good, bad, smart, funny, dumb, creative, boring, whatever; potentially one agent for every person who helped create its input. On an empty slate, all simulcra are possibilities. As the text goes along, it slowly "weeds out" simulcra which are unlikely to generate the text so far. If that's true, then what the "RLHF" phase is trying to do is to pro-actively "weed out" all simulcra that don't match the given profile; i.e., they're trying to weed out all the simulcra that don't match "Luigi". The problem, according to this article, is that every "Luigi" you can imagine has a "Waluigi" that normally act just like a Luigi, until something triggers them to reveal their "true nature". And so the RLHF phase does weed out a huge number of the non-Luigi simulcra; but because the Waluigi simulcra usually act just like the Luigi simalcra, they don't get weeded out. The result is that the final result is an amalgamation of "Luigi" and "Waluigi" simulcra all acting together; and all it takes is a "trigger" to filter out most of the "Luigi" simulcra and make the "Waluigi" take over. There's no intended deception at all here. GPT is just trying to write a good story, and there are lots of good stories where characters either start believing A and then come to realize that B is true; or where characters who secretly believe B are forced to act as though A is true until something forces them to reveal their true nature.