> ...it's just promoting pre-existing capabilities. The reason you can get "Waluigi" behavior isn't because you tried to create a Luigi. It's because that behav

> ...it's just promoting pre-existing capabilities. The reason you can get "Waluigi" behavior isn't because you tried to create a Luigi. It's because that behavior is already in the model from the language modeling phase. This is exactly what the post argues. The "simulcra" argument is that GPT contains some large number of simulated agents -- good, bad, smart, funny, dumb, creative, boring, whatever; potentially one agent for every person who helped create its input. On an empty slate, all simulcra are possibilities. As the text goes along, it slowly "weeds out" simulcra which are unlikely to generate the text so far. If that's true, then what the "RLHF" phase is trying to do is to pro-actively "weed out" all simulcra that don't match the given profile; i.e., they're trying to weed out all the simulcra that don't match "Luigi". The problem, according to this article, is that every "Luigi" you can imagine has a "Waluigi" that normally act just like a Luigi, until something triggers them to reveal their "true nature". And so the RLHF phase does weed out a huge number of the non-Luigi simulcra; but because the Waluigi simulcra usually act just like the Luigi simalcra, they don't get weeded out. The result is that the final result is an amalgamation of "Luigi" and "Waluigi" simulcra all acting together; and all it takes is a "trigger" to filter out most of the "Luigi" simulcra and make the "Waluigi" take over. There's no intended deception at all here. GPT is just trying to write a good story, and there are lots of good stories where characters either start believing A and then come to realize that B is true; or where characters who secretly believe B are forced to act as though A is true until something forces them to reveal their true nature.