3 ms·

> We include one example in Figure 26, where clear state-tracking behavior is demonstrated. Figure 26 appears to start with "we need to predict the output", an

> We include one example in Figure 26, where clear state-tracking behavior is demonstrated. Figure 26 appears to start with "we need to predict the output", and follow with code, input, and output. Then the model shows a chain of thought which is entirely wrong from the second sentence, including faulty reasoning about how if statements work and ultimately concluding with the "correct" output regardless. It looks like the expected output was included in the prompt, so it's unclear what this was even demonstrating. Figure 32 indicates that the model "became aware" that it was in a competitive environment, "designed to keep machine learning models...guessing". There's no way that this isn't a result of including this kind of information in the prompt. Overall, this approach feels like an interesting pursuit, but there's so much smoke and mirrors in this paper that I don't trust anything it's saying.

iTokio 1y ago
I skimmed through the paper and the code and got the same conclusion. It’s overhyped, filled with marketing language. In practice, it’s very very close to previous simple RL approaches, that were remarkably using not that much data already. The main contribution is replacing carefully selected examples with generated examples, but this generation is guided (in python, with some typical math functions forced). It’s akin to replacing some manual tests with mutation testing. Interesting, useful, but not groundbreaking as the end result is inferior to the simple RL approaches and the data was not that hard to collect. It is an interesting approach to generalize to other domains where there might be less data available or less easy to curate
robblbobbl 1y ago
Fair enough
CBiddulph 1y ago
I checked Figure 26 - the way it's presented is a bit confusing, but the model prompt doesn't include the expected output. All the model sees is "Here is the function f, the input provided 'cookie', and we need to predict the output." plus the code. "Input:" and "Output:" are shown for the benefit of the human reader. The CoT does seem pretty nonsensical. It might be an instance of vestigial reasoning: https://www.lesswrong.com/posts/6AxCwm334ab9kDsQ5/vestigial-reasoning-in-rl https://www.lesswrong.com/posts/6AxCwm334ab9kDsQ5/vestigial-... (not to promote my own blog post) I agree Figure 32 is not that concerning - it just says that humans are not that intelligent, which is a little weird, but doesn't indicate that it's plotting against us. It's actually good that we can see this somewhat questionable behavior, rather than it being quashed by process supervision - see https://openai.com/index/chain-of-thought-monitoring/ https://openai.com/index/chain-of-thought-monitoring/

iTokio 1y ago

I skimmed through the paper and the code and got the same conclusion. It’s overhyped, filled with marketing language. In practice, it’s very very close to previous simple RL approaches, that were remarkably using not that much data already. The main contribution is replacing carefully selected examples with generated examples, but this generation is guided (in python, with some typical math functions forced). It’s akin to replacing some manual tests with mutation testing. Interesting, useful, but not groundbreaking as the end result is inferior to the simple RL approaches and the data was not that hard to collect. It is an interesting approach to generalize to other domains where there might be less data available or less easy to curate

robblbobbl 1y ago

Fair enough

CBiddulph 1y ago

I checked Figure 26 - the way it's presented is a bit confusing, but the model prompt doesn't include the expected output. All the model sees is "Here is the function f, the input provided 'cookie', and we need to predict the output." plus the code. "Input:" and "Output:" are shown for the benefit of the human reader. The CoT does seem pretty nonsensical. It might be an instance of vestigial reasoning: https://www.lesswrong.com/posts/6AxCwm334ab9kDsQ5/vestigial-reasoning-in-rl https://www.lesswrong.com/posts/6AxCwm334ab9kDsQ5/vestigial-... (not to promote my own blog post) I agree Figure 32 is not that concerning - it just says that humans are not that intelligent, which is a little weird, but doesn't indicate that it's plotting against us. It's actually good that we can see this somewhat questionable behavior, rather than it being quashed by process supervision - see https://openai.com/index/chain-of-thought-monitoring/ https://openai.com/index/chain-of-thought-monitoring/