>> I'm pretty sure that's correct, right? So GPT-4 is worse than GPT-3.5 at zero-shot examples of this problem, but better if made to continuously reflect? That

>> I'm pretty sure that's correct, right? So GPT-4 is worse than GPT-3.5 at zero-shot examples of this problem, but better if made to continuously reflect? That's a pretty big result that is totally unexplored. If the correct answer is in the model, it will eventually generate it, if you ask it for enough generations. That's how we understand random performance to work, and not a "pretty big result". The question is whether the model can generate the correct answer consistently and not just at random. More to the point, the question the article above is setting out to answer is whether language models can recognise identifier swaps. If it takes them a lengthy interaction to generate a correct result, this is a pretty strong hint that they can't.