You can tell LLMs to do things or not do things, though, to an extent. Consider how many instructions were in Bing AI’s prompt. [1] It didn’t follow those ins

You can tell LLMs to do things or not do things, though, to an extent. Consider how many instructions were in Bing AI’s prompt. [1] It didn’t follow those instructions perfectly - after all, one of the instructions was to not reveal the instructions - but even in that case, it revealed the instructions because it was instructed to ignore previous instructions. More generally, it followed most of the instructions most of the time. I think I agree with you that ‘once we stop needing RLHF’ is a reasonable line in the sand to draw, but it does feel somewhat arbitrary. LLMs are probably going to get better and better at following instructions, and more generally staying coherent over long sessions, until RLHF starts to be more and more redundant. There won’t be one moment when we can point to a model and say “see, that has agency when the last one didn’t”. Or so I predict, at any rate, since that’s how most of GPT’s progress to date has gone. [1] https://www.theverge.com/23599441/microsoft-bing-ai-sydney-secret-rules https://www.theverge.com/23599441/microsoft-bing-ai-sydney-s...