You have to think of the next token prediction as the immediate next 'thought'. We have trained LLMs to be forced to immediately say the right thing. This is a

You have to think of the next token prediction as the immediate next 'thought'. We have trained LLMs to be forced to immediately say the right thing. This is akin to how real humans almost immediately will formulate an answer but then think before responding.. Except imagine if someone had a probe that looked into your mind and made you commit to the first thing you thought. Companies like anthropic literally train their model on the introspected thoughts of the earlier model (constitutional AI). The model is capable of 'thought' if the system doesn't force its hand