That is not their reasoning though - it is something they think a human would write given the predicate (question that expects you to provide the reasoning behi

That is not their reasoning though - it is something they think a human would write given the predicate (question that expects you to provide the reasoning behind the answer). For something to reason it needs the ability to have certain goals and to perform action which it thinks are the most optimal to reaching those goals. Like setting hypotheses and producing a path towards proving them - reasoning. The LLM only correlates, so it's "reasoning" is something like "most often people answered 4 to 2+2 then that I should write". That's why it gives out confidently complete gibberish as it works with correlation and not causality. I think much closer to that goal of real reasoning are world models - check out something like DreamerV3 or what Yann Le Cunn is talking about.