That seems to already be happening with o1 and Orion. Instead of rewarding the network directly for finding a correct answer, reasoning chains that end up with

That seems to already be happening with o1 and Orion. Instead of rewarding the network directly for finding a correct answer, reasoning chains that end up with the correct answer is fed back into the training set. That way you're training it to develop reasoning processes that end up with correct answers. And for math problems, you're training it to find ways of generating "proofs" that happen to produce the right result. While this means that reasoning patterns that are not stricly speaking 100% consistent can be learned, that's not necessarily even a disadvantage, since this allows it to find arguments that are "good enough" to produce the correct output, even where a fully watertight proof may be beyond it. Kind of like physicists have taken shortcuts like the Dirac Delta function, even before mathematicians could verify that the math was correct. Anyway, by allowing AI's to generate their own proofs, the number of proofs/reasoning chains for all sorts or problems can be massively expanded, and AI may even invent new ways of reasoning that humans are not even aware of. (For instance because they require combining more factors in one logical step than can fit into human working memory.)