> The current generation of LLMs will never generalize on problems whose expected complexity is significantly harder than the training set. I think it could be

> The current generation of LLMs will never generalize on problems whose expected complexity is significantly harder than the training set. I think it could be done, using same method that has recently been successfully used to distill reasoning power of a large model into a smaller one... The trick would be to self-generate a fine tuning dataset by generating "think step by step" responses to challenging problems. By training on "step by step" output you teach the model how to reason, rather than just what is the desired output (which without step by step might be outside of it's capabilities).