It failed at around day 3 in advent of code, and could only pass those as they likely had seen the exact solutions to the same, but differently asked questions.

It failed at around day 3 in advent of code, and could only pass those as they likely had seen the exact solutions to the same, but differently asked questions. It really can’t do any novel thought though, which is necessary for mathematics. Language models are fundamentally limited here (even a 3 years old can easily follow some easy rules “ad infinitum”, like counting. Language models have a limited window for inference)