No, but they were good at answering formalized versions of the same word problems. What this tells us is that a 3B LLM can retain enough NLU to understand thos

No, but they were good at answering formalized versions of the same word problems. What this tells us is that a 3B LLM can retain enough NLU to understand those word problems. Which isn't particularly surprising? And also that the same LLM can solve a math or logic problem it understands. Which is a lot more impressive, because early LLMs were already quite good at NLU, but notoriously bad at things like math, logic and iterative problem solving. This 3B model existing tells us we're beginning to figure out how to imbue models with those capabilities reliably.