I suspect dataset contamination is at play here. It fails pretty basic maths questions (not arithmetic, that would be understandable) that surely it should be a

I suspect dataset contamination is at play here. It fails pretty basic maths questions (not arithmetic, that would be understandable) that surely it should be able to do in order to get its benchmark results on MATH. EDIT: Also it hallucinates way more than GPT-4 does. It's possible this is due to a bad system prompt rather than a dumb model.