I don't know. OpenAI claims that GPT-4 can be in the 89th percentile on the SAT Math exam. At the same time, I asked GPT-4 to compare two six digit number and i

I don't know. OpenAI claims that GPT-4 can be in the 89th percentile on the SAT Math exam. At the same time, I asked GPT-4 to compare two six digit number and in some cases failed. When I asked for a why, it started to make up completely false math rules, e.g. to compare six digit numbers you need to check their 7th digit, which is not just nonsense (where is the 7th digit in a 6 digit number) but shows a complete lack of understanding. So while I believe it can produce "accurate" answer for SAT questions, as it was trained on those, I also believe that it doesn't have any kind of math logic it follows. I'm not sure this dichotomy can be solved with "more" training. It's also telling that in some cases it fails, and in some cases it doesn't, you just need to change the numbers, even with simple numbers.