AI companies love standardized tests because that's what LLMs are really good at: short problems with objective solutions. LLMs do worse on problems that are mo

AI companies love standardized tests because that's what LLMs are really good at: short problems with objective solutions. LLMs do worse on problems that are more open-ended, like SWE-bench, which GPT-4 achieved a whopping 1.74% on.