> For example, they are dismal at math problems that aren't just slight variations of problems they've seen before. I know plenty of teachers who would describ

> For example, they are dismal at math problems that aren't just slight variations of problems they've seen before. I know plenty of teachers who would describe their students the exact same way. The difference is mostly one of magnitude (of delta in competence), not quality. Also, I think it's important to note that by "could be solved by high school / talented middle school students" you mean "specifically designed to challenge the top ~1% of them". Because if you say "LLMs only manage to beat 99% of middle schoolers at math", the claim seems a whole lot different.