I think this is a clear case where “human-level” is a totally meaningless expression, since you have to ask about which humans[1]. On one hand, these were CS un

I think this is a clear case where “human-level” is a totally meaningless expression, since you have to ask about which humans[1]. On one hand, these were CS undergrads, so naively I would expect their score to be at least a bit higher than the white-collar worker average. On the other hand, these undergrads weren’t actually familiar with most of the software being tested - surely less familiar than GPT-4! And on the other-other hand I would expect an actual IT professional to do better. There is a difficult conflation with subject-matter knowledge and actual intelligence which makes “human-level” especially problematic. “CS undergrad-level” doesn’t roll off the tongue, but “human-level” seems downright misleading. Nevertheless, this is a striking result, indicating categorical weakness in out-of-distribution reasoning. And I think these tasks being so familiar - real human problem-solving where your ability to think makes up for gaps in knowledge - helps illustrate the problem with transformers: they’re just not smart enough. [1] I get very annoyed with discussions about “AGI” for systems that are totally incapable of solving problems that hunter-gatherers need to solve. But that’s another rant.