Presumably the test designers designed it in a way hoping that if something can pass test X, then it must also be able to do Y, making it a good test of Y abili

Presumably the test designers designed it in a way hoping that if something can pass test X, then it must also be able to do Y, making it a good test of Y abilities. Then someone comes along and figures out how to pass the test without being able to do Y. You see it all the time in ML stuff. "Only a true AI could X" (with an implicit "and that would imply Y") and then people figure out how to build a system that does X but doesn't imply Y. Turns out it's amazingly hard to build good standardize tests that don't fall into "When a measure becomes a target, it ceases to be a good measure."