It’s like an IQ test. I think it’s a good way to probe the “cognition” (if you will) of the model. Sure, humans would probably differ in how they draw that pic

It’s like an IQ test. I think it’s a good way to probe the “cognition” (if you will) of the model. Sure, humans would probably differ in how they draw that picture. IQ 150 humans would almost all get the positional relationships correct though, even if the precise interpretation of “above” might differ. The fact that none of the images were correct under any interpretation of the relationships shows something meaningful, namely that this model can’t understand long chains of relationships.