Why wouldn't we be able to evaluate their performance and compare them to humans? The purpose of test datasets is to do just that, and new ones are created ever

Why wouldn't we be able to evaluate their performance and compare them to humans? The purpose of test datasets is to do just that, and new ones are created every day. By combining several of them, we can create a decent benchmark. We could even include robotic abilities but I think this is not necessary. Let's say: adversarial Turing test + MMLU + coding competence (e.g. AAPS or Leetcode) + ARC (IQ-type test) + Montezuma's Revenge and other games like Stratego or Diplomacy + USMLE (medical exam) + IMO (math) + self driving + ... You can even make it harder: have human judges blindly evaluate new scientific papers in math or theoretical physics for acceptance, see if AI can create highly-rated new apps, write a highly-rated book, compose a hit song...