3 ms·
Agreed, but this is hardly limited to AI benchmarks. All benchmarks suffer from this. It’s almost cute the researchers have discovered that AI is not some wildl
by brookst 2y ago
Agreed, but this is hardly limited to AI benchmarks. All benchmarks suffer from this. It’s almost cute the researchers have discovered that AI is not some wildly different thing.
- voxl 2y agoIt might behoove the reader to remind themselves that goodhearts law is a conjecture, not a law. Similar to Poe's Law, it's an observation about something, but that doesn't make it universal, or some kind of physical, social, or economic law. In other words, you don't get to assume that the tests aren't measuring the right thing, even if AIs are being designed to do well on the tests. That is a claim that demands evidence.