I have a kind of meta point on this whole discussion. We like metrics because they are not subjective. We don’t like metrics because they tend to be insufficie

I have a kind of meta point on this whole discussion. We like metrics because they are not subjective. We don’t like metrics because they tend to be insufficiently specified optimization problems (I.e. we are looking at 3 variables when we really need to look at 1000+). We dance around this with “Goodhart’s law” type statements, but this is blatantly insufficient. My hypothesis is the brain is obviously working with raw data - photons and sound pressure changes fundamentally and is actually solving the broader optimization problem- albeit subconsciously in the immensely powerful associative cortex. If we were to present the same problem to AI, it would end up being similarly subjective - influenced by its training and structure. In summary, human intuition is currently the most powerful computational device. It should be objectively trusted over and above any gameable metric. Thus only non-gameable metrics should be accepted and intense scrutiny should be place on all metrics used.