I don't mean to make you write a dissertation but to say that AI benchmarks can "capture properties that generalize from examining performance against a small s

I don't mean to make you write a dissertation but to say that AI benchmarks can "capture properties that generalize from examining performance against a small set of tasks" is a bald assertion. It's a hypothesis without a theory behind it. I can profile some code and then tell you where the slow parts are. Unless there's some explanatory power to an AI benchmark, it's a bit like benchmarking pillows visually.