3 ms·

I think that’s totally fair! I guess I look at this less as an “ah ha! They’re all cheating!” and more of a “were you guys even aware of what the benchmarks re

by jmye 6mo ago

I think that’s totally fair! I guess I look at this less as an “ah ha! They’re all cheating!” and more of a “were you guys even aware of what the benchmarks represented and how they checked them?”

mzelling 6mo ago
That's a great way to look at it. The paper is a reality check for anyone who thinks of benchmarks as these monolithic, oracular judges of performance. It highlights the soft underbelly of benchmarking.