> If it is possible to cheat on the benchmarks used to judge AI performance You make this sound like it's a new thing, but it's been a thing since benchmarks s

> If it is possible to cheat on the benchmarks used to judge AI performance You make this sound like it's a new thing, but it's been a thing since benchmarks started being a thing in ML, way before LLMs or even before attention was all you needed. The general population, and even less so the developer community, shouldn't blindly follow benchmark scores, they don't really tell you much about how the model is to use in practice anyways. If you something that gives you that idea, you need your own private benchmark with test cases based on what you use LLMs for, and don't share this suite publicly.