2 ms·
That's the obvious answer, yes. But if they are doing it, why should anyone assume the competition isn't doing it? If it is possible to cheat on the benchmarks
by crote 4mo ago
That's the obvious answer, yes. But if they are doing it, why should anyone assume the competition isn't doing it?
If it is possible to cheat on the benchmarks used to judge AI performance, how can the general population be certain that any of the AI "innovation" is genuine? Is there true development here worth the many-billion-dollar investments, or are we seeing an industry-wide case of them doing a Theranos by faking the results and hoping they can do real innovation before anyone finds out?
- embedding-shape 4mo ago> If it is possible to cheat on the benchmarks used to judge AI performance You make this sound like it's a new thing, but it's been a thing since benchmarks started being a thing in ML, way before LLMs or even before attention was all you needed. The general population, and even less so the developer community, shouldn't blindly follow benchmark scores, they don't really tell you much about how the model is to use in practice anyways. If you something that gives you that idea, you need your own private benchmark with test cases based on what you use LLMs for, and don't share this suite publicly.