4 ms·
You’re probably correct, but the problem I have is that even a lot of “scientific” benchmarks have similar issues. The whole AI benchmark landscape seems to suf
by stingraycharles 5mo ago
You’re probably correct, but the problem I have is that even a lot of “scientific” benchmarks have similar issues. The whole AI benchmark landscape seems to suffer from similar issues, that benchmarks don’t appear to be matching real world experience (eg open weight Chinese models scoring super high on all benchmarks, while clearly being inferior when using in real life).
- shimman 5mo agoNo offense, but honestly look into who are making these "benchmarks" it's all industry insiders little to no academics to be found. If you listen to actual academics on the subject they are far more realistic than what VC fools will tell you otherwise, which makes sense: one has an incentive to further the knowledge of humanity while the other wants to rat fuck the world to make a quick buck.