3 ms·
It's a useless benchmark if everyone aces it. If some models do better than others and none saturate it, then is has some value, no? Permitting comparison is th
by andy99 1y ago
It's a useless benchmark if everyone aces it. If some models do better than others and none saturate it, then is has some value, no? Permitting comparison is the point.
- neom 1y agoI agree, hence the heavy couching, so to your point I'm def just ranting a bit, because: I just think it would be more valuable to see some kind of MoA, I guess what I'm talking about is a bit of a different measurement, thinking in terms of economic outlook and understanding where we are and what can be done. I suspect more of this will shape our ability to understand how frontier models will impact the economy. Maybe I should just do my own evaluation, heh. :) Edit: although to argue against myself, I suppose once a model can one-shot this stuff, my MoA comments become moot.