4 ms·
I always find it odd when a new benchmark comes out and uses models that are all 6 months - a year old. I’m pretty sure all these models are >1 year old. This
by spott 2y ago
I always find it odd when a new benchmark comes out and uses models that are all 6 months - a year old. I’m pretty sure all these models are >1 year old.
This looks super interesting and useful, but without Claude or o1 data points, it isn’t clear if it is saturated or not, and without more modern open source models, it doesn’t give me a useful signal to choose a model.
- stonogo 2y agoThe point of the paper is introducing the benchmarking tool. It doesn't matter whether they've got the latest hotness for that; the idea is that anyone can then take the tool and use it to evaluate whatever models they care about.
- t-writescode 2y agoThis is an academic paper with 81 sources. Those take a bit of time to write and get approved. This is less of a streamlined benchmark and more of a research paper with a bunch of firsts that had to be designed, standardized and build. A benchmark is more like a program that is run and whose output is evaluated. Those are coming later, but they're not like this. Arguably to make it "scientifically valuable" rather than anecdotally valuable, this sort of work has to come first to figure out what and/or how to benchmark in the first place.