2 ms·
It tells me that the benchmark is probably leaking into training data, and going to the benchmark site : > Model was published after the competition date, maki
by rafaelmn 11mo ago
It tells me that the benchmark is probably leaking into training data, and going to the benchmark site :
> Model was published after the competition date, making contamination possible.
Aside from eval on most of these benchmarks being stupid most of the time, these guys have every incentive to cheat - these aren't some academic AI labs, they have to justify hundreds of billions being spent/allocated in the market.
Actually trying the model on a few of my daily tasks and reading the reasoning traces all I'm seeing is same old, same old - Claude is still better at "getting" the problem.