There seems to be some part of the LLM capabilities being lost by hacking some benchmarks. Claude 3.7 is a great example of a model clearly beating 3.5 in all

There seems to be some part of the LLM capabilities being lost by hacking some benchmarks. Claude 3.7 is a great example of a model clearly beating 3.5 in all benchmarks, but slowly destroying my code base by adding lots of extra lines or hacking around my instructions (adding ,,if'' statements when I want it to change the code to handle a case instead of understanding what change is really needed to be done for it). I still prefer o1 pro and a lot of those leapfrogging in benchmarks don't translate to being smarter anymore.