A pet peeve of mine is that these papers practically never benchmark against humans before they make grandiose claims of how bad these models are. If you tried

A pet peeve of mine is that these papers practically never benchmark against humans before they make grandiose claims of how bad these models are. If you tried to get a human to write out a solution to Tower of Hanoi with a large number of disks, you'd get a whole lot of refusals, and even if you cajoled people into doing it, you'd get a whole lot of errors because people would get sloppy. That said, there were some useful bits - basically the notion that we can't expect to just ramp inference budgets higher and higher for reasoning models and keep getting gains without improving the underlying LLMs too. I just wish they'd been more rigorous, and stuck to that.