Has anyone done a benchmark on these reasoning models compared to simply prompting "non-reasoning" LLMs with massive chain of thought? For example, a go to tes

Has anyone done a benchmark on these reasoning models compared to simply prompting "non-reasoning" LLMs with massive chain of thought? For example, a go to test I've used (but will have to stop using soon) is: "Write some JS code to find the smallest four digit prime number whose digits are in strictly descending order" That prompt, on its own, usually leads to an incorrect response with non-reasoning models. They almost always forget the "smallest" part, and give the largest four digit prime with descending digits instead. If I prompt o1, it takes longer, but gives the correct answer. If I prompt DeepSeek R1 with that, it takes a long time (like three minutes) of really unhinged looking reasoning, but then produces a correct answer. Which is cool, but... If I just add "Take an extensive amount of time to think about how to approach this problem before hand, analyzing the problem from all angles. You should write at least three paragraphs of analysis before you write code", then Sonnet consistently produces correct code (although 4o doesn't). This really makes me wonder to what extent the "reasoning" strategies even matter, and to what extent these models are just "dot-dot-dotting"[1] their way into throwing more computation at the problem. Note that an important point in the "dot by dot" paper was that models that weren't retrained to understand filler tokens didn't benefit from them. But I think that's pretty unsurprising, since we already know that models behave erratically when fed extremely out-of-distribution outputs (cf. glitch tokens). So a plausible explanation here is that what these models are learning to do is not output valid reasoning steps, but to output good in-distribution token sequences which give them more time to find the right answer. The fact that DeepSeek's "thinking" looks like what I'd call "vaguely relevant garbage" makes me especially suspicious that this is what's happening. [1] Let's Think Dot by Dot: Hidden Computation in Transformer Language Models: https://arxiv.org/abs/2404.15758 https://arxiv.org/abs/2404.15758