3 ms·
This says more about benchmarks than R1, which I do believe is absolutely an impressive model. For instance, in coding tasks, Sonnet 3.5 has benchmarked below
by qeternity 2y ago
This says more about benchmarks than R1, which I do believe is absolutely an impressive model.
For instance, in coding tasks, Sonnet 3.5 has benchmarked below other models for some time now, but there is fairly prevalent view that Sonnet 3.5 is still the best coding model.
- thegeomaster 2y agoLiveBench (which I like because it tries very hard to avoid contamination) ranks Sonnet 3.5 second only to o1 (which is totally expected).
- behnamoh 2y agono, sonnet 3.5 is #7 on LiveBench, even below DeepSeek V3.
- thegeomaster 2y agoThe parent comment was talking about coding specifically, not the average score. I see o1 at 69.69, and Claude 3.5 Sonnet at 67.13.
- sebastiennight 2y agoo1's score looks like exactly what I would expect Elon Musk to aim for with Grok's benchmarks
- parav 2y agoLiveCodingBench has DeepSeekR1 at #3 after O1-high and O1-medium https://livecodebench.github.io/leaderboard.html https://livecodebench.github.io/leaderboard.html
- radu_floricica 2y agoSonnet's strength was always comprehending the problem and its context. It happened to also be pretty good at generating code, but what it actually made it its first really useful model was that it understood _what_ to code and how to communicate.
- Alex-Programs 2y agoExactly - it works better in the real world, where there's a lot less context than a clinical benchmark, and you're just trying to get the answer without writing an essay.
- mordae 2y agoBecause it listens actively and asks questions.