4 ms·
I don't know why everyone keeps echoing this, my experience with Deepseek-R1, from a coding perspective at least, has been underwhelming at best. Much better ex
by jjordan 1y ago
I don't know why everyone keeps echoing this, my experience with Deepseek-R1, from a coding perspective at least, has been underwhelming at best. Much better experience with GPT 4.1 (and even better with Claude, but that's a different price category).
- am17an 1y agoI'm not arguing which model is better for your use-case. I'm saying in general as it's "powerful" as GPT 4.1 in a lot of benchmarks, and you can peak underneath the hood, even make it better for your said use-case
- seunosewa 1y agoDo you mean V3? V3 is 4.1 level or above.
- iJohnDoe 1y agoI got the impression that 03-mini or 03-mini-high were meant for coding? GPT 4.1 was meant for creative writing, not coding?
- conradev 1y agoIt’s good at a lot of things: GPT‑4.1 scores 54.6% on SWE-bench Verified, improving by 21.4%abs over GPT‑4o and 26.6%abs over GPT‑4.5—making it a leading model for coding. https://openai.com/index/gpt-4-1/ https://openai.com/index/gpt-4-1/
- coliveira 1y agoThey are trained on these "benchmarks", that's why they score better.
- elzbardico 1y agoIf they were trained on those benchmarks they would score 100%
- coliveira 1y agoThey show how bad they are, they cannot score 100% on benchmarks they were trained on.
- jorvi 1y agoThis is everyone with every model. People sang praise from the roof for Google's Gemini 2.5 models, but in many things for me they can't even beat Deepseek V3.
- CamperBob2 1y agoWhat would be an example of 2.5 Pro failing against R1 (which is what you'd actually want to compare it to)?
- jorvi 1y agoR1 sometimes fails against V3 for me too, so its not a specific dig against Gemini. In terms of code and science, Gemini is way, way too verbose in its output, and because of that it ends up getting confused by itself and hurting the quality of longer windows. R1 does this too, but it poisons itself in the reasoning loop. You can see it during the streaming, literally criss-crossing its thoughts and thinking itself into loops before it finally arrives at an answer. On top of that, both R1 and Gemini Pro / Flash are mediocre at anything creative. I can accept that from R1, since it's mainly meant as more of a "hard sciences" model, but Gemini is meant to be an all-purpose model. If you pit Gemini, Deepseek R1 and Deepseek V3 against each other in a writing contest, V3 will blow both of them out of the water.
- CamperBob2 1y agoAgreed on the last point, V3 is terrifyingly good at narrative writing. And yes, R1 talks itself out of correct answers almost as often as it talks itself into them. But in general 2.5 Pro is an extremely strong model. It may lose out in some respects to o3-pro, but o3-pro is so much slower that its utility tends to be limited by my own attention span. I don't think either would have much to fear from V3, though, except possibly in the area of short fiction composition.
- Zambyte 1y agoIn my experience, all reasoning models feel (vibely) worse at structured output like code versus comparable non-reasoning models, but far better at knowledge-based answering.
- hnfong 1y agoA lot of software (eg. ollama) has confusingly named Deepseek's distill/finetunes of other base models "DeepSeek-R1" as well. See eg. https://www.threads.com/@si.fong/post/DKSdUOHzaBB https://www.threads.com/@si.fong/post/DKSdUOHzaBB I wonder whether you're actually running the proper DeepSeek-R1 model, or one of those lesser finetunes?