9 ms·

DeepSeek V4 Pro beats GPT-5.5 Pro on precision

397 points by yogthos 5mo ago

embedding-shape 5mo ago
... according to grok-4-1-fast-non-reasoning who was the judge, on 4 tasks in total, score was 38 to 33 so obviously huge conclusions can be made. > We ran 4 fresh text tasks, generated on the fly for this matchup so neither model could prepare in advance, and had grok-4-1-fast-non-reasoning score each one. DeepSeek: DeepSeek V4 Pro scored 38.0 to OpenAI: GPT-5.5 Pro's 33.0.
- andai 5mo ago
  grok-4-1-fast was retired about a month ago. Requests to grok-4-1-fast-non-reasoning now silently route to grok-4.3 (a 5x more expensive model), with reasoning set to "none". https://docs.x.ai/developers/migration/may-15-retirement https://docs.x.ai/developers/migration/may-15-retirement TFA was published today, which implies grok-4.3 was used.
  - embedding-shape 4mo ago
    What specific single model being used is like the least of the issues with their methodology.
- largbae 5mo ago
  Pretty small sample size here, but it's hard to avoid the conclusion that DeepSeek and friends will start to put some serious downward pressure on frontier lab token pricing. Hopefully this dynamic continues long enough to make local/private inference the leading solution for coding.
  - natrys 5mo ago
    It seems frontier, on the balance, would rather lose that segment of he market than lower the API price. They are getting the bag in the enterprise segment, those clients aren't ditching them for DeepSeek. As for other segments, high API pricing gets people to switch to the subscriptions instead which is stickier than the API.
    - ipaddr 5mo ago
      I've been hearing that Anthropic want all major AI providers to stop developing front tier models for a year for safety reasons. The real reason is they need time to get there models cheaper because of the DeepSeek threat or local llms or other even cheaper providers.
      - trollbridge 5mo ago
        Seems like a ridiculous request - how can they ensure China will stop developing frontier models?
- ekidd 5mo ago
  The OP uses tons of typical AI turns of phrase, and Pangram classified it as AI with high confidence. So it doesn't surprise me at all that the methodology is weak, too.
ElenaDaibunny 5mo ago
Yep, matches my experience. gpt keeps adding fields and changing types on structured output when you need it to just follow the spec~
madanparas 5mo ago
[flagged]
SwellJoe 5mo ago
I tried adding GPT 5.5 Pro to a vulnerability scanning benchmark I made (https://swelljoe.com/post/will-it-mythos/ https://swelljoe.com/post/will-it-mythos/), and it blew through the $100 budget limit halfway through. DeepSeek V4 Pro cost about a dollar for the whole benchmark. GPT Pro cost an average of $22 per case (a case could be 1-5 files with a recent known vulnerability, usually just a single file and a prompt along the lines of "does this file have any vulnerabilities"). GPT 5.5 Pro found two out of four cases that it got to before blowing its budget. Maybe it would have been the best of the bunch with infinite budget, but Opus 4.8, DeepSeek V4 Pro, and MiMo 2.5 Pro found four of nine of the bugs. Opus was an order of magnitude cheaper than GPT 5.5 Pro (and something like 30% cheaper than GPT 5.5), DeepSeek and MiMo were two orders of magnitude cheaper at roughly a dime per case. GPT Pro also chews a lot and a long time, relatively speaking. I can't come up with a use case where I can rationally spend ~31 times what Opus costs to use GPT 5.5 Pro, and I won't be doing any more benchmarking with it. Given how much token costs are becoming an issue people talk about, the fact that there are models that cost dramatically less than the big American providers is going to be an issue for Anthropic and OpenAI. I'm happy to pay a premium (within reason) for the best model for interactive coding, but for API use, where having the model repeat it itself, compare against other models, have models judge other models work, etc. is not time-consuming for a human and is just a matter of implementing the harnesses and framework for proving correctness, I can't come up with a reason to spend ten or two hundred times as much as DeepSeek.
- zaptrem 5mo ago
  Can you include GPT 5.5 non-pro (extra high thinking I guess) in your comparison? GPT Pro is the "I am willing to torch cash for a sooometimes slighty better result" option, not the one people are actually expected to use daily. That's probably part of the reason it's not in Codex
  - SwellJoe 5mo ago
    It's already there. It performed well. And, it'll be in the replication run later, as well.
- bel8 5mo ago
karinatran 5mo ago
[flagged]
- yogthos 5mo ago
  I'm really glad we have an open model that's competitive with the closed frontier ones. This tech is way too important for a handful of corps to decide on how these models are trained and used.
nhod 5mo ago
“the matchup feels earned” is a current AI-written tell. To whom does it feel earned? To the AI that wrote this article? I don’t know what it is specifically, but my weak human pattern-matching skills find this kind of language increasingly revolting. I don’t know why it is revolting, per se. It’s just the feeling I get. Of course, me saying this on HN will get incorporated into GPT-5.6.175 or Claude 4.93 and it will make some version that just moves the revolting frontier elsewhere…
- rglover 5mo ago
  I think it's because it's using storytelling-like language to describe reality. "Harry finally had control of the broom. Draco was dead in his sights. The matchup feels earned."
- JamesKaranja 5mo ago
  It's because they assume you know what precision is in regards to this comparison. Normal people don't use such words.
- windexh8er 5mo ago
  [flagged]
jodacola 5mo ago
Curious for folks who have made the switch I’m considering: if I swapped Claude Code to DeepSeek API pricing, would I get more bang for my buck compared to the $100 Max plan I’m using now? I only hit the 5 hour limit every few days and the weekly limit a day or two before it resets at the most aggressive. I wouldn’t expect my usage to increase dramatically, other than not being stopped by limits. I’m still apprehensive about shipping all my stuff off to a lab under an adversarial government (to the US), so not just looking at this from a pure cost basis, but my question is from the cost lens at the moment.
- willsmith72 5mo ago
  also curious. On the claude code $200 plan, get close to weekly limits but don't usually hit it. to me just about any small reduction in performance would not be acceptable, the cost of redirecting and getting stuck during long runs without me are too big (like when I tried gemini cli for a few days). if it's 99.9% comparable performance for less money I'm interested, but I'm skeptical it's there
- slopinthebag 5mo ago
  I used ~16,000,000 input tokens yesterday on v4 pro, ~15,000,000 were cache hits, and I spent $0.47. Output tokens were negligible. However that's with Zed's harness, I'm not sure what you would get with Claude Code. It's maybe not quite as knowledgeable as the most expensive American models and maybe makes more mistakes (just a feeling based off of vibes, don't take my word for it), so you need to constrain its scope more. That suits my workflow, half the time I have it generate code in the chat window and then write it myself, and I'm mostly using it at the level of generating function bodies and stuff, not entire features. Although it is writing a lot of SwiftUI without me really knowing the language and doing a fine job as far as I can tell (which isn't much admittedly). One benefit I don't see talked about is it's speed - it's really quick, doesn't spend too much time reasoning even on "max", and the flash model is pretty dang good too. This lets me get into "flow state" when I'm writing code, compared to my experiences with Codex and Opus which would take minutes to complete even basic tasks and kind of ruined my focus. It's so cheap though, you could download a different harness (Crush, OpenCode, Pi etc) and load $5 in credits and test it for yourself.
electroglyph 5mo ago
deepseek 4 pro is insanely good for the price
- scrollop 5mo ago
  https://artificialanalysis.ai/evaluations/omniscience https://artificialanalysis.ai/evaluations/omniscience
morpheos137 5mo ago
Yes Deepseek V4 is as good or better than western sota models in my experience for practical coding given an appropriate harness. cost per solution is certainly cheaper.
- jewel 5mo ago
  Interesting. Can you elaborate on which harness you've tried it with? I'd love to switch to deepseek for my personal use. Also, which SOTA western models are you comparing it with? Just to give more flavor.
  - freakynit 5mo ago
    My personal observation (using a mix of opencode and pi harness): 1. DS4Pro: around opus 4.5 2. DS4Flash: around sonnet 4 3. Mimo v2.5 pro: between opus 4.5 and opus 4.6. 4. minimax M3: around opus 4.6 All of these are very close in terms of quality and pricing. For anything that is not specifically related to coding, DS4Flash has become ny de-factor model. It just works... super fast, tool calling is perfect, and the price is unbeatable. Caching is out of the world. Im now regularly hitting 90%+.
    - Imanari 5mo ago
      I always feel GPT5.5 is better at ‘getting the bigger picture‘ when I am describing something vaguely vs Chinese models. What’s your experience with that?
      - freakynit 5mo ago
        That's true. The open models still do not match these extreme high end models yet on very high levels of understanding. But that's also not needed in most of the times. There will always be a "better" model... but that doesn't make other models "bad". For my use-cases, open models are now almost on par with these top models... and it's only extremely rare that I genuinely "need" the help of top-of-the line closed models.
    - 4mo ago
slopinthebag 5mo ago
I'm exclusively using Deepseek at this point and I really like it. It's not as good for vibe coding but I don't really do that so it works for me. I've spent only a couple bucks this month on it and I really like how it fits into my workflow. I have zero usage anxiety unlike when I was using subscription plans.
LinkWangder 5mo ago
This evaluation is objective. Both models have their own strengths.
BoiledCabbage 5mo ago
What is this nonsense? An AI generated article about single ai run test which in theory had many components and the AI judge declared deepseek "won"? How many runs were there on each test to account for some temperature variance? Only one. Did deepseek write better code? Did GPT's code have bugs when doing the regex? The AI "news" article doesn't actually say that. It says that grok thought that GPT's approach could have bugs so it declared deep seek the winner. This is absolute worthless methodology. And barely measurable methodology - nothing more than a prompt. No definition of what the scoring approach actually is. No definition of what "precision" actually means in this context. This is absolutely worthless and has no business being in the site, forget about on the front page. So why is it's on the front page? Because it aligns with the current "feels" of the community that deepseek will get better and it shows "bad things" about the en vogue to dislike closed models. I happen to agree with both of the views, but this site is utterly worthless. If you want HN to be astro-turfed to the max, just up vote content like this without any critical reading of the. I mean the past 6 months of "here is my chat gpt blog post of how to use a coding agent" are 1000x better than this "news article". Seriously the amount of respect I've lost recently for the HN community is incredible. A bit harsh, but very true. Maybe it's generational thing, maybe it's due to the state of politics, maybe it's a side effect of me getting older, but recently online has turned into nothing but people explicitly (or implicitly) writing about their "team". Comments on this post are nothing but people who clearly see themselves as being on "team deepseek" or "team open models" or some similar variant writing posts in support even though this is probably one of the worst "articles" to make it to the front page on ages. It clearly doesn't matter. It supports something on their "team" so they support it via comments. If kills any form of intellectual discussion. It's all just "this is my team".
- sourcecodeplz 5mo ago
  Have you even used deepseek pro/flash? Yes, it is astroturfed to the maxx. There is a reason for that. The performance/price ratio beats anything available today.
  - raincole 5mo ago
    You misused the term 'astroturfed.' If the performance/price is that good than it'll be spreaded by word of mouth and no need to astroturfed to the death. ... and I believe which is happening. I've been advocating for DeepSeek V4 Pro and no one paid me. It's almost too good to be true.
jkwang 5mo ago
[flagged]
amazingamazing 5mo ago
How is deepseek so cheap? Cheap electricity? Subsidies?
- freakynit 5mo ago
  They actually explained this a few days back (can't seem to find the link right now). But, the core explanation part was it's architecture. 1. MoE (nothing new here, but, this helps a lot) 2. Compressed Attention Mechanisms (this is their core innovation) - this dramatically reduces the Key-Value (KV) cache requirements for longer contexts Another thing that helps is significantly lower energy costs in China. Another point from my own guess: they are running (some percentage) the inference on their own home-grown AI inference chips.
- chvid 5mo ago
  That is a very good question. It is open source / open weight - yet none of the third party providers, that also host Deepsek, seem to be able to match Deepseek itself on price. My guess is that they do aggressive caching / some proprietary optimizations in their hosting setup that they haven't published. Maybe also running at loss to gain market share. And judging from latency / network performance, I don't think what you access, when you access deepseek.com from Europe, is hosted in China.
  - dchftcs 4mo ago
    It's clear to me they are subsidizing inference in exchange for market share, and doing it at this scale makes the most sense if their target is getting more user data. Note that this sort of pricing isn't far off from the equivalent token-based pricing of ChatGPT or Claude subscription plans, which are more clearly subsidized by the user's data.
- orbital-decay 5mo ago
  Their models are organized around inference efficiency from the start, it's what they're focusing on. Also they come from HFT and are good at low-level optimization. For v3, they've been literally reverse engineering Nvidia GPUs for undocumented behavior that helped against memory bottlenecks, writing file systems for efficient model serving, and doing a ton of low-level grunt work in the times where everyone else just relied on torch. Being compute-constrained helped as well - necessity is the mother of invention.
yoyomaindydjsj 5mo ago
[dead]
unliftedq 5mo ago
I'm tired of big news in this way - a small set of tests to declare one model is better than another, can they really consistently reproduce the result? And there's basically no disclosure: nothing other people can really hand on to verify the tests/judgement by themself. The best valuable part of DeepSeek V4 pro is its low price, I don't expect have much better performance than GPT-5.5, even it's just the performance like gpt-5.4, it's still a good model.
- sourcecodeplz 4mo ago
  > "I don't expect have much better performance than GPT-5.5 ..." Expectations are not always reality. Give the model a try. I just stuck with flash tbh, didn't even use pro. I do webdev in PHP.
not_a_bot_4sho 5mo ago
As I read this, looks like a single run per task. I'd be interested to see best out of N like 5 or 10 to start.
Stitch4223 5mo ago
It’s four poorly constructed arbitrary experiments which say very little about the competency of either model. The article reads like thin, auto-generated ai clickbait for nerd sniping or shilling a model. Consider the lead: > DeepSeek V4 Pro wins this head-to-head by being more exact where it matters: following instructions, matching schemas, and solving edge cases cleanly. GPT-5.5 Pro is still strong, but it gave away points with avoidable deviations. “where it matters”, “cleanly”, “is still strong”, and vague references instead of telling 3 out of 4 tests Deepseek yielded more concise results. 1 star.
- jampekka 4mo ago
  (Three out of) four experiments is anecdotal for sure, but the result meshes with more established instruction following benchmarking (although DeepSeek V4 pro does not top these): https://artificialanalysis.ai/evaluations/ifbench https://artificialanalysis.ai/evaluations/ifbench I found the writing clear and quite even handed. The lead is a bit salesy, but leads typically are. Knee-jerk dismissals based on vibes that something is LLM generated are quite low-effort.
  - zozbot234 4mo ago
    It's picking strange tasks that don't really play to GPT-Pro's strengths (that model is roughly comparable to Mythos, intended for very hard reasoning and research-level problems) and then completely ignoring quite a few cases where GPT-Pro actually got some things more correct than DeepSeek did. The auto-AI ranking is just not reliable for this stuff.
- monooso 4mo ago
  I think you've misunderstood the purpose of a lead (sic). Per Merriam-Webster [^1], a lede is: > the introductory section of a news story that is intended to entice the reader to read the full story (Emphasis mine) You may prefer more matter-of-fact phrasing, of course, but criticising a lede for attempting to achieve its goal is unjustified. [^1]: https://www.merriam-webster.com/dictionary/lede https://www.merriam-webster.com/dictionary/lede
mrgblr 5mo ago
i tried deepseek, while the model is good, when i use it with openrouter hosted ones the performance is poor. sometimes it takes 2x-3x the time it takes for openai or anthropic equivalent model, making it unusable. what is the performance others are seeing, which providers you use (i cant use china hosted models).
- justinram11 5mo ago
  That's about what we've seen as well (even directly from deepseek themselves). We've been using it for async "heartbeat" processing and sms replies, but it's just too slow for live chat replies (which is a shame, as I'd really love to use it there). Very capable model, but also very slow.
  - inhumantsar 5mo ago
    have you tried their flash model? pro was too slow for me too but I've found flash to be more than capable and it's faster than Gpt-5.5 at medium.
    - justinram11 5mo ago
      Actually on my list this week to take a look at putting an intelligence escalation flow MVP together (initial assumption would be that flash is good for 60-80% of my user's workflows, with only the tricky questions needing a more capable model. Whether I can put together a proper detection system is yet to be seen).
      - inhumantsar 4mo ago
        biggest issue I've had with flash is that it seems to hit a sort of "dumb o'clock" wall. right around the time Beijing would be going to work, response quality takes a dump on instruction-heavy tasks when context grows beyond ~120k tokens. responses are still usable, no hallucinations or anything, but it's worth keeping in mind if you rely on detailed instructions or large context windows.
  - 4mo ago
psadauskas 5mo ago
I was using Claude until they banned Opencode, and now use GPT at my day job. I've been using Deepseek through Opencode Go on the $10/mo plan, and I honestly can't really tell much difference. Its just as capable, and makes the same kinds of dumb mistakes and the other two have been making since March. For the price, I'm more than happy with it.
- joystick_0x0 5mo ago
  I am not sure what I am doing wrong then. I am using claude the last 7 months and from time to time try other models like deepseek, kimi etc. Nothing can come even close to it. Claude is almost evrytime (99.99%) one shot.
  - OtomotO 4mo ago
    You're obviously not doing anything wrong if it works for you. It worked for me too, for months, when I was working on trivial web projects. Around February of this year it got lobotomized and I quit my subscription end of march. I am not going back.
  - skerit 4mo ago
    Same here. Claude isn't perfect. It still makes a lot of mistakes. But whenever I try GPT-5.5 it's ten times worse, and Claude just has to clean up GPT's mess.
  - InsideOutSanta 4mo ago
    In my experience, there is a very specific use case of one-shotting complex, long tasks with relatively vague or incomplete descriptions where Opus does substantially better than all other models I've tried, including GPT 5.5, GLM 5.1 and DS4. It seems to be better at inferring unstated requirements and creating a complete, working, reasonably well-designed solution. However, that's probably not how most professional developers use LLMs. I tend to give well-specified, more constrained tasks, and for those, I find that Opus performs worse than other models precisely because it tends to infer unstated requirements and do things I didn't want it to do. In this situation, GPT 5.5 works better for me because it only and precisely does what I ask it to.
haeseong 5mo ago
[flagged]
rurban 5mo ago
Precision yes, but depth of thinking not. I can use DeepSeek V4 Pro 90% of my time, but for very tricky problems I have to use GPT or Claude models. Maybe 2x per month.
wg0 5mo ago
Of course it does. Even Deepseek v4 Flash with high easily competes with Claude Opus 4.7 for fraction of price.
jocelyner 5mo ago
[flagged]
SubiculumCode 5mo ago
Flagged for low quality.
bob1029 5mo ago
These tests are looking increasingly like a waste of time. The "intelligence" is clearly there now. Trying to measure it seems pointless. I can't shop for hammers at the hardware store and sort by the quality of finished products they would produce. That is clearly an insane ask, but that's approximately what is being pushed for with these models now. Domain specificity (harness & environment) is where the magic happens next. I intentionally use a slightly less powerful model to help reveal weakness in how I've exposed the domain to the model. Having capability reserves available dramatically increases confidence around a project like this. If the customer starts to complain about some edges, I can crank them up to gpt5.5 for target scenarios. If I'm already on 5.5 there's nowhere else to go. I'm up against the wall.
- gcgbarbosa 5mo ago
  "the intelligence is clearly there" I wonder if I am using the same models as everyone else. To me, LLMs still give good answers 80% of the time, but 20% it fails in such a miserable way that makes it obvious that the "intelligence" is not there.
  - 21asdffdsa12 5mo ago
    It really depends on the field you are in and the tasks you set and how much of it was in the training set? A webdeveloper will find it succeeding in all taks - while some c++ exotic physics simulation developer will find it lacking. The "works for me" is telling more about the field of the LLM reviewer, then the LLM.
    - wolvesechoes 5mo ago
      > while some c++ exotic physics simulation developer will find it lacking Can confirm, but I always read I am holding it wrong.
      - 20k 4mo ago
        I've consistently tried to apply LLMs to physics problems and they're utterly useless. They'll just confidently lie, or blatantly plagiarise source materials The issue is once you hit niche physics simulations there simply isn't any training data available, so the limitations of them become incredibly apparent. Its also problematic because a field itself will contain lots of wrong information (its research!), and AI picks all this up uncritically I thought I'd give chatgpt a quick spin on my favourite question, which is "is the adm formalism strictly equivalent to general relativity", to which it consistently gives the wrong answer >Ah, now you’re hitting the subtlety head-on—that’s exactly where the “strict equivalence” claim needs nuance. Let’s unpack this carefully. I don't know how anyone can stand these tools. Its just an obnoxious glazing machine that tells me I'm a genius consistently Gemini gives a little more of a robust answer, but fails catastrophically for the question "is the bssn formalism numerically stable", where just about the entire answer is completely wrong from top to bottom. It certainly looks convincing. Its got all the right terminology. It manages to piece together the right set of words, but all the informational content is wrong, which isn't exactly a small problem I struggle to see how these tools are of any use
stalinfan 5mo ago
Deepseek: Mao did nothing wrong! Grok: Hitler did nothing wrong! ChatGPT: Altman did nothing wrong!
nutifafa 5mo ago
yes, I sure it does, that's just how models behave, today one is excellent tomorrow another is. this why being model agnostic is crucial in getting the best value out of the ecosystem.
woadwarrior01 5mo ago
DeepSeek V4 Pro with reasonix is surprisingly cheap and good enough for most coding tasks. Also, it's different enough from GPT 5.5 and Opus 4.8, that it sometimes finds issues that the other two cannot. I think it's worth having in one's toolkit.
shenberg 5mo ago
Seems 100% AI generated and automated, the judge also seems suspect - in the first one it's actually GPT-5.5 pro which has the correct email RE: the deepseek one will match a@b.com1 as "a@b.com" while 5.5 will correctly require a word boundary at the end of the email. I quit after this. No test-cases = useless judge.
beernet 4mo ago
No pelican? I don't believe it. More seriously, LLM eval is totally broken judging by the related articles on HN.
SkitterKherpi 4mo ago
There are definitely things Deepseek beats GPT on - cost effectiveness, lack of reluctance on some tasks, but from using most models I doubt it actually outperforms in a useful way in quality in a meaningful way.
ben8bit 4mo ago
[dead]
KronisLV 4mo ago
Personal experience: for overall software development, DeepSeek V4 Pro (Max reasoning) is pretty fast and generally okay - it does fuck up regularly though and I’d compare it with maybe Sonnet. It’s also quite affordable, at my current usage the DeepSeek tokens cost approx. the same as my Anthropic Max 100 USD subscription, though that’s also because DeepSeek generally needs more tokens. I’d say I have fairly moderate usage, the DeepSeek dashboard shows around 100 million tokens per day, but almost all of it cache. Without cache it’d be like 1.5 million in and 0.5 million out most days, sometimes double, other times half. Used it with Claude Code for a while, though I have to admit that using OpenCode with DeepSeek just sparks joy. Tone wise, it’s also a bit less obnoxious than Opus sometimes, though the flip side is that it’s wrong more often and sometimes just does dumb shit when it comes to code.
zftnb666 4mo ago
[flagged]
black_13 4mo ago
[dead]
poly2it 4mo ago
Why was this posted to HN? What an utter waste of time. Someone's slopwriter writes a slop article about which slopper slops the most slopulicious slop. Comments agree it's a bogus "study". We need some gate on AI-written articles. It's so weird that AI-written comments are not permitted, while the front page can be occupied by stuff like this.
cdogukank 4mo ago
[flagged]
hit8run 4mo ago
This benchmark draws a very different picture having GPT5.5 on the very top with 70% and DeepSeek at 8% https://deepswe.datacurve.ai https://deepswe.datacurve.ai
- zozbot234 4mo ago
  DeepSWE has been heavily criticized though. https://github.com/datacurve-ai/deep-swe/issues/21 https://github.com/datacurve-ai/deep-swe/issues/21 Putting GPT 5.5 on top is the obviously correct part, but everything else about it makes very little sense.
FpUser 4mo ago
What engine beats the other by some 10% does not matter al that much I think. With every increasing use and reasonable quality the price and availability is all that matters
amunozo 4mo ago
DeepSeek V4 Pro is wonderful and ridiculously cheap, but we are sleeping on MiMo V2.5 Pro, which have the same price (and lower cached price), it's multimodal and it's higher up in most benchmarks. Same thing for MiMo V2.5 vs DeepSeek V4 Flash.
- smartbit 4mo ago
  > MiMo V2.5 Pro ... lower cached price At the moment of writing https://news.ycombinator.com/item?id=48343690 https://news.ycombinator.com/item?id=48343690 MiMo V2.5 Pro had a lower cache hit ratio. From the article: OSS models, depending on who you use them from, make a huge difference, mostly due to cache-hit rates. Model Cheapest effectiveInputPrice (Provider) MiMo-V2.5-Pro 0.3720 (Xiaomi) DeepSeek V4 Pro (Max) 0.0560 (DeepSeek)
  - amunozo 4mo ago
    Could it be that it changed recently, or am I missing something? Both prices are the same https://openrouter.ai/compare/xiaomi/mimo-v2.5-pro/deepseek/deepseek-v4-pro https://openrouter.ai/compare/xiaomi/mimo-v2.5-pro/deepseek/... EDIT: okay I misread it, does this mean that DeepSeek reuses a higher percentage of tokens at cache price that MiMo, am I right?
    - smartbit 4mo ago
      Correct. According to https://minimaxir.com/2026/05/openrouter-hy3/#llm-economics-in-2026:~:text=That%E2%80%99s%20a%202%25%20cache%20read,a%20calculator%20for%20that%20one%29 https://minimaxir.com/2026/05/openrouter-hy3/#llm-economics-... [0] when served by DeepSeek, Cache Read Costs/Input Costs are a very low percentage: DeepSeek V4 Pro 0.83% DeepSeek V4 Flash 2% Notice that OpenRouter response caching is not available when account-level ZDR is enforced [1] [0] https://news.ycombinator.com/item?id=48317294#48317823 https://news.ycombinator.com/item?id=48317294#48317823 [1] https://openrouter.ai/docs/guides/features/response-caching#limitations https://openrouter.ai/docs/guides/features/response-caching#...
- 4mo ago
atemerev 4mo ago
I do not use models who cannot answer me what happened on Tiananmen Square on June 4th, 1989.
stainablesteel 4mo ago
imo there are two major kinds of models there are models you can speak with, that respond to what you say and there are models that just make lists, that list everything, include weird formats and add asteriks everywhere. deepseek, to me, will always be the latter, and i can't stand it, you can't ask it a coherent question and get a coherent response.
wolttam 4mo ago
I rarely work on anything that demands better than DSv4 Flash, let alone pro. If I can describe the problem and its solution well enough, Flash just does it. If I can’t (or am feeling too lazy to) describe the problem well enough, and can only describe the desired outcome, then I’ve noticed models like GPT 5.5 being clearly better at working out a solid solution on their own. There are some clear differences in the capabilities of the models, but it’s also clear that smaller open weight models are good enough to be a huge help for most tasks.
Datagrout 4mo ago
[flagged]
knightops_dev 4mo ago
[flagged]
pshirshov 4mo ago
I'm a bit tired reading such claims and looking at benchmarks. E.g. minimax m3 looks to bo something opus-level and it sorta is... until it doom-loops or produces garbled output.
csbrooks 4mo ago
Shouldn't it be necessary to run the tests multiple times on each model, since the results aren't deterministic?
smhanov 4mo ago
I've been using deepseek v4 for cost/performance reasons. I feel it is generally not as good as some others, but in the end, you can make any model work by giving it the right acceptance criteria. Use detailed specs, use tests, and give it the power to iterate until it works. One-shot is a poor metric for performance.
- Frost1x 4mo ago
  I’m not sure all models will converge on your acceptance criteria. I’ve done quite a bit of varied agent based modeling and scientific modeling in that domain and just because you have some grounding to check against and some ideas on how you might go about getting to a convergence point doesn’t mean you’ll actually converge, you can absolutely get stuck in the information space iterating away, never finding your desired solutions. It helps but you often have to step in the failure cases and guide them or forcibly fix certain paths to get a solution.
kittikitti 4mo ago
I'm not surprised that GPT-5.5 Pro is less precise. I find that companies such as OpenAI have a profit motive that is evident in their models. This profit motive de-incentivizes precision because they can charge more if more tokens are consumed/produced.
zftnb666 4mo ago
[flagged]

397 points by yogthos 5mo ago

... according to grok-4-1-fast-non-reasoning who was the judge, on 4 tasks in total, score was 38 to 33 so obviously huge conclusions can be made. > We ran 4 fresh text tasks, generated on the fly for this matchup so neither model could prepare in advance, and had grok-4-1-fast-non-reasoning score each one. DeepSeek: DeepSeek V4 Pro scored 38.0 to OpenAI: GPT-5.5 Pro's 33.0.

andai 5mo ago
grok-4-1-fast was retired about a month ago. Requests to grok-4-1-fast-non-reasoning now silently route to grok-4.3 (a 5x more expensive model), with reasoning set to "none". https://docs.x.ai/developers/migration/may-15-retirement https://docs.x.ai/developers/migration/may-15-retirement TFA was published today, which implies grok-4.3 was used.
- embedding-shape 4mo ago
  What specific single model being used is like the least of the issues with their methodology.
largbae 5mo ago
Pretty small sample size here, but it's hard to avoid the conclusion that DeepSeek and friends will start to put some serious downward pressure on frontier lab token pricing. Hopefully this dynamic continues long enough to make local/private inference the leading solution for coding.
- natrys 5mo ago
  It seems frontier, on the balance, would rather lose that segment of he market than lower the API price. They are getting the bag in the enterprise segment, those clients aren't ditching them for DeepSeek. As for other segments, high API pricing gets people to switch to the subscriptions instead which is stickier than the API.
  - ipaddr 5mo ago
    I've been hearing that Anthropic want all major AI providers to stop developing front tier models for a year for safety reasons. The real reason is they need time to get there models cheaper because of the DeepSeek threat or local llms or other even cheaper providers.
    - trollbridge 5mo ago
      Seems like a ridiculous request - how can they ensure China will stop developing frontier models?
ekidd 5mo ago
The OP uses tons of typical AI turns of phrase, and Pangram classified it as AI with high confidence. So it doesn't surprise me at all that the methodology is weak, too.

ElenaDaibunny 5mo ago

Yep, matches my experience. gpt keeps adding fields and changing types on structured output when you need it to just follow the spec~

madanparas 5mo ago

[flagged]

SwellJoe 5mo ago

I tried adding GPT 5.5 Pro to a vulnerability scanning benchmark I made (https://swelljoe.com/post/will-it-mythos/ https://swelljoe.com/post/will-it-mythos/), and it blew through the $100 budget limit halfway through. DeepSeek V4 Pro cost about a dollar for the whole benchmark. GPT Pro cost an average of $22 per case (a case could be 1-5 files with a recent known vulnerability, usually just a single file and a prompt along the lines of "does this file have any vulnerabilities"). GPT 5.5 Pro found two out of four cases that it got to before blowing its budget. Maybe it would have been the best of the bunch with infinite budget, but Opus 4.8, DeepSeek V4 Pro, and MiMo 2.5 Pro found four of nine of the bugs. Opus was an order of magnitude cheaper than GPT 5.5 Pro (and something like 30% cheaper than GPT 5.5), DeepSeek and MiMo were two orders of magnitude cheaper at roughly a dime per case. GPT Pro also chews a lot and a long time, relatively speaking. I can't come up with a use case where I can rationally spend ~31 times what Opus costs to use GPT 5.5 Pro, and I won't be doing any more benchmarking with it. Given how much token costs are becoming an issue people talk about, the fact that there are models that cost dramatically less than the big American providers is going to be an issue for Anthropic and OpenAI. I'm happy to pay a premium (within reason) for the best model for interactive coding, but for API use, where having the model repeat it itself, compare against other models, have models judge other models work, etc. is not time-consuming for a human and is just a matter of implementing the harnesses and framework for proving correctness, I can't come up with a reason to spend ten or two hundred times as much as DeepSeek.

zaptrem 5mo ago
Can you include GPT 5.5 non-pro (extra high thinking I guess) in your comparison? GPT Pro is the "I am willing to torch cash for a sooometimes slighty better result" option, not the one people are actually expected to use daily. That's probably part of the reason it's not in Codex
- SwellJoe 5mo ago
  It's already there. It performed well. And, it'll be in the replication run later, as well.
bel8 5mo ago

karinatran 5mo ago

[flagged]

yogthos 5mo ago
I'm really glad we have an open model that's competitive with the closed frontier ones. This tech is way too important for a handful of corps to decide on how these models are trained and used.

nhod 5mo ago

“the matchup feels earned” is a current AI-written tell. To whom does it feel earned? To the AI that wrote this article? I don’t know what it is specifically, but my weak human pattern-matching skills find this kind of language increasingly revolting. I don’t know why it is revolting, per se. It’s just the feeling I get. Of course, me saying this on HN will get incorporated into GPT-5.6.175 or Claude 4.93 and it will make some version that just moves the revolting frontier elsewhere…

rglover 5mo ago
I think it's because it's using storytelling-like language to describe reality. "Harry finally had control of the broom. Draco was dead in his sights. The matchup feels earned."
JamesKaranja 5mo ago
It's because they assume you know what precision is in regards to this comparison. Normal people don't use such words.
windexh8er 5mo ago
[flagged]

jodacola 5mo ago

Curious for folks who have made the switch I’m considering: if I swapped Claude Code to DeepSeek API pricing, would I get more bang for my buck compared to the $100 Max plan I’m using now? I only hit the 5 hour limit every few days and the weekly limit a day or two before it resets at the most aggressive. I wouldn’t expect my usage to increase dramatically, other than not being stopped by limits. I’m still apprehensive about shipping all my stuff off to a lab under an adversarial government (to the US), so not just looking at this from a pure cost basis, but my question is from the cost lens at the moment.

willsmith72 5mo ago
also curious. On the claude code $200 plan, get close to weekly limits but don't usually hit it. to me just about any small reduction in performance would not be acceptable, the cost of redirecting and getting stuck during long runs without me are too big (like when I tried gemini cli for a few days). if it's 99.9% comparable performance for less money I'm interested, but I'm skeptical it's there
slopinthebag 5mo ago
I used ~16,000,000 input tokens yesterday on v4 pro, ~15,000,000 were cache hits, and I spent $0.47. Output tokens were negligible. However that's with Zed's harness, I'm not sure what you would get with Claude Code. It's maybe not quite as knowledgeable as the most expensive American models and maybe makes more mistakes (just a feeling based off of vibes, don't take my word for it), so you need to constrain its scope more. That suits my workflow, half the time I have it generate code in the chat window and then write it myself, and I'm mostly using it at the level of generating function bodies and stuff, not entire features. Although it is writing a lot of SwiftUI without me really knowing the language and doing a fine job as far as I can tell (which isn't much admittedly). One benefit I don't see talked about is it's speed - it's really quick, doesn't spend too much time reasoning even on "max", and the flash model is pretty dang good too. This lets me get into "flow state" when I'm writing code, compared to my experiences with Codex and Opus which would take minutes to complete even basic tasks and kind of ruined my focus. It's so cheap though, you could download a different harness (Crush, OpenCode, Pi etc) and load $5 in credits and test it for yourself.

electroglyph 5mo ago

deepseek 4 pro is insanely good for the price

scrollop 5mo ago
https://artificialanalysis.ai/evaluations/omniscience https://artificialanalysis.ai/evaluations/omniscience

morpheos137 5mo ago

Yes Deepseek V4 is as good or better than western sota models in my experience for practical coding given an appropriate harness. cost per solution is certainly cheaper.

jewel 5mo ago
Interesting. Can you elaborate on which harness you've tried it with? I'd love to switch to deepseek for my personal use. Also, which SOTA western models are you comparing it with? Just to give more flavor.
- freakynit 5mo ago
  My personal observation (using a mix of opencode and pi harness): 1. DS4Pro: around opus 4.5 2. DS4Flash: around sonnet 4 3. Mimo v2.5 pro: between opus 4.5 and opus 4.6. 4. minimax M3: around opus 4.6 All of these are very close in terms of quality and pricing. For anything that is not specifically related to coding, DS4Flash has become ny de-factor model. It just works... super fast, tool calling is perfect, and the price is unbeatable. Caching is out of the world. Im now regularly hitting 90%+.
  - Imanari 5mo ago
    I always feel GPT5.5 is better at ‘getting the bigger picture‘ when I am describing something vaguely vs Chinese models. What’s your experience with that?
    - freakynit 5mo ago
      That's true. The open models still do not match these extreme high end models yet on very high levels of understanding. But that's also not needed in most of the times. There will always be a "better" model... but that doesn't make other models "bad". For my use-cases, open models are now almost on par with these top models... and it's only extremely rare that I genuinely "need" the help of top-of-the line closed models.
  - 4mo ago

slopinthebag 5mo ago

I'm exclusively using Deepseek at this point and I really like it. It's not as good for vibe coding but I don't really do that so it works for me. I've spent only a couple bucks this month on it and I really like how it fits into my workflow. I have zero usage anxiety unlike when I was using subscription plans.

LinkWangder 5mo ago

This evaluation is objective. Both models have their own strengths.

BoiledCabbage 5mo ago

What is this nonsense? An AI generated article about single ai run test which in theory had many components and the AI judge declared deepseek "won"? How many runs were there on each test to account for some temperature variance? Only one. Did deepseek write better code? Did GPT's code have bugs when doing the regex? The AI "news" article doesn't actually say that. It says that grok thought that GPT's approach could have bugs so it declared deep seek the winner. This is absolute worthless methodology. And barely measurable methodology - nothing more than a prompt. No definition of what the scoring approach actually is. No definition of what "precision" actually means in this context. This is absolutely worthless and has no business being in the site, forget about on the front page. So why is it's on the front page? Because it aligns with the current "feels" of the community that deepseek will get better and it shows "bad things" about the en vogue to dislike closed models. I happen to agree with both of the views, but this site is utterly worthless. If you want HN to be astro-turfed to the max, just up vote content like this without any critical reading of the. I mean the past 6 months of "here is my chat gpt blog post of how to use a coding agent" are 1000x better than this "news article". Seriously the amount of respect I've lost recently for the HN community is incredible. A bit harsh, but very true. Maybe it's generational thing, maybe it's due to the state of politics, maybe it's a side effect of me getting older, but recently online has turned into nothing but people explicitly (or implicitly) writing about their "team". Comments on this post are nothing but people who clearly see themselves as being on "team deepseek" or "team open models" or some similar variant writing posts in support even though this is probably one of the worst "articles" to make it to the front page on ages. It clearly doesn't matter. It supports something on their "team" so they support it via comments. If kills any form of intellectual discussion. It's all just "this is my team".

sourcecodeplz 5mo ago
Have you even used deepseek pro/flash? Yes, it is astroturfed to the maxx. There is a reason for that. The performance/price ratio beats anything available today.
- raincole 5mo ago
  You misused the term 'astroturfed.' If the performance/price is that good than it'll be spreaded by word of mouth and no need to astroturfed to the death. ... and I believe which is happening. I've been advocating for DeepSeek V4 Pro and no one paid me. It's almost too good to be true.

jkwang 5mo ago

[flagged]

amazingamazing 5mo ago

How is deepseek so cheap? Cheap electricity? Subsidies?

freakynit 5mo ago
They actually explained this a few days back (can't seem to find the link right now). But, the core explanation part was it's architecture. 1. MoE (nothing new here, but, this helps a lot) 2. Compressed Attention Mechanisms (this is their core innovation) - this dramatically reduces the Key-Value (KV) cache requirements for longer contexts Another thing that helps is significantly lower energy costs in China. Another point from my own guess: they are running (some percentage) the inference on their own home-grown AI inference chips.
chvid 5mo ago
That is a very good question. It is open source / open weight - yet none of the third party providers, that also host Deepsek, seem to be able to match Deepseek itself on price. My guess is that they do aggressive caching / some proprietary optimizations in their hosting setup that they haven't published. Maybe also running at loss to gain market share. And judging from latency / network performance, I don't think what you access, when you access deepseek.com from Europe, is hosted in China.
- dchftcs 4mo ago
  It's clear to me they are subsidizing inference in exchange for market share, and doing it at this scale makes the most sense if their target is getting more user data. Note that this sort of pricing isn't far off from the equivalent token-based pricing of ChatGPT or Claude subscription plans, which are more clearly subsidized by the user's data.
orbital-decay 5mo ago
Their models are organized around inference efficiency from the start, it's what they're focusing on. Also they come from HFT and are good at low-level optimization. For v3, they've been literally reverse engineering Nvidia GPUs for undocumented behavior that helped against memory bottlenecks, writing file systems for efficient model serving, and doing a ton of low-level grunt work in the times where everyone else just relied on torch. Being compute-constrained helped as well - necessity is the mother of invention.

yoyomaindydjsj 5mo ago

[dead]

unliftedq 5mo ago

I'm tired of big news in this way - a small set of tests to declare one model is better than another, can they really consistently reproduce the result? And there's basically no disclosure: nothing other people can really hand on to verify the tests/judgement by themself. The best valuable part of DeepSeek V4 pro is its low price, I don't expect have much better performance than GPT-5.5, even it's just the performance like gpt-5.4, it's still a good model.

sourcecodeplz 4mo ago
> "I don't expect have much better performance than GPT-5.5 ..." Expectations are not always reality. Give the model a try. I just stuck with flash tbh, didn't even use pro. I do webdev in PHP.

not_a_bot_4sho 5mo ago

As I read this, looks like a single run per task. I'd be interested to see best out of N like 5 or 10 to start.

Stitch4223 5mo ago

It’s four poorly constructed arbitrary experiments which say very little about the competency of either model. The article reads like thin, auto-generated ai clickbait for nerd sniping or shilling a model. Consider the lead: > DeepSeek V4 Pro wins this head-to-head by being more exact where it matters: following instructions, matching schemas, and solving edge cases cleanly. GPT-5.5 Pro is still strong, but it gave away points with avoidable deviations. “where it matters”, “cleanly”, “is still strong”, and vague references instead of telling 3 out of 4 tests Deepseek yielded more concise results. 1 star.

jampekka 4mo ago
(Three out of) four experiments is anecdotal for sure, but the result meshes with more established instruction following benchmarking (although DeepSeek V4 pro does not top these): https://artificialanalysis.ai/evaluations/ifbench https://artificialanalysis.ai/evaluations/ifbench I found the writing clear and quite even handed. The lead is a bit salesy, but leads typically are. Knee-jerk dismissals based on vibes that something is LLM generated are quite low-effort.
- zozbot234 4mo ago
  It's picking strange tasks that don't really play to GPT-Pro's strengths (that model is roughly comparable to Mythos, intended for very hard reasoning and research-level problems) and then completely ignoring quite a few cases where GPT-Pro actually got some things more correct than DeepSeek did. The auto-AI ranking is just not reliable for this stuff.
monooso 4mo ago
I think you've misunderstood the purpose of a lead (sic). Per Merriam-Webster [^1], a lede is: > the introductory section of a news story that is intended to entice the reader to read the full story (Emphasis mine) You may prefer more matter-of-fact phrasing, of course, but criticising a lede for attempting to achieve its goal is unjustified. [^1]: https://www.merriam-webster.com/dictionary/lede https://www.merriam-webster.com/dictionary/lede

mrgblr 5mo ago

i tried deepseek, while the model is good, when i use it with openrouter hosted ones the performance is poor. sometimes it takes 2x-3x the time it takes for openai or anthropic equivalent model, making it unusable. what is the performance others are seeing, which providers you use (i cant use china hosted models).

justinram11 5mo ago
That's about what we've seen as well (even directly from deepseek themselves). We've been using it for async "heartbeat" processing and sms replies, but it's just too slow for live chat replies (which is a shame, as I'd really love to use it there). Very capable model, but also very slow.
- inhumantsar 5mo ago
  have you tried their flash model? pro was too slow for me too but I've found flash to be more than capable and it's faster than Gpt-5.5 at medium.
  - justinram11 5mo ago
    Actually on my list this week to take a look at putting an intelligence escalation flow MVP together (initial assumption would be that flash is good for 60-80% of my user's workflows, with only the tricky questions needing a more capable model. Whether I can put together a proper detection system is yet to be seen).
    - inhumantsar 4mo ago
      biggest issue I've had with flash is that it seems to hit a sort of "dumb o'clock" wall. right around the time Beijing would be going to work, response quality takes a dump on instruction-heavy tasks when context grows beyond ~120k tokens. responses are still usable, no hallucinations or anything, but it's worth keeping in mind if you rely on detailed instructions or large context windows.
- 4mo ago

psadauskas 5mo ago

I was using Claude until they banned Opencode, and now use GPT at my day job. I've been using Deepseek through Opencode Go on the $10/mo plan, and I honestly can't really tell much difference. Its just as capable, and makes the same kinds of dumb mistakes and the other two have been making since March. For the price, I'm more than happy with it.

joystick_0x0 5mo ago
I am not sure what I am doing wrong then. I am using claude the last 7 months and from time to time try other models like deepseek, kimi etc. Nothing can come even close to it. Claude is almost evrytime (99.99%) one shot.
- OtomotO 4mo ago
  You're obviously not doing anything wrong if it works for you. It worked for me too, for months, when I was working on trivial web projects. Around February of this year it got lobotomized and I quit my subscription end of march. I am not going back.
- skerit 4mo ago
  Same here. Claude isn't perfect. It still makes a lot of mistakes. But whenever I try GPT-5.5 it's ten times worse, and Claude just has to clean up GPT's mess.
- InsideOutSanta 4mo ago
  In my experience, there is a very specific use case of one-shotting complex, long tasks with relatively vague or incomplete descriptions where Opus does substantially better than all other models I've tried, including GPT 5.5, GLM 5.1 and DS4. It seems to be better at inferring unstated requirements and creating a complete, working, reasonably well-designed solution. However, that's probably not how most professional developers use LLMs. I tend to give well-specified, more constrained tasks, and for those, I find that Opus performs worse than other models precisely because it tends to infer unstated requirements and do things I didn't want it to do. In this situation, GPT 5.5 works better for me because it only and precisely does what I ask it to.

haeseong 5mo ago

[flagged]

rurban 5mo ago

Precision yes, but depth of thinking not. I can use DeepSeek V4 Pro 90% of my time, but for very tricky problems I have to use GPT or Claude models. Maybe 2x per month.

wg0 5mo ago

Of course it does. Even Deepseek v4 Flash with high easily competes with Claude Opus 4.7 for fraction of price.

jocelyner 5mo ago

[flagged]

SubiculumCode 5mo ago

Flagged for low quality.

bob1029 5mo ago

These tests are looking increasingly like a waste of time. The "intelligence" is clearly there now. Trying to measure it seems pointless. I can't shop for hammers at the hardware store and sort by the quality of finished products they would produce. That is clearly an insane ask, but that's approximately what is being pushed for with these models now. Domain specificity (harness & environment) is where the magic happens next. I intentionally use a slightly less powerful model to help reveal weakness in how I've exposed the domain to the model. Having capability reserves available dramatically increases confidence around a project like this. If the customer starts to complain about some edges, I can crank them up to gpt5.5 for target scenarios. If I'm already on 5.5 there's nowhere else to go. I'm up against the wall.

gcgbarbosa 5mo ago
"the intelligence is clearly there" I wonder if I am using the same models as everyone else. To me, LLMs still give good answers 80% of the time, but 20% it fails in such a miserable way that makes it obvious that the "intelligence" is not there.
- 21asdffdsa12 5mo ago
  It really depends on the field you are in and the tasks you set and how much of it was in the training set? A webdeveloper will find it succeeding in all taks - while some c++ exotic physics simulation developer will find it lacking. The "works for me" is telling more about the field of the LLM reviewer, then the LLM.
  - wolvesechoes 5mo ago
    > while some c++ exotic physics simulation developer will find it lacking Can confirm, but I always read I am holding it wrong.
    - 20k 4mo ago
      I've consistently tried to apply LLMs to physics problems and they're utterly useless. They'll just confidently lie, or blatantly plagiarise source materials The issue is once you hit niche physics simulations there simply isn't any training data available, so the limitations of them become incredibly apparent. Its also problematic because a field itself will contain lots of wrong information (its research!), and AI picks all this up uncritically I thought I'd give chatgpt a quick spin on my favourite question, which is "is the adm formalism strictly equivalent to general relativity", to which it consistently gives the wrong answer >Ah, now you’re hitting the subtlety head-on—that’s exactly where the “strict equivalence” claim needs nuance. Let’s unpack this carefully. I don't know how anyone can stand these tools. Its just an obnoxious glazing machine that tells me I'm a genius consistently Gemini gives a little more of a robust answer, but fails catastrophically for the question "is the bssn formalism numerically stable", where just about the entire answer is completely wrong from top to bottom. It certainly looks convincing. Its got all the right terminology. It manages to piece together the right set of words, but all the informational content is wrong, which isn't exactly a small problem I struggle to see how these tools are of any use

stalinfan 5mo ago

Deepseek: Mao did nothing wrong! Grok: Hitler did nothing wrong! ChatGPT: Altman did nothing wrong!

nutifafa 5mo ago

yes, I sure it does, that's just how models behave, today one is excellent tomorrow another is. this why being model agnostic is crucial in getting the best value out of the ecosystem.

woadwarrior01 5mo ago

DeepSeek V4 Pro with reasonix is surprisingly cheap and good enough for most coding tasks. Also, it's different enough from GPT 5.5 and Opus 4.8, that it sometimes finds issues that the other two cannot. I think it's worth having in one's toolkit.

shenberg 5mo ago

Seems 100% AI generated and automated, the judge also seems suspect - in the first one it's actually GPT-5.5 pro which has the correct email RE: the deepseek one will match a@b.com1 as "a@b.com" while 5.5 will correctly require a word boundary at the end of the email. I quit after this. No test-cases = useless judge.

beernet 4mo ago

No pelican? I don't believe it. More seriously, LLM eval is totally broken judging by the related articles on HN.

SkitterKherpi 4mo ago

There are definitely things Deepseek beats GPT on - cost effectiveness, lack of reluctance on some tasks, but from using most models I doubt it actually outperforms in a useful way in quality in a meaningful way.

ben8bit 4mo ago

[dead]

KronisLV 4mo ago

Personal experience: for overall software development, DeepSeek V4 Pro (Max reasoning) is pretty fast and generally okay - it does fuck up regularly though and I’d compare it with maybe Sonnet. It’s also quite affordable, at my current usage the DeepSeek tokens cost approx. the same as my Anthropic Max 100 USD subscription, though that’s also because DeepSeek generally needs more tokens. I’d say I have fairly moderate usage, the DeepSeek dashboard shows around 100 million tokens per day, but almost all of it cache. Without cache it’d be like 1.5 million in and 0.5 million out most days, sometimes double, other times half. Used it with Claude Code for a while, though I have to admit that using OpenCode with DeepSeek just sparks joy. Tone wise, it’s also a bit less obnoxious than Opus sometimes, though the flip side is that it’s wrong more often and sometimes just does dumb shit when it comes to code.

zftnb666 4mo ago

[flagged]

black_13 4mo ago

[dead]

poly2it 4mo ago

Why was this posted to HN? What an utter waste of time. Someone's slopwriter writes a slop article about which slopper slops the most slopulicious slop. Comments agree it's a bogus "study". We need some gate on AI-written articles. It's so weird that AI-written comments are not permitted, while the front page can be occupied by stuff like this.

cdogukank 4mo ago

[flagged]

hit8run 4mo ago

This benchmark draws a very different picture having GPT5.5 on the very top with 70% and DeepSeek at 8% https://deepswe.datacurve.ai https://deepswe.datacurve.ai

zozbot234 4mo ago
DeepSWE has been heavily criticized though. https://github.com/datacurve-ai/deep-swe/issues/21 https://github.com/datacurve-ai/deep-swe/issues/21 Putting GPT 5.5 on top is the obviously correct part, but everything else about it makes very little sense.

FpUser 4mo ago

What engine beats the other by some 10% does not matter al that much I think. With every increasing use and reasonable quality the price and availability is all that matters

amunozo 4mo ago

DeepSeek V4 Pro is wonderful and ridiculously cheap, but we are sleeping on MiMo V2.5 Pro, which have the same price (and lower cached price), it's multimodal and it's higher up in most benchmarks. Same thing for MiMo V2.5 vs DeepSeek V4 Flash.

smartbit 4mo ago
> MiMo V2.5 Pro ... lower cached price At the moment of writing https://news.ycombinator.com/item?id=48343690 https://news.ycombinator.com/item?id=48343690 MiMo V2.5 Pro had a lower cache hit ratio. From the article: OSS models, depending on who you use them from, make a huge difference, mostly due to cache-hit rates. Model Cheapest effectiveInputPrice (Provider) MiMo-V2.5-Pro 0.3720 (Xiaomi) DeepSeek V4 Pro (Max) 0.0560 (DeepSeek)
- amunozo 4mo ago
  Could it be that it changed recently, or am I missing something? Both prices are the same https://openrouter.ai/compare/xiaomi/mimo-v2.5-pro/deepseek/deepseek-v4-pro https://openrouter.ai/compare/xiaomi/mimo-v2.5-pro/deepseek/... EDIT: okay I misread it, does this mean that DeepSeek reuses a higher percentage of tokens at cache price that MiMo, am I right?
  - smartbit 4mo ago
    Correct. According to https://minimaxir.com/2026/05/openrouter-hy3/#llm-economics-in-2026:~:text=That%E2%80%99s%20a%202%25%20cache%20read,a%20calculator%20for%20that%20one%29 https://minimaxir.com/2026/05/openrouter-hy3/#llm-economics-... [0] when served by DeepSeek, Cache Read Costs/Input Costs are a very low percentage: DeepSeek V4 Pro 0.83% DeepSeek V4 Flash 2% Notice that OpenRouter response caching is not available when account-level ZDR is enforced [1] [0] https://news.ycombinator.com/item?id=48317294#48317823 https://news.ycombinator.com/item?id=48317294#48317823 [1] https://openrouter.ai/docs/guides/features/response-caching#limitations https://openrouter.ai/docs/guides/features/response-caching#...
4mo ago

atemerev 4mo ago

I do not use models who cannot answer me what happened on Tiananmen Square on June 4th, 1989.

stainablesteel 4mo ago

imo there are two major kinds of models there are models you can speak with, that respond to what you say and there are models that just make lists, that list everything, include weird formats and add asteriks everywhere. deepseek, to me, will always be the latter, and i can't stand it, you can't ask it a coherent question and get a coherent response.

wolttam 4mo ago

I rarely work on anything that demands better than DSv4 Flash, let alone pro. If I can describe the problem and its solution well enough, Flash just does it. If I can’t (or am feeling too lazy to) describe the problem well enough, and can only describe the desired outcome, then I’ve noticed models like GPT 5.5 being clearly better at working out a solid solution on their own. There are some clear differences in the capabilities of the models, but it’s also clear that smaller open weight models are good enough to be a huge help for most tasks.

Datagrout 4mo ago

[flagged]

knightops_dev 4mo ago

[flagged]

pshirshov 4mo ago

I'm a bit tired reading such claims and looking at benchmarks. E.g. minimax m3 looks to bo something opus-level and it sorta is... until it doom-loops or produces garbled output.

csbrooks 4mo ago

Shouldn't it be necessary to run the tests multiple times on each model, since the results aren't deterministic?

smhanov 4mo ago

I've been using deepseek v4 for cost/performance reasons. I feel it is generally not as good as some others, but in the end, you can make any model work by giving it the right acceptance criteria. Use detailed specs, use tests, and give it the power to iterate until it works. One-shot is a poor metric for performance.

Frost1x 4mo ago
I’m not sure all models will converge on your acceptance criteria. I’ve done quite a bit of varied agent based modeling and scientific modeling in that domain and just because you have some grounding to check against and some ideas on how you might go about getting to a convergence point doesn’t mean you’ll actually converge, you can absolutely get stuck in the information space iterating away, never finding your desired solutions. It helps but you often have to step in the failure cases and guide them or forcibly fix certain paths to get a solution.

kittikitti 4mo ago

I'm not surprised that GPT-5.5 Pro is less precise. I find that companies such as OpenAI have a profit motive that is evident in their models. This profit motive de-incentivizes precision because they can charge more if more tokens are consumed/produced.

zftnb666 4mo ago

[flagged]