21 ms·
DeepSeek-v3.1
- hodgehog11 1y agoFor reference, here is the terminal-bench leaderboard: https://www.tbench.ai/leaderboard https://www.tbench.ai/leaderboard Looks like it doesn't get close to GPT-5, Claude 4, or GLM-4.5, but still does reasonably well compared to other open weight models. Benchmarks are rarely the full story though, so time will tell how good it is in practice.
- seunosewa 1y agoThe DeepSeek R1 in that list is the old model that's been replaced. Update: Understood.
- yorwba 1y agoYes, and 31.3% is given in the announcement as the performance of the new v3.1, which would put it in sixteenth place.
- deleted 1y ago[deleted]
- coliveira 1y agoMy personal experience is that it produces high quality results.
- amrrs 1y agoAny example or prompt you use to make this statment?
- imachine1980_ 1y agoI remember asking for quotes about the Spanish conquest of South America because I couldn't remember who said a specific thing. The GPT model started hallucinating quotes on the topic, while DeepSeek responded with, "I don't know a quote about that specific topic, but you might mean this other thing." or something like that then cited a real quote in the same topic, after acknowledging that it wasn't able to find the one I had read in an old book. i don't use it for coding, but for things that are more unique i feel is more precise.
- sync 1y agoI'm doing coreference resolution and this model (w/o thinking) performs at the Gemini 2.5-Pro level (w/ thinking_budget set to -1) at a fraction of the cost.
- SV_BubbleTime 1y agoVine is about the only benchmark I think is real. We made objective systems turn out subjective answers… why the shit would anyone think objective tests would be able to grade them?
- YetAnotherNick 1y agoDepends on the agent. Rank 5 and 15 are claude 4 sonnet, and this stands close to 15th.
- guluarte 1y agotbh companies like anthopic, openai, create custom agents for specific benchmarks
- bazmattaz 1y agoDo you have a source for this? I’m intrigued
- guluarte 1y agohttps://www-cdn.anthropic.com/07b2a3f9902ee19fe39a36ca638e5ae987bc64dd.pdf https://www-cdn.anthropic.com/07b2a3f9902ee19fe39a36ca638e5a... "we iteratively refine prompting by analyzing failure cases and developing prompts to address them."
- amelius 1y agoAren't good benchmarks supposed to be secret?
- wkat4242 1y agoThis industry is currently burning billions a month. With that much money around I don't think any secrets can exist.
- noodletheworld 1y agoHow can a benchmark be secret if you post it to an API to test a model on it? "We totally promise that when we run your benchmark against our API we won't take the data from it and use to be better at your benchmark next time" :P If you want to do it properly you have to avoid any 3rd party hosted model when you test your benchmark, which means you can't have GPT5, claude, etc. on it; and none of the benchmarks want to be 'that guy' who doesn't have all the best models on it. So no. They're not secret.
- dmos62 1y agoHow do you propose that would work? A pipeline that goes through query-response pairs to deduce response quality and then uses the low-quality responses for further training? Wouldn't you need a model that's already smart enough to tell that previous model's responses weren't smart enough? Sounds like a chicken and egg problem.
- segmondy 1y agogarbage benchmark, inconsistent mix of "agent tools" and models. if you wanted to present a meaningful benchmark, the agent tools will stay the same and then we can really compare the models. there are plenty of other benchmarks that disagree with these, with that said. from my experience most of these benchmarks are trash. use the model yourself, apply your own set of problems and see how well it fairs.
- paradite 1y agoHey. I like your roast on benchmarks. I also publish my own evals on new models (using coding tasks that I curated myself, without tools, rated by human with rubrics). Would love you to check out and give your thoughts: Example recent one on GPT-5: https://eval.16x.engineer/blog/gpt-5-coding-evaluation-underwhelming-performance https://eval.16x.engineer/blog/gpt-5-coding-evaluation-under... All results: https://eval.16x.engineer/evals/coding https://eval.16x.engineer/evals/coding
- jstummbillig 1y agoWhich benchmarks are not garbage? I don't consider myself super special. I think it should be doable to create a benchmark that beats me having to test every single new model.
- tonyhart7 1y agoYeah but the pricing is insane, I don't care about SOTA if its not break my bank
- seunosewa 1y agoIt's a hybrid reasoning model. It's good with tool calls and doesn't think too much about everything, but it regularly uses outdated tool formats randomly instead of the standard JSON format. I guess the V3 training set has a lot of those.
- ivape 1y agoWhat formats? I thought the very schema of json is what allows these LLMs to enforce structured outputs at the decoder level? I guess you can do it with any format, but why stray from json?
- seunosewa 1y agoSometimes it will randomly generate something like this in the body of the text: ``` <tool_call>executeshell <arg_key>command</arg_key> <arg_value>echo "" >> novels/AI_Voodoo_Romance/chapter-1-a-new-dawn.txt</arg_value> </tool_call> ``` or this: ``` <｜toolcallsbegin｜><｜toolcallbegin｜>executeshell<｜toolsep｜>{"command": "pwd && ls -la"}<｜toolcallend｜><｜toolcallsend｜> ``` Prompting it to use the right format doesn't seem to work. Claude, Gemini, GPT5, and GLM 4.5, don't do that. To accomodate DeepSeek, the tiny agent that I'm building will have to support all the weird formats.
- deleted 1y ago[deleted]
- irthomasthomas 1y agoCan't you use logit bias to help with this? Might depend how they are tokenized.
- ilaksh 1y agoMaybe you have your temperature turned up too high.
- refulgentis 1y agoIn the modes in APIs, the sampling code essentially "rejects and reinference" any token sampled that wouldn't create valid JSON under a grammar created from the schema. Generally, the training is doing 99% of the work, of course, it's just "strict" means "we'll check it's work to the point a GBNF grammar created from the schema will validate." One of the funnier info scandals of 2025 has been that only Claude was even close to properly trained on JSON file edits until o3 was released, and even then it needed a bespoke format. Geminis have required using a non-formalized diff format by Aider. Wasn't until June Gemini could do diff-string-in-JSON better than 30% of the time and until GPT-5 that an OpenAI model could. (Though v4a, as OpenAI's bespoke edit format is called, is fine because it at least worked well in tool calls. Geminis was a clown show, you had to post process regular text completions to parse out any diffs)
- esafak 1y agoIt seems behind Qwen3 235B 2507 Reasoning (which I like) and gpt-oss-120B: https://artificialanalysis.ai/models/deepseek-v3-1-reasoning https://artificialanalysis.ai/models/deepseek-v3-1-reasoning Pricing: https://openrouter.ai/deepseek/deepseek-chat-v3.1 https://openrouter.ai/deepseek/deepseek-chat-v3.1
- bigyabai 1y agoThose Qwen3 2507 models are the local creme-de-la-creme right now. If you've got any sort of GPU and ~32gb of RAM to play with, the A3B one is great for pair-programming tasks.
- pdimitar 1y agoDo you happen to know if it can be run via an eGPU enclosure with f.ex. RTX 5090 inside, under Linux? I'm considering buying a Linux workstation lately and I want it full AMD. But if I can just plug an NVIDIA card via an eGPU card for self-hosting LLMs then that would be amazing.
- gunalx 1y agoYou would still need drivers and all the stuff difficult with nvidia in linux with a egpu. (Its not nessecarily terrible just suboptimal) Rather just add the second GPU in the Workstation, or just run the llm in your AMD GPU.
- pdimitar 1y agoOh, we can run LLMs efficiently with AMD GPUs now? Pretty cool, I haven't been following, thank you.
- DarkFuture 1y agoI've been running LLM models on my Radeon 7600 XT 16GB for past 2-3 months without issues (Windows 11). I've been using llama.cpp only. The only thing from AMD I installed (apart from latest Radeon drivers) is the "AMD HIP SDK" (very straight forward installer). After unzipping (the zip from GitHub releases page must contain hip-radeon in the name) all I do is this: llama-server.exe -ngl 99 -m Qwen3-14B-Q6_K.gguf And then connect to llamacpp via browser to localhost:8080 for the WebUI (its basic but does the job, screenshots can be found on Google). You can connect more advanced interfaces to it because llama.cpp actually has OpenAI-compatible API.
- abtinf 1y agoUnrelated, but it would really be nice to have a chart breaking down Price Per Token Per Second for various model, prompt, and hardware combinations.
- imranq 1y agoThere is one: https://pricepertoken.com/ https://pricepertoken.com/
- rapind 1y agoClaude's Opus pricing is nuts. I'd be surprised if anyone uses it without the top max subscription.
- jjani 1y agoSure I do, but not as part of any tools, just for one-off conversations where I know it's going to be the best out there. For tasks where reasoning helps little to none, it's often still number one.
- memothon 1y agoSome people have startup credits
- tmoravec 1y agoFWIW I have the €20 Pro plan and exchange maybe 20 messages with Opus (with thinking) every day, including one weeks-long conversation. Plus a few dozen Sonnet tasks and occasionally light weight CC. I'm not a programmer, though - engineering manager.
- theuurviv467456 1y ago[flagged]
- simianparrot 1y ago[flagged]
- tehjoker 1y agoIncredible how "keeping their people down" means leaps in personal wealth and happiness for huge swathes of the population and internal criticism is that it is a "poverty reduction machine" that is too focused.
- jamiek88 1y agoTell that to the Uyghurs if you can get into their concentration camp to have a chat.
- jaggs 1y agoYep, comment arrives right on time. Nicely played. :)
- bigyabai 1y agoThey're not mutually exclusive. Lots of terrible and mismanaged governments rely on genius short-term economic exploitation, like Syria, Iran, India, Korea, etc. What would be incredible is China sticking the landing to a third-sector economy. Plenty of countries have industrialized over the past century, only a handful became true service economies.
- tehjoker 1y agoYou can only become a service economy by outsourcing everything to another country. It's not a sustainable model and it's related to imperialism.
- danielhanchen 1y agoFor local runs, I made some GGUFs! You need around RAM + VRAM >= 250GB for good perf for dynamic 2bit (2bit MoE, 6-8bit rest) - can also do SSD offloading but it'll be slow. ./llama.cpp/llama-cli -hf unsloth/DeepSeek-V3.1-GGUF:UD-Q2_K_XL -ngl 99 --jinja -ot ".ffn_.*_exps.=CPU" More details on running + optimal params here: https://docs.unsloth.ai/basics/deepseek-v3.1 https://docs.unsloth.ai/basics/deepseek-v3.1
- tw1984 1y agofor such dynamic 2bit, is there any benchmark results showing how many performance I would give up compared to the original model? thanks.
- danielhanchen 1y agoCurrently no, but I'm running them! Some people on the aider discord are running some benchmarks!
- cowpig 1y ago@danielhanchen do you publish the benchmarks you run anywhere?
- danielhanchen 1y agoWe had benchmarks for Llama 4 and Gemma 3 at https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs - for others I normally refer to https://discord.com/channels/1131200896827654144/1282240423661666337 https://discord.com/channels/1131200896827654144/12822404236... which is the Aider Polygot Discord - they always benchmark our quants :)
- segmondy 1y agoif you are running a 2bit quant, you are not giving up performance but gaining 100% performance since the alternative is usually 0%. Smaller quants are for folks who won't be able to run anything at all, so you run the largest you can run relative to your hardware. I for instance often ran Q3_K_L, I don't think of how much performance I'm giving up, but rather how without Q3, I won't be able to run it at all. With that said, for R1, I did some tests against 2 public interfaces and my local Q3 crushed them. The problem with a lot of model providers is we can never be sure what they are serving up and could take shortcuts to maximize profit.
- xmichael909 1y agoSeems to hallucinate more than any model I've ever worked with in the past 6 months.
- dude250711 1y agoDid they "borrow" bad data this time?
- energy123 1y agoWhat context length did you use?
- Leynos 1y agoDeepSeek is bad for hallucinations in my experience. I wouldn't trust its output for anything serious without heavy grounding. It's great for fantastical fiction though. It also excels at giving characters "agency".
- bgilroy26 1y agoWhere would you go to find people posting their AI generated fiction? I haven't been able to find it on Reddit
- CamperBob2 1y agoAmazon, Barnes & Noble, Powell's, the usual places.
- 1gn15 1y agoAO3 has several tags for it.
- bgilroy26 1y agoI should have said, I am looking for posted chat logs where the prompts are shared as well. I really enjoy the process of making stories with AI and I am curious to see how others do the same thing.
- jbellis 1y agoAbout halfway between V3 and Qwen3 Coder. https://brokk.ai/power-ranking?version=openround-2025-08-20&models=ds-v3.1%2Cgpt5-mini%2Ck2%2Cq3c%2Cv3 https://brokk.ai/power-ranking?version=openround-2025-08-20&...
- indigodaddy 1y agoIs gpt-5 Mini free from any providers?
- drmidnight 1y agoDuck.ai has it as an option
- deleted 1y ago[deleted]
- dr_dshiv 1y agoCheep! $0.56 per million tokens in — and $1.68 per million tokens out.
- NiekvdMaas 1y agoThat's actually a big bump from the previous pricing: $0.27/$1.10
- kenmacd 1y agoAnd unfortunately no more half price 8-hours a day either :(
- manishsharan 1y agoThe next cheapest and capable model is GLM 4.5 at $0.6 per million tokens in and $2.2 per million tokens out. Glad to see DeepSeek is still be the value king. But I am sti disappointed with the price increase.
- aussieguy1234 1y agoThey say the SWE bench verified score is 66%. Claude Sonnet 4 is 67%. Not sure if the 1% difference here is statistically significant or not. I'll have to see how things go with this model after a week, once the hype has died down.
- loog5566 1y agoI'm doing this model
- snippai 1y agoLooks quite competitive among open-weight models, but I guess still behind GPT-5 or Claude a lot.
- greenavocado 1y agoI have yet to see evidence that it is better for agentic coding tasks than GLM-4.5
- postalrat 1y agoIs that it? Nothing else you haven't seen evidence for?
- greenavocado 1y agoJust that
- vitaflo 1y agoSad to see the off peak discount go. I was able to crank tokens like crazy and not have it cost anything. That said the pricing is still very very good so I can't complain too much.
- rsanek 1y agoLooks to be the ~same intelligence as gpt-oss-120B, but about 10x slower and 3x more expensive? https://artificialanalysis.ai/models/deepseek-v3-1-reasoning https://artificialanalysis.ai/models/deepseek-v3-1-reasoning
- petesergeant 1y agoI don't think you're necessarily wrong, but your source is currently only showing a single provider. Comparing: https://openrouter.ai/openai/gpt-oss-120b https://openrouter.ai/openai/gpt-oss-120b and https://openrouter.ai/deepseek/deepseek-chat-v3.1 https://openrouter.ai/deepseek/deepseek-chat-v3.1 for the same providers is probably better, although gpt-oss-120b has been around long enough to have more providers, and presumably for hosters to get comfortable with it / optimize hosting of it.
- okasaki 1y agoMy experience is that gpt-oss doesn't know much about obscure topics, so if you're using it for anything except puzzles or coding in popular languages, it won't do well as the bigger models. It's knowledge seems to be lacking even compared to gpt3. No idea how you'd benchmark this though.
- easygenes 1y agoSomething I was doing informally that seems very effective is asking for details about smaller cities and towns and lesser points of interest around the world. Bigger models tend to have a much better understanding and knowledge base for the more obscure places.
- scotty79 1y agoI would really love if they figured out how to train a model that doesn't have any such knowledge baked it, but knows where to look for it. Maybe even has a clever database for that. Knowing this kind of trivia like this consistently of the top of your head is a sign of deranged mind, artificial or not.
- fariszr 1y agoIs it foot at tool use? For me tool use is table stakes, if a model can't use tools then its almost useless.
- dsign 1y agoSome of it is in Kagi already. Impressive from both DeepSeek and Kagi.
- donbreo 1y agoIt still cant name all the states in India
- Leynos 1y agoThat's interesting. I am curious about the extent of the training data in these models. I asked Kimi K2 for an account of growing up in my home town in Scotland, and it was ridiculously accurate. I then asked it to do the same for a similarly sized town in Kerala. ChatGPT suggested that while it was a good approximation, K2 got some of the specifics wrong.
- tw1984 1y agojust saw this on Chinese internet - deepseek officially mentioned that v3.1 is trained using UE8M0 FP8 as that is the FP8 to be supported by the next gen Chinese AI chip. so basically - some Chinese next gen AI chips is coming, deepseek is working with them to get its flagship model trained using such domestic chips. interesting time ahead! just imagine what it could do to NVIDIA share price when deepseek releases a SOTA new model trained without using NVIDIA chips.
- Narciss 1y agoV interesting, thanks for sharing
- Alifatisk 1y agoTime to short Nvidia?
- hangonhn 1y agoNo because people never really talk about the quantity of the alternatives -- i.e. Huawei Ascent. Even if Huawei can match the quality, their yields are still abysmal. The numbers I've heard are in the hundreds of thousands vs. millions by Nvidia. In the near future, Nvidia's dominance is pretty secure. The only thing that can threaten it is if this whole AI thing isn't worth what some people imagined it is worth and people start to realize this.
- ychan268 1y agoNo evidence v3.1 is trained on Chinese chips(they said very ambiguously, only said they adapted the model for Chinese chips, could be training, could be inference) Anyway, from my experience, if China really has advanced AI chips for SOTA model, I am sure propaganda machine will go all out, look how they boasted Huawei CPU that’s two generations behind Qualcomm and TSMC
- unleaded 1y agonot sure if its just chat.deepseek.com but one strange thing I've noticed is that now it replies to like 90% of your questions with "Of course.", even when it doesnt fit the prompt at all. maybe it's the backend injecting it to be more obedient? but you can tell it `don't begin the reply to this with "of" ending "course"` and it will listen. it's very strange Some people on reddit (very reliable source I know) are saying it was trained on a lot of Gemini and I can see that. for example it does that annoying thing gemini does now where when you use slang or really any informal terms it puts them in quotes in its reply
- edg5000 1y ago> for example it does that annoying thing gemini does now where when you use slang or really any informal terms it puts them in quotes in its reply Haven´t used Gemini much, but the time I used it, it felt very academic and theoretical compared to Opus 4. So that seems to fit. But I'll have to do more evaluation of the non-Claude models to get a better idea of the differences.
- pradn 1y agoAll this points to "personality" being a big -- and sticky -- selling point for consumer-facing chat bots. People really did like the chatty, emoji-filled persona of the previous ChatGPT models. So OpenAI was ~forced to adjust GPT-5 to be closer to that style. It raises a funny "innovator's dilemma" that might happen. Where an incumbent has to serve chatty consumers, and therefore gets little technical/professional training data. And a more sober workplace chatbot provider is able to advance past the incumbent because they have better training data. Or maybe in a more subtle way, chatbot personas give you access to varying market segments, and varying data flywheels.
- reassess_blind 1y agoBubble popped yet?
- d4rkp4ttern 1y agoIt’s a very smart move for DeepSeek to put out an Anthropic-compatible API, similar to Kimi-k2, GLM4.5 (Puzzled as to why Qwen didn’t do this). You can set up a simple function in your .zhsrc to run Claude-Code with these models: https://github.com/pchalasani/claude-code-tools/tree/main?tab=readme-ov-file#-using-claude-code-with-open-weight-anthropic-api-compatible-llm-providers https://github.com/pchalasani/claude-code-tools/tree/main?ta...
- jodleif 1y agoQwen have their own competitor to Claude Code.
- larrysalibra 1y agoWow thanks! I just ran into my claude code session limit like an hour ago and tried the method you linked and added 10 CNY to a deepseek api account and an hour later i've got 7.77 CNY left and have used 3.3 million tokens. I'm not confident enough to say it's as good as claude opus or even sonnet, but it seems not bad! I did run into an api error when my context exceeded deepseek's 128k window and had to manually compact the context.
- axpy906 1y agoHmm. It’s still not close to paid frontier on SWE bench.
- asaddhamani 1y agoIn my experience, Qwen 3 coder has been very good for agentic coding with Cline. I tried DeepSeek v3.1 and wasn't pleased with it.
- deleted 1y ago[deleted]
- guerrilla 1y agoSo, is the output price there why most models are extremely verbose? Is it just a ploy to make extra cash? It's super annoying that I have to constantly tell it to be more and more concise.
- diggan 1y ago> It's super annoying that I have to constantly tell it to be more and more concise. While system promting is the easy way of limiting the output in a somewhat predictable manner, have you tried setting `max_tokens` when doing inference? For me that works very well for constraining the output, if you set it to 100 you get very short answers while if you set it to 10,000 you can very long responses.
- niteshpant 1y agohow can deepseek be so cheap* yet so effective? *pricing: MODEL deepseek-chat deepseek-reasoner 1M INPUT TOKENS (CACHE HIT) $0.07 1M INPUT TOKENS (CACHE MISS) $0.56 1M OUTPUT TOKENS $1.68
- Alifatisk 1y agoI think it's because of a combination between the MoE model architecture and the inference done in large batches and run in parallel
- hereme888 1y agoReminder DeepSeek is a Chinese company whose headstart is attributed to stealing IP from American companies. Without the huge theft, they'd be nowhere.
- bobro 1y agoCan you contrast this with Western companies? What are the Chinese companies stealing that Western companies aren’t? Do you mean tech or content?
- hereme888 1y agoEthics of Chinese vs. Western companies? Everything. I'm sure you're aware of how many hundreds of $billions of American IP are stolen by Chinese companies.
- bobro 1y agoI’m not asking broadly about difference in ethics. I’m asking specifically about IP theft in the AI space.
- hereme888 1y agoI'm not aware of any proven IP theft by American companies in the AI space. Many pending legal challenges. None yet proven.
- bobro 1y agoAlright. So there’s proven IP theft by the Chinese companies with completed legal proceedings?
- hereme888 1y agoWith completed legal proceedings, at least two cases: Xiaolang Zhang was sentenced in 2024 for stealing Apple's AI autonomous vehicle tech for Chinese AI company XPeng. Xiang Haitao was sentenced in 2022 for stealing Monsanto's AI predictive algorithm for a Chinese research institute.
- CuriouslyC 1y agoCries in 128k context. Probably will be a good orchestrator though, can always delegate to Gemini.
- techlatest_net 1y ago[dead]
- caycep 1y agothis might be OT and covered somewhere else but what's the latest/greatest on these models and their effect on the linguistics field, vs. what does the latest and greatness in linguistics feel about these models?