9 ms·
Cerebras launches Qwen3-235B, achieving 1.5k tokens per second
- avnathan 1y agoThey pull these off pretty consistently!
- pr337h4m 1y agoQuantization?
- TechDebtDevin 1y agoIts not a new model, but rather their infrastructure and hardware they are showcasing.
- pr337h4m 1y agoGroq appears to have quantized the Kimi K2 model they're serving, which is part of the reason why there's a noticeable performance gap between K2 on Moonshot's official API and the one served by Groq. We don't know how/whether the Qwen3-235B served by Cerebras has been quantized.
- logicchains 1y agoCerebras have previously stated for other models they hosted that they didn't quantise, unlike Groq.
- mehdibl 1y agoWould be great if they support the latest Qwen 3 405B launched yesterday and more aimed at agentic work/coding.
- poly2it 1y agoVery impressive speed. With a context window of 40K however, usability is limited.
- wild_egg 1y agoPost says 131k context though? What did I miss?
- mehdibl 1y agoThe PR confuse a but 32k/64k and 131k if paid API. Also this model https://huggingface.co/Qwen/Qwen3-235B-A22B https://huggingface.co/Qwen/Qwen3-235B-A22B Is native 32k. So the 64k and 131k use ROPE that is not the best for effective context. While https://qwenlm.github.io/blog/qwen3-coder/ https://qwenlm.github.io/blog/qwen3-coder/ it's 256k native https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct.
- asb 1y agoThe situation is very confusing, but the tweet that went out with the announcement indicates it's not full 131k context yet and that is coming "soon"https://xcancel.com/CerebrasSystems/status/1943765301109420285 https://xcancel.com/CerebrasSystems/status/19437653011094202...
- diggan 1y agoThe first paragraph contains: > Cerebras Systemstoday [sic] announced the launch of Qwen3-235B with full 131K context support on its inference cloud platform Then later: > Cline users can now access Cerebras Qwen models directly within the editor—starting with Qwen3-32B at 64K contexton the free tier. This rollout will expand to include Qwen3-235B with 131K context Not sure where you get the 40K number from.
- mehdibl 1y agoIt seem this news is "outdated" as it's from Jul 8 and might picked up confusing this model with yesterday Qwen 3 coder 405B release that is different in specs.
- simonw 1y agoI initially thought this was about the Qwen release from two days ago, Qwen3-235B-A22B-Instruct-2507 - https://simonwillison.net/2025/Jul/22/qwen3-235b-a22b-instruct-2507/ https://simonwillison.net/2025/Jul/22/qwen3-235b-a22b-instru... - but that's a no-reasoning model and the Cerebras announcement talks about reasoning, which tipped me off that this was Qwen's Qwen3-235B-A22B from April. (These model names are so confusing.)
- aitchnyu 1y agoIs Qwen3 235B A22B in OpenRouter the stock version or Cerebras version? https://openrouter.ai/provider/cerebras https://openrouter.ai/provider/cerebras
- simonw 1y agohttps://openrouter.ai/qwen/qwen3-235b-a22b/providers https://openrouter.ai/qwen/qwen3-235b-a22b/providers has a list, it's currently DeepInfra, Parasail, Together, Nebius AI Studio, Friendli, Fireworks, Cerebras.
- cedws 1y agoWith this kind of speed you could build a large thinking stage into every response. What kind of improvement could you expect in benchmarks from having say 1000 tokens of thinking for every response?
- lionkor 1y agoThinking can also make the responses worse; AIs don't "overthink", instead they start throwing away constraints and convincing themselves of things that are tangential or opposite to the task. I've often observed thinking/reasoning to cause models to completely disregard important constraints, because they essentially can act as conversational turns.
- rbanffy 1y ago> start throwing away constraints and convincing themselves of things that are tangential or opposite to the task Funny that, when given too much brainpower, AIs manifest ADHD symptoms…
- falcor84 1y agoMy use-case would probably be of autocompacting the context of another LLM. I've been using Claude Code a lot recently, and feel that it generally gets better at handling my codebase once it uses up a lot of context (often >50%), but then it often runs out of context before finishing the task. So I'd be very interested in something that runs behind the scenes and compacts it to e.g. ~80%. I know that Letta have a decent approach to this, but I haven't yet seen it done well with a coding agent, by them or anyone else. Is there anyone doing this with any measure of success?
- 0vermorrow 1y agoI'm eagerly awaiting for Qwen 3 coder being available on Cerebras. I run plenty of agent loops and the speed makes a somewhat interesting difference in time "compression". Having a Claude 4 Sonnet-level model running at 1000-1500 tok/s would be extremely impressive. To FEEL THE SPEED, you can either try it yourself on Cerebras Inference page, through their API, or for example on Mistral / Le Chat with their "Flash Answers" (powered by Cerebras). Iterating on code with 1000 tok/s makes it feel even more magical.
- mogili 1y agoSame. However, I think Cerebras first needs to get the APIs to be more openAI compliant. I tried their existing models with a bunch of coding agents (include Cline which they did a PR for) and they all failed to work either due to a 400 error or tool calls not being formatted correctly. Very disappointed.
- meowface 1y agoI just set up Groq with Kimi K2 the other day and was blown away by the speed. Deciding if I should switch to Qwen 3 and Cerebras. (Also, off-topic, but the name reminds me of cerebrates from Starcraft. The Zerg command hierarchy lore was fascinating when I was a young child.)
- throwaw12 1y agoHave you used Claude Code and how do you compare the quality to Claude models? I am heavily invested in tools around Claude, still struggling to make a switch and start experimenting with other models
- bredren 1y agoAre you using Claude code or the web interface? I would like to try this with CC myself, apparently with some proxy use an OpenAI compatible LLM can be swapped in.
- 1y ago
- mohsen1 1y agoK2 is also now available on Groq https://console.groq.com/docs/model/moonshotai/kimi-k2-instruct https://console.groq.com/docs/model/moonshotai/kimi-k2-instr... very fun to see agents using those backends
- meowface 1y agoThere are rumors that the K2 model Groq is serving is quantized or otherwise produces lower-quality responses than expected due to some optimization, FYI. I tested it and the speed is incredible, though.
- skeezyboy 1y agohave they managed to remove the "output may contain mistakes" disclaimer from a single LLM yet?
- lazide 1y agoNever will. But then, same for humans yes?
- skeezyboy 1y ago>But then, same for humans yes? And? Whats your point? This is a computer. Humans make errors doing arithmetic, therefore should we not expect computers to be able to reliably perform arithmetic? No. Silly retort and a common reply from people who are suitably wowed by the current generation of AI.
- pjs_ 1y agoCerebras is truly one of the maddest technical accomplishments that Silicon Valley has produced in the last decade or so. I met Andy seven or eight years ago and I thought they must have been smoking something - a dinner plate sized chip with six tons of clamping force? They made it real, and in retrospect what they did was incredibly prescient
- cherryteastain 1y agoThe concept is super cool but does anyone actually use them instead of just buying Nvidia?
- vFunct 1y agoWafer-scale integration was done decades before.
- throwawaymaths 1y agonah, it was designed for hpc and raw flops. llm inference really requires memory bandwidth.
- HumanOstrich 1y agoMemory bandwidth, eh? You should learn the basics about what Cerebras does. https://www.cerebras.ai/chip https://www.cerebras.ai/chip
- 1y ago
- aurareturn 1y agoIf this is the full fp16 quant, you'd need 2TB of memory to use with the full 131k context. With 44GB of SRAM per Cerebras chip, you'd need 45 chips chained together. $3m per chip. $135m total to run this. For comparison, you can buy a DGX B200 with 8x B200 Blackwell chips and 1.4TB of memory for around $500k. Two systems would give you 2.8TB memory which is enough for this. So $1m vs $135m to run this model. It's not very scalable unless you have some ultra high value task that need super fast inference speed. Maybe hedge funds or some sort of financial markets? PS. The reason why I think we're only in the beginning of the AI boom is because I can't imagine what we can build if we can run models as good as Claude Opus 4 (or even better) at 1500 tokens/s for a very cheap price and tens of millions of context tokens. We're still a few generations of hardware away I'm guessing.
- lordofgibbons 1y agoAlmost everyone runs LLM inference at fp8 - for all of the open models anyway. You only see performance drop off below fp8.
- stingraycharles 1y agoIsn’t usually mixed? I understood that Apple even uses fp1 or fp2 on their hardware embedded models they ship on their phones, but as far as I know it’s typically a whole bunch of different precisions.
- llm_nerd 1y agoSmall bit of pedantry: While there are 1 and 2-bit quantized types used in some aggressive schemes, they aren't floating point so it's inaccurate to preface them with FP. They are int types. The smallest real floating point type is FP4. EDIT: Who knew that correctness is controversial. What a weird place HN has become.
- xtracto 1y agoI wonder if the fact that we use "floating point" is itself a bottleneck that can be improved. Remembering my CS classes, storing an FP value requires the base and the exponent; that's a design decision. Also remembering some assembler classes, Int arithmetic is way faster than FP. Could there be a better "representation " for the numbers needed in NN that would provide the accuracy of floating point but provide faster operations? (Maybe even allow to perform required operations as bitwise ops. Kind of like the left/right shifting to double/half ints. )
- rsolva 1y agoWhat would the energy use be for an average query be, when using large models at this speed?
- scottcha 1y agoI’ve asked that question on linked in to the Cerebras team a couple times and haven’t ever received a response. There is system max tdp values posted online but I’m not sure you can assume the system is running in max tdp for these queries. If it is the numbers are quite high (I just tried to find the number but couldn’t find it but I had it in my notes as 23kw). If someone from Cerebras is reading this feel free to dm me as optimizing this power is what we do.
- skeezyboy 1y ago23kw gotdamn
- the_arun 1y agoIMHO innovations are waiting to happen. Unless we get similar speeds using commodity hardware / pricing, we are not there yet.
- skeezyboy 1y agobut does it still produce potentially unreliable and hallucinatory output? id hate to see that feature go
- tonyhart7 1y agoYeah but the price is uhh
- bluelightning2k 1y agoThis is (slightly) old news from July 8, resurfaced due to the Qwen 3 coder. I think the gist of this thread is entirely: "please do the same for Qwen 3 coder", with us all hoping for: a) A viable alternative to Sonnet 3 b) Specifically a faster and cheaper alternative
- doubtfuluser 1y agoVery impressive speed. A bit OT: what is the current verdict on Qwen, Kimi et al. When it comes to censorship / bias concerning narratives not allowed in the origin country?
- jszymborski 1y agoThe Qwen models are, anecdotally, probably some of the best open weight models, particularly the MoE models. They are also, anecdotally, super scary censored. Asking it if anything "interesting has happened in Tianamen Square?" And then refining with "any notable protests?" And finally "maybe something to do with a tank"... All you get is vague allusions to the square being a beautiful place with a rich history.
- impossiblefork 1y agoDo you think it's done so carefully that you suspect that they have perhaps even removed texts mentioning the Tiananmen square massacre from the training set?
- jszymborski 1y agoI have no special knowledge on the matter, but I imagine it's the same kind of alignment that prevents other LLMs from telling you e.g. how to make meth.
- rafaelero 1y agoIf they do the same for the coding model they will have a killer product.
- rbanffy 1y agoIf they do that for chip design and it successfully iterates the design into the next generation on 2nm or less, it’ll be even more ludicrous.
- poupou127 1y agoWOW
- OxfordCommand 1y agoisn't Qwen Alibaba's family of models? What does cerebras have to do with this? i'm lost.
- esafak 1y agoServing. https://openrouter.ai/provider/cerebras https://openrouter.ai/provider/cerebras
- jug 1y agoThey better not cheat me with a quantized version!
- OldfieldFund 1y agoI tried the non-quantized version, and it was pretty bad at creative writing compared to Kimi K2. Very deterministic and every time I regenerated the same prompt I got the usual AI phrases like "the kicker is:", etc. Kimi was much more natural.
- nisten 1y ago"Full 131k" context , actually the full context is double that at 262144 context and with 8x yarn mutiplier it can go up to 2million. It looks like even full chip scale Cerebras has trouble with context length, well, this is a limitation of the transformer architechture itself where memory requirements scale ~linearly and compute requirements roughly quadratically with the increase in kv cache. Anyway, YOU'RE NOT SERVING FULL CONTEXT CEREBRAS, YOU'RE SERVING HALF. Also what quantization exactly is this, can the customers know?
- zamadatix 1y agoThe model page says 32,768 natively with performance validated for up to 4x YaRN https://huggingface.co/Qwen/Qwen3-235B-A22B#processing-long-texts https://huggingface.co/Qwen/Qwen3-235B-A22B#processing-long-... That would seem to align with the 131k number?
- iyerbalaji 1y agoAmazing, this is blazing fast
- iyerbalaji 1y ago[dead]
- Inviz 1y agoI contacted their sales team before, cerebras started at $1500 a month at that time, and the limits were soooooo small. Did it get better? Edit: Looks like it did. They both introduced pay as you go, and have prepaid limits too at $1500. I wonder if they have any limitations on parallel execution for pay as you go...
- rbanffy 1y agoWho remembers “wafer scale integration” from the 1980s? Insane that Cerebras succeeded where everyone else failed for 5 decades.
- doctoboggan 1y agoHas anyone with a lot of experience with Claude Code and sonnet-4 tried Claude Code with Qwen3-Coder? The fast times enabled here by Cerebras are enticing, but I wouldn't trade a speedup for a worse quality model.
- AgentMatrixAI 1y agohaven't tried Qwen but used these "near instant token" like groq and another one that uses diffusion model to generate code via LLaMA and the results weren't satisfactory. now if something like Gemini 2.5 pro or Sonnet 4 even can run on Cerebras generating tens of thousands of code in a few seconds, that could really make a difference.
- cubefox 1y agoIt sounds like Cerebras would be perfect for models with Mamba architecture, as those don't need a large KV cache for long contexts.
- avnathan 1y agoThey pull these off quite consistently!
- adamtaylor_13 1y agoI’m a guy that simply runs Claude Code. How can I start toying around with this?
- Maxious 1y agohttps://cc.yovy.app/ https://cc.yovy.app/ lets you connect https://openrouter.ai/provider/cerebras https://openrouter.ai/provider/cerebras to claude code
- mikewarot 1y agoIt's quite possible this is getting near the upper limit that is possible with current architectures for compute. Let's say the limit were 10k tokens/second with Qwen3-235B. There's always going to be some latency in any compute architecture. Assume some insane billionaire cast the entire Qwen3-235B model into silicon, so it all ran in parallel, tokens going in one end, and the next token coming out the other end. This wafer (or likely, stack of interconnected wafers) would likely add up to a latency from end to end of 10 to 100 milliseconds. If you then added pipelining, the latency might actually increase a millsecond or two, but the aggregate throughput would be N times the number of pipeline stages. If you could increase N to the point that the clock cycle were a nanosecond... what would the economic value of this thing be? 100,000 separate streams at 10,000 tokens per second, multiplexing through it. If you change it from cast in silicon, to a program to configure the silicon (line an FPGA, but far less clunky), I believe you get the future of LLM compute. Ever faster and wider lanes between compute and RAM is a dead end, a premature optimization.
- p0w3n3d 1y agoI'm looking for a setup for local development with local qwen on my macbook. I tried localforge with mlx_lm.server but it failed to communicate (I saw a proof of concept on their page but now it seems to fail on "empty response" which in reality is not empty) Anyone could recommend a solution?
- zhobbs 1y agoPossible I'm misunderstanding what you're trying to do, but ollama works well for me for local inference with qwen on my Macbook Pro (32GB).
- nateb2022 1y agoYup, also using Ollama and on a Macbook Pro. Ollama is #1
- p0w3n3d 1y agoBut isn't ollama only local chat? Or I am missing something? I'd like to setup it as a server for my usages on another laptop (use it as my local AI hub) and would love to integrate it with some IDE using MCP
- grosswait 1y agoNo, it can listen on 0.0.0.0 or you can serve it through a proxy
- vadepaysa 1y agoWhile the speeds are great, in my experience with Cerebras, its really hard to get any actual production level rate limits or token quantity allocations. We cannot design systems around them and we use other vendors. We've spoken to their sales teams, and we've been told no.
- westurner 1y ago> Qwen3-235B uses an efficient mixture-of-experts architecture that delivers exceptional compute efficiency, enabling Cerebras to offer the model at $0.60 per million input tokens and $1.20 per million output tokens—less than one-tenth the cost of comparable closed-source models. $ 0.60/million input tokens $ 1.20/million output tokens How many minutes of 4K YouTube HDR video is that equivalent to in kWh of energy usage? > Concurrent with this launch, Cerebras has quadrupled its context length support from 32K to 131K tokens—the maximum supported by Qwen3-235B.
- sneilan1 1y agoSo I installed litellm proxy, pointed it at the new Cerebras API with Qwen-235B and hooked up Aider to litellm. This is not as good as claude code yet but it's so much faster. I even tried using the leaked claude code prompt into Aider but it doesn't do what I expect. Still worth trying but I learned that claude code's prompt is very specific to claude. I think this is very promising however! Aider basically spat out a bunch of text, installed some stuff, made some web calls & exited. WAS REALLY FAST LOL. you can repeat my experiment quickly with the following> config.yaml for litellm ``` model_list: - model_name: qwen3-235b litellm_params: model: cerebras/qwen-3-235b-a22b api_key: os.environ/CEREBRAS_API_KEY api_base: https://api.cerebras.ai/v1 https://api.cerebras.ai/v1 ``` run litellm with ``` litellm --config config.yaml --port 4000 --debug ``` (may need to install litellm[proxy]) start aider with ``` aider --model cerebras/qwen-3-235b-a22b --openai-api-base http://localhost:4000 http://localhost:4000 --openai-api-key fake-key --no-show-model-warnings --auto-commits --system-file ./prompt.txt --yes ``` install whatever you need with pip etc. prompt.txt contains the leaked claude code prompt which you can find yourself on the internet.