25 ms·
Easily the most interesting part of this announcement is buried in the second to last paragraph: "We're also launching GPT‑5.6 Sol on Cerebras at up to 750 tok
by gandreani 3mo ago
Easily the most interesting part of this announcement is buried in the second to last paragraph:
"We're also launching GPT‑5.6 Sol on Cerebras at up to 750 tokens per second in July, bringing frontier intelligence to customers at unprecedented speed. Access will initially be limited to select customers as we expand capacity."
750 tokens/s on a frontier model is going to be extremely interesting. I doubt this new version is anything but a version bump in terms of capabilities but if we can start getting these answers back faster, they end up being more useful.
Just off the top of my head, I can think of the tedious task of finding certain functionality within a codebase. I usually can't beat an AI agent harness at this task today. If the AI model is 3x faster I have less of chance.
- helloplanets 3mo agoOpenAI also announced two days ago that they're starting to make Cerebras style chips themselves [0], will be interesting to see how fast SotA model inference will be by the end of the year. [0]: https://openai.com/index/openai-broadcom-jalapeno-inference-chip/ https://openai.com/index/openai-broadcom-jalapeno-inference-...
- WarmWash 3mo agoCerebras is different than what jalapeno is. Jalepeno is for mass scale inference. Cerebras is extremely expensive and difficult to scale, hence the limited release.
- jauntywundrkind 3mo agoI don't see any indications that OpenAI is doing wafer-scale work. I tend to doubt they would. Cerebras notably doesn't have a kv, is wildly high bandwidth, but within/across the chip, not able to dump/restore kv super well. I doubt openai is going to build something that is as expensive to run. Also, wafer-scale is absurdly hard & weird to pull off, so I doubt that would be their first foray.
- mlyle 3mo agoI don't understand how you refer to this as "Cerebras-style". Cerebras is wafer-scale and unique. Jalapeno is an inference-optimized conventional chip.
- paxys 3mo agoEven if their chip is a difference maker, end of the year is wayy too optimistic. It’ll at minimum be a multi-year effort to bring it to production at scale.
- sberens 3mo agoFor comparison, openrouter says opus 4.8 is ~55 tokens/s and fast mode is ~102. 750 tokens/s for their largest model is going to be nuts
- gandreani 3mo agoUsing gpt-5.4-mini in off-peak hours already feels like super-speed to me. That's probably no more than 100-150 tk/s. I can't imagine 750! I've always eyed Cerebras but never had a use for it that would justify paying for the API directly. Although now that I think about it, trying out the API would probably cost less than a subscription for a month...
- kegs_ 3mo agoI have a pretty good use case for gpt-oss. The amount of time savings has actually been wild. Definitely worth a try. Just to be clear, it gets like 2000tok/s
- embedding-shape 3mo agoThe ChatGPT subscription gives you access to the -spark model(s) in Codex which are blazing fast (but pretty dumb) which I think runs on Cerebras hardware too.
- rrvsh 3mo agois this specifically in codex? have been trying to use the models for months on opencode then pi but it says chatgpt subscriptions don't have access to it - i was under the assumption that OpenAI doesn't lock down their models based on harness a la Claude Code
- cactusplant7374 3mo agoWhat plan are you on? It is only available to Pro users.
- 3mo ago
- tontinton 3mo agoYep this is a glimpse into the future of 500+ t/s, which is in my opinion the next big thing that validates Jevon's paradox (the models are already smart enough)
- Gingersnap123 3mo ago[dead]
- devmor 3mo ago“Smart enough” really depends on how many other people have encountered a problem close enough to yours and solved it somewhere on the open internet, IMO. Most of the frontier models can, when prompted and tooled correctly, do a lot of “reasoning” tasks that amount to resolving how the user has explained a particular widely known paradigm. The more difficult and obscure the issues you provide them with, the faster you notice them reward hacking by altering the criteria until they are no longer attempting to solve the problem. Using “advisor” style loops helps hold this off at the cost of tokens, but there is still a fairly short limit at which they will essentially give up if they can’t find all of the necessary information - sometimes the issue is actually worse if they find a small amount of information instead of nothing - they’ll extrapolate from that tiny piece of data and generate plausible-sounding hallucinations almost every time. And god forbid your problem involves doing something a different way than the majority of people do it. Unless you can write a full spec on it, the models will repeatedly spiral back into adjusting everything about your problem until it matches one of the most popular approaches in their training data.
- vb-8448 3mo ago> how many other people have encountered a problem close enough to yours and solved it somewhere on the open internet I'm 100% sure that all our web, cc, codex or whatsoever sessions are used in the training, RL or either both. This makes the size of the universe models know about at least one order of magnitude bigger than the open internet.
- beepbooptheory 3mo ago
- cruffle_duffle 3mo ago"we can start getting these answers back faster, they end up being more useful." Dude, 10x token speed is going to be absolutely nuts. Half the "parallel subagent workflow" business seems to be driven simply as a means to avoid tapping your thumbs waiting for the infernal robot to finish something. If things come back speedy quick all the time, it should keep up with the "speed of the human" and let me stay focused on one thread instead of half a dozen. Plus the cost of screwing up gets significantly lower because you just re-fire with an adjusted prompt and iterate. Someday these things will be 100x as fast as they are today and that is when things will get insane.
- Terretta 3mo agoit also makes the parent brain-dead because all those subtokens are missing from the context thus unable to steer the hyper dimensional context driven generation, and the subagent is dumb as a post so synthesizes something very weedsy while you're specifically attempting to understand the forest
- RALaBarge 3mo agoYou have an agent spawn the agents for you! You can ask Claude to do it for you, he is happy to use sonnet when you ask for grok and opus high when you ask for deepseek.
- eli 3mo agoI'm skeptical of how fast "up to" 750t/s really means. Maybe if they make it extremely expensive so it frees up enough capacity? GPT‑5.3‑Codex‑Spark currently runs on Cerebras chips and it's giving me around 150t/s. Still relatively very fast, but nowhere near the 1,000t/s they claimed at launch. (Also it's not a very good model.) That said, I'm super bought in to faster models being better for most use cases than smarter models.
- beering 3mo agoSoon the bottleneck will be how fast your laptop can grep for a string.
- aurareturn 3mo agoIf it's 150 t/s, that's barely faster than Nvidia GPUs who are batching a lot more and are a lot more cost effective. Add in the Groq piece and Nvidia claims it can do 400 tokens/s.
- eli 3mo agoI assume it’s just oversubscribed. I’m sure it “can” go faster. But yeah that was my point.
- donquichotte 3mo ago> I can think of the tedious task of finding certain functionality within a codebase. I usually can't beat an AI agent harness at this task today. Yup, I remember "racing" the AIs to figure things out in codebases just a year ago. Today, I have no chance. Whether it is due to degraded reasoning capabilities on my part or better models, I don't know.
- abustamam 3mo agoAt least in my case, much of the code in the codebase I'm working on is AI generated so even if I have an accurate mental model of how everything works, I have no idea where any of it is located or named.
- rasengan 3mo agoTo be fair, whenever I join a pre-existing code-base [1], it's the same. I have no idea and have to map it out ;) [1] Not AI codebases (and of course, AI code bases I guess)
- markrages 3mo agoI can't be the only one whose memory is so bad that I am like this in my own code base.
- seekup 3mo agoI seem to remember - but cannot find, even with an AI boost - someone's "law of computing" or somesuch describing the amount of time that has to pass before code you wrote is indistinguishable to you from code written by someone else. At any rate the interval is not so long.
- pipes 3mo agoYou are not. :) my memory is disturbingly fried.
- anon7725 3mo ago
- motoboi 3mo agobean in mind that "GPT‑5.6 Sol on Cerebras at up to 750 tokens per second" not necessarily means the same model (in terms of inference result). It can mean anything like a very quantized model, a different level of model activation per inference etc. Of course we can trust that wouldn't name the same thing with different levels of intelligence, right? Right?
- beering 3mo agoyeah but it’s trivial to just try it out and compare.
- lostmsu 3mo agoDoes the Cerebras variant offer input caching and corresponding discounts? Last I checked Cerebras would not cache or would cache but not give discounts for the cached input, making it impractical for agentic use and multiturn conversations.
- swalsh 3mo agoThis would be amazing for some of our "real-time" workflows, that need to fallback to AI for one reason or another. What used to happen is a rules based system did the majority of work, and occasional corner case would fall back to humans. Then we moved AI in, still not real time, but much faster. Cerebras could make that even faster.
- bob1029 3mo agoAt a certain rate we will be able to move towards continuous / real-time inference systems. The discrete, turn based solutions are quite confining with how they must be trained. Continuous and real-time would fundamentally alter the domain. From an information theory perspective we are still in dial-up territory with regard to the actual information rate. 750 tokens per second would be a really bad dialup connection. Imagine 10 millions tokens per second.
- dennisy 3mo agoThat would be interesting. Do you feel most of the speed upgrade will come from the software or hardware side?
- mikepurvis 3mo agoIs there anyone exploring or writing about this in public? I've felt for a while that the turn-based model was not quite right, but also felt too stupid and ill-informed to have much of an opinion about what else it could be.
- b112 3mo agoI have an active 'sleep' mode, where when the user is AFK the LLM goes into a loop with a sleep 10 between turns, and determines (via tool use) if something should be done. That's still a 'turn' in a way, but it's all the LLM just sort of sitting around like a human would, pondering what to do next. But I could imagine after each space(eg, word) having a 27b model on a nice rig, with thinking off, doing a quick look at the sentence and determine if it should interrupt and start a real turn with thinking on. Which kind of is non-turn based in a way. If you're typing fast, it might hit that run every 3 or 4 words, but that's sort of how a human might be when a person is talking to them. That is, waiting for enough info to interrupt, if needed. There might be a way to process chunks of a sentence using commas as break points, eg for comma delimitated phrases in sentences, so the whole sentence doesn't need to be re-processed each "should I break in" assessment at word break. Could be fascinating. Could actually do some of this right now. I don't think this is what the parent poster was thinking, but the idea even at this level seems fun.
- ai_fry_ur_brain 3mo agoFrom what I know about batch processing/ concurrency in inference this is a pipe dream... Or its going to cost an arm and a leg. I think they're lying or its going to be a much smaller model and not "frontier"
- kolinko 3mo agoYou have speculative decoding that easily increases speed 2-4 times with no loss of quality, and of course MoA architectures that speed up inference 10 times or more, although with some quality loss. Better hardware, and other techniques on top of that and you speed up even further.
- qznc 3mo agohttps://mikeveerman.github.io/tokenspeed/?rate=750&mode=think https://mikeveerman.github.io/tokenspeed/?rate=750&mode=thin... This is what 750tps looks like, I guess.
- buddhistdude 3mo agoJust to think what this will look like in a couple of years.
- OGWhales 3mo agoHopefully like this (but smarter): https://chatjimmy.ai/ https://chatjimmy.ai/
- nomel 3mo agoThis is genuinely confusing to my senses. The future is going to be so strange/neat/me unemployed.
- falcor84 3mo ago> strange/neat/me unemployed I'm not sure if that's what you were going for, but I read it as if it were written by The Board in the game Control, and found myself with the appropriate level of existential dread.
- matheusmoreira 3mo agoThe future is totally illegible to me. I love these AI models, but I feel like I'm going to be jobless within 10 years. Anomie is at an all time high right now.
- TacticalCoder 3mo ago> I usually can't beat an AI agent harness at this task today. If the AI model is 3x faster I have less of chance. Yes: we have these new tools that are extremely good at helping us search through our codebases. Not just to find where/how functionalities are implemented: IMO bug searching is even way more powerful. But: why would you want to compete with AI to do that? I cannot compete with grep/ripgrep... And I'm cool with that. This lets you focus more on the more interesting parts, where AI/LLMs suck fat balls.
- _fat_santa 3mo agoI still use GPT-5.3-codex-spark which also runs on the Cerebras chips. Spark can run at >1000 tok/s but it's highly limited in it's context window size so it's not suitable many workflows. Granted this will be a bit slower (relatively speaking) but it will still be awesome.
- philip1209 3mo agoSame - I had some "AI-assisted coding interviews" where I had to bring my own AI tools, and found the speed of codex-spark to be important for making progress quickly (and not sitting there waiting on Opus to think for 10 minutes).
- js2 3mo ago> second to last There's a word for this that you should never pass up an opportunity to use: penultimate. (You should also never pass up the opportunity to use "defenestrate," but it sadly does not apply here.)
- kolinko 3mo agoA friend of mine had his visa accepted because of this. He was explaining what he plans to do in US and he threw in “penultimate” into a sentence somewhere. The council stopped him, said that if he knows such words he definitely won’t overstay his visit to work as a dishwasher, and accepted his B1/B2. Seriously. Not sure if it would be the same if he used “defenestrate” when talking about his plans.
- v-erne 3mo ago>> penultimate Oh that's a word I haven't heard in a while. And I know it mainly thanks to Monty Python great sketch: https://m.youtube.com/watch?v=l9Aj7W3g1qo https://m.youtube.com/watch?v=l9Aj7W3g1qo
- linzhangrun 3mo agoI saw videos of coding with Mimo-V2.5-Pro UltraSpeed, which is advertised at 1,000 tokens/s, which is very impressive.: https://www.bilibili.com/video/BV1fME16uEW7 https://www.bilibili.com/video/BV1fME16uEW7 If the time-to-first-token latency also greatly improved, this could be very useful for end-to-end in controls, like autonomous driving for example.
- trollbridge 3mo agoIt’s awesome, particularly since it’s at DeepSeek tier prices (3X of DS-V4-Pro). At 1,000 tok/sec though you can really rip through tokens. (About $9 an hour if you manage to run the output nonstop.) It tends to cost more than DS since it doesn’t seem to have as many input cache hits.
- nop17 3mo ago3x faster burn than 3x expensive token, generate more tokens, more fees
- jeswin 3mo agoAt thousands of tokens per second, LLMs (harnesses) can start to do a broader tree search of possibilities even in inefficient token space. This unlocks capabilities outside programming.
- yiyingzhang 3mo agoIt all depends on the context window size. A small context size with fast performance won't be very useful today, as most workloads (like requests behind codex) usually have very long context.
- Cryptosale75 3mo agoCerebras is Milli Vanilli. They spend 10 years burning cash on a failed idea (which is frankly insane, since they should have figured out the limitations of heir stack in like... a weekend) and struck accidental gold with their 'Giant ass wafer'. The company is valued like they broke open the grail, when in reality it's more like they bought a Cybertruck, got it stuck in the mud, and realized "You know what this thing does better than all other cars... shovel mud" I'm shorting Cerebras with margin to virtually zero.
- easygenes 3mo agoThis is a strange one. We know the hardware capabilities of Cerebras force them to do aggressive REAP pruning to serve Kimi K2.6. Meaning that about 750B parameters is the upper limit of what they can serve economically. Not sure if this means Sol is smaller than anyone thinks or that they're just going to charge so much that a very inefficient serving regime is feasible.
- Avery29 3mo agoThe speed sounds great，faster models make that gap much more visible..
- qnleigh 3mo agoLast I heard, Cerebras chips were entire wafers and would be extremely expensive. How could OpenAI possibly have enough of these to serve a popular model at scale?
- kingreflex 3mo agothis means they also earn at a faster rate in some setups :)
- trollbridge 3mo agoThis is something Xioami already did with MiMo-2.5-Pro a month ago, and at a higher speed (1,000 t/s). 750 tps at GPT-5.5-Pro prices would be ruinous!
- _boffin_ 3mo agoThat statement gives a lot of insights into the possible model size. Llama 3.1 405b runs @ ~900t/s: https://www.cerebras.ai/blog/llama-405b-inference https://www.cerebras.ai/blog/llama-405b-inference From initial vibe research (which is totally not correct by any means) ~13.5k concurrent streaming clients capacity