6 ms·
I've been waiting so long for something amazing to come out of the OpenAI and Cerebras collaboration. > In our evaluations, GPT-5.6 Sol on Ultrafast mode answe
by iamcoder18 3mo ago
I've been waiting so long for something amazing to come out of the OpenAI and Cerebras collaboration.
> In our evaluations, GPT-5.6 Sol on Ultrafast mode answered all 2,500 HLE questions in 11 hours and 11 minutes. Claude Fable 5 needed 78 hours and 27 minutes, more than three days of continuous compute, to arrive at the same conclusions. In other words, Ultrafast worked through the frontier of human knowledge in a single working day, achieving comparable accuracy nearly 7× faster.
This is actually insane.
Hopefully the release ultrafast of Terra and Luna too.
- piyh 3mo agoFeels like the 90's again where single threaded speed is improving fast. ASICs and wafer scale rather than node shrinks, but end result to me the consumer feels the same.
- sscaryterry 3mo agoYep, https://en.wikipedia.org/wiki/Andy_and_Bill%27s_law https://en.wikipedia.org/wiki/Andy_and_Bill%27s_law
- beng-nl 3mo agoI wish, but I don’t think this qualifies as single thread performance. Cerebras is whole wafer computation though right? So you get the result sooner, but the same amount of computation has occurred, just with huge parallelism.
- wrsh07 3mo agoSeems like they will do Sol first while capacity constrained? I can't imagine the margins they'll be charging
- sscaryterry 3mo agoI think this is what they need to actually have a margin :)
- zozbot234 3mo agoAnswering 2,500 independent questions is an embarrassingly parallel workload, all it needs is scale out. It would be more meaningful to know how much time was required for a single complete answer to a difficult HLE question.
- desmondl 3mo agoThe animation in the article shows GPT-5.6 Sol Ultrafast solving a question at 3 secs vs Fable's 27 secs. The "11 hour vs 78 hours" animation implied that they were solved in sequence and not in parallel
- whimsicalism 3mo agoi assume they meant serially....
- jcims 3mo ago>Answering 2,500 independent questions is an embarrassingly parallel workload So is calculating the total time required to answer all of the questions.
- baq 3mo agoAmadahl’s law dictates it’s approximately always better (as in, more efficient computationally) to have one super fast thing than many slower things doing the same job in parallel.
- moralestapia 3mo agoSuch a deep and smart insight. But why is that important if they're measuring time?
- moralestapia 3mo agoAnd just like that, GP disappears forever ...
- xyzsparetimexyz 3mo agoYes. but for embarrassingly parallel tasks it barely matters.
- sixtyj 3mo agoOutput from Cerebras with GPT model is 750 tokens per second. Don’t blink. (Chatjimmy has 14,200 TPS.)
- tomrod 3mo agoChatJimmy is a much smaller model and, AFAIK, has no reasoning capability. Absolutely insane raw speed, like a supercar, while Sol is more like a freight truck.
- sixtyj 3mo agoAt such output speed, I wouldn’t expect reasoning. (But I didn’t know it, thanks.) 700 TPS with reasoning is awesome and it speeds things up. Cerebras as public traded company is worth keeping an eye what they produce.
- msdz 3mo ago> At such output speed, I wouldn’t expect reasoning. As the sibling comment to yours mentioned, if they had a reasoning model “hardware-ified” onto a custom chip (as is their plan for IIRC this or next year, a new ASIC), it’d output fast decode speeds for the regular output as well as reasoning sections. Both would be ≈equally fast.
- senordevnyc 3mo agoYeah, I thought reasoning was literally just chain of thought in the output token stream, with the model itself adding delimiters to indicate what part of the output is internal reasoning, and what part is an answer to the user. Is that wrong?
- beering 3mo agoYou are right, reasoning is unrelated to tokens per second.
- 3mo ago
- rvz 3mo agoBeen waiting since Cerebras-GPT. [0] [0] https://news.ycombinator.com/item?id=35490837 https://news.ycombinator.com/item?id=35490837
- christoff12 3mo agoAn irrational gripe of mine is how GPT uses 7× instead of 7x. I recognize that the former is the multiplication symbol, but I don't think it should be used that way.
- Daneel_ 3mo agoIf anything, writing "7x" is actually just a poor emulation of writing "7×". Saying something is "done at 7x speed" should be read as "done at seven times speed" not as "done at seven x speed". So using the 'times' (multiplication) symbol is the better form in my opinion; it just happens to be significantly easier to type "x" instead, which is how we got here.
- TacticalCoder 3mo agoBut speech evolves. And we all pronounce "ten ex", not "ten times". Actual speech usage very clearly pronounce the letter 'x'. So it's 10x. And no need for Unicode codepoints.
- Daneel_ 3mo agoI fully agree that speech evolves, but maybe this is just in the US? I've never really heard anyone say it here in Australia, it's usually "times", although admittedly almost no one uses the phrase/format here.
- christoff12 3mo agoAh, yes. My gripe is certainly US American flavored.
- shussjjs 3mo agoA number plus an “x” is used beyond Trumpistan. I have never in my life heard anyone say “ten ex”.
- _flux 3mo ago
- andai 3mo agoI'm finding Luna suprisingly adequate for my work. I slept on it due to the benchmarks, but it's very fast and even on low reasoning I'm finding it more than adequate for "menial" work. (The speed is crucial for "interactive" work -- if a model is fast enough it goes from "async" to "real time", subjectively, which is a huge difference.) In fact, I'd say it's overqualified for the kind of work I'm doing, because it spends >half the time verifying trivial changes (and the verification isn't as helpful as you'd expect, even with bigger models). Maybe I can prompt it to be less aggressive about that (the new GPT models do it even without prompting). Anyway, Ultrafast Luna would be amazing, though I strongly doubt they can offer Cerebras at anything approaching the current prices. Now we wait for Moore's Law? :)
- epolanski 3mo agoI feel odd to use these models, because it feels like a faster model doesn't feel that much faster if it spends reading files, making edits and running checks. It feels too situational.
- andai 3mo agoWhich one of those takes time?
- phoghed 3mo agoLuna has impressive results on DeepSWE. I mostly split work between Luna and Sol. If something seems simple enough I always try it with Luna first.
- shepherdjerred 3mo agoLuna is incredible considering the cost. It’s very good for routine work.
- trollbridge 3mo agoLuna and Terra are great, but so are DeepSeek Flash, MiMo, Qwen 3.6, and even Gemma for some things. MiMo Pro has had UltraSpeed for a while.
- 3mo ago
- aetherspawn 3mo agoI discovered yesterday that the “amazing thing that comes out of OpenAI” is Sol, due to its token efficiency. Dollar for tokens, Sol and Fable are the same price. However, Sol uses (literally: in testing) around 10-100x less output tokens compared to Fable for the same task. We run our frontier models nearly 24/7, so switching to Sol will save us around $500 per day. And, due to less guardrails, Sol also performed better, and we lost less tokens due to guardrails shutting down sessions (I feel like it’s illegal to take $50 of someone’s token money and then shut down a session with guardrails before they get an answer, and yet Anthropic do it to us constantly… either take our money and commit, or trigger the guardrails immediately)
- minraws 3mo agoWait isn't Fable like 2x more expensive if we compare under 272k tokens
- ChadMoran 3mo agoThe comment you're replying to almost feels like it was written by a bot or am I crazy?
- w4yai 3mo agoI agree. Weird to use <“> and <”> characters. Maybe written on phone, but certainly not on keyboard.
- gunalx 3mo agoCommon on non us keebs though.
- jrflo 3mo agoOther languages use different characters for quotes, if anything that's an indication that's not written by a LLM because it's not favoring the standard English character. https://en.wikipedia.org/wiki/Quotation_mark#Specific_language_features https://en.wikipedia.org/wiki/Quotation_mark#Specific_langua...
- krzyk 3mo agoIsn't ultrafast just making hundreds of subagents?
- anentropic 3mo agoNo, that's Claude "ultracode"