4 ms·
Looks like solid incremental improvements. The UI oneshot demos are a big improvement over 4.6. Open models continue to lag roughly a year on benchmarks; pretty
by vessenes 9mo ago
Looks like solid incremental improvements. The UI oneshot demos are a big improvement over 4.6. Open models continue to lag roughly a year on benchmarks; pretty exciting over the long term. As always, GLM is really big - 355B parameters with 31B active, so it’s a tough one to self-host. It’s a good candidate for a cerebras endpoint in my mind - getting sonnet 4.x (x<5) quality with ultra low latency seems appealing.
- deleted 9mo ago[deleted]
- mckirk 9mo agoNote that this is the Flash variant, which is only 31B parameters in total. And yet, in terms of coding performance (at least as measured by SWE-Bench Verified), it seems to be roughly on par with o3/GPT-5 mini, which would be pretty impressive if it translated to real-world usage, for something you can realistically run at home.
- deleted 9mo ago[deleted]
- HumanOstrich 9mo agoI tried Cerebras with GLM-4.7 (not Flash) yesterday using paid API credits ($10). They have rate limits per-minute and it counts cached tokens against it so you'll get limited in the first few seconds of every minute, then you have to wait the rest of the minute. So they're "fast" at 1000 tok/sec - but not really for practical usage. You effectively get <50 tok/sec with rate limits and being penalized for cached tokens. They also charge full price for the same cached tokens on every request/response, so I burned through $4 for 1 relatively simple coding task - would've cost <$0.50 using GPT-5.2-Codex or any other model besides Opus and maybe Sonnet that supports caching. And it would've been much faster.
- deleted 9mo ago[deleted]
- twalla 9mo agoI hope cerebras figures out a way to be worth the premium - seeing two pages of written content output in the literal blink of an eye is magical.
- Miraste 9mo agoI wonder why they chose per minute? That method of rate limiting would seem to defeat their entire value proposition.
- p91paul 9mo agoIn general, with per minute rate limiting you limit load spikes, and load spikes are what you pay for: they force you to ramp up your capacity, and usually you are then slow to ramp down to avoid paying the ramp up cost too many times. A VM might boot relatively fast, but loading a large model into GPU memory takes time.
- mlyle 9mo agoThe pay-per-use API sucks. If you end up on the $50/mo plan, it's better, with caveats: 1 million tokens per minute, 24 million tokens per day. BUT: cached tokens count full, so if you have 100,000 tokens of context you can burn a minute of tokens in a few requests.
- solarkraft 9mo agoIt’s wild that cached tokens count full - what’s in it for you to care about caching at all then? Is the processing speed gain significant?
- mlyle 9mo agoNot really worth it, in general. It does reduce latency a little. In practice, you do have a continuing context, though, so you end up using it whether you care or not.
- behnamoh 9mo ago> The UI oneshot demos are a big improvement over 4.6. This is a terrible "test" of model quality. All these models fail when your UI is out of distribution; Codex gets close but still fails.
- ttoinou 9mo agoSonnet was already very good a year ago, do open weights model right are as good ?
- jasonjmcghee 9mo agoFwiw Sonnet 4.5 is very far ahead of where sonnet was a year ago
- cmrdporcupine 9mo agoFrom my experience, Kimi K2, GLM 4.7 (not flash, full), Mistral Large 3, and DeepSeek are all about Sonnet 4 level. I prefer GLM of the bunch. If you were happy with Claude at its Sonnet 3.7 & 4 levels 6 months ago, you'll be fine with them as a substitute. But they're nowhere near Opus 4.5
- Workaccount2 9mo agoUnless one of the open model labs has a breakthrough, they will always lag. Their main trick is distilling the SOTA models. People talk about these models like they are "catching up", they don't see that they are just trailers hooked up to a truck, pulling them along.
- skrebbel 9mo agoHow does this work? Do they buy lots of openai credits and then hit their api billions of times and somehow try to train on the results?
- deleted 9mo ago[deleted]
- g-mork 9mo agodont forget the plethora of middleman chat services with liberal logging policies. i've no doubt there is a whole subindustry lurking in here
- skrebbel 9mo agoi wasn't judging, i was asking how it works. why would openai/anthrophic/google let a competitor scrape their results in sufficient amounts that it lets them train their own thing?
- victorbjorklund 9mo agoI think the point is that they can't really stop it. Let's say that I purchase API credits, and I let the resell it to DeepSeek. That's going to be pretty hard for OpenAI to figure out and even if they figure it out and they stop me there will be thousands of other companies willing to do that arbitrage. (Just for the record, I'm not doing this, but I'm sure people are.) They would need to be very restrictive about who is allowed to use the API and not and that would kill their growth because because then customers would just go to Google or another provider that is less restrictive.
- pseudony 9mo agoI hear this said, but never substantiated. Indeed, I think our big issue right now is making actual benchmarks relevant to our own workloads. Due to US foreign policy, I quit claude yesterday and picked up minimax m2.1 We wrote a whole design spec for a project I’ve previously written a spec for with claude (but some changes to architecture this time, adjacent, not same). My gut feel ? I prefer minimax m2.1 with open code to claude. Easiest boycot ever. (I even picked the 10usd plan, it was fine for now).
- deleted 9mo ago[deleted]