6 ms·
Watching the livestream now, the improvement over their current models on the benchmarks is very small. I know they seemed to be trying to temper our expectatio
by doctoboggan 1y ago
Watching the livestream now, the improvement over their current models on the benchmarks is very small. I know they seemed to be trying to temper our expectations leading up to this, but this is much less improvement than I was expecting
- famouswaffles 1y agoI mean that's just the consequence of releasing a new model every couple months. If Open AI stayed mostly silent since the GPT-4 release (like they did for most iterations) and only now released 5 then nobody would be complaining about weak gains in benchmarks.
- moduspol 1y agoWell it was their choice to call it GPT 5 and not GPT 4.2.
- famouswaffles 1y agoIt is significantly better than 4, so calling it 4.2 would be rather silly.
- amilios 1y agoIs it? That's not super obvious from the results they're showing.
- famouswaffles 1y agoYes it is, if we're talking about the original GPT-4 release or even GPT-4o. What about the results they've shown is not obvious?
- amilios 1y agoI see incremental improvements in almost all domains?
- jononor 1y agoIf everyone else had stayed silent as well, then I would agree. But as it is right now they are juuust about managing to match the current pace of the other contenders. Which actually is fine, but they have previously set quite high expectations. So some will probably be disappointed at this.
- cardine 1y agoIf they had stayed silent since GPT-4, nobody would care what OpenAI was releasing as they would have become completely irrelevant compared to Gemini/Claude.
- lawlessone 1y agoim sure i am repeating someone else but sounds like we're coming over the s-curve
- Bluestein 1y agoMy thought exactly.- Diminished returns.- ... here's hoping it leads to progress.-
- z7 1y agoGPT-5 is #1 on WebDev Arena with +75 pts over Gemini 2.5 Pro and +100 pts over Claude Opus 4: https://lmarena.ai/leaderboard https://lmarena.ai/leaderboard
- virgildotcodes 1y agoThis same leaderboard lists a bunch of models, including 4o, beating out Opus 4, which seems off.
- afro88 1y agoIn my experience Opus 4 isn't as good for day to day coding tasks as Sonnet 4. It's better as a planner
- zamadatix 1y ago"+100 points" sounds like a lot until you do the ELO math and see that means 1 out of 3 people still preferred Claud Opus 4's response. Remember 1 out of 2 would place the models dead even.
- degrews 1y agoThat eval hasn't been relevant for a while now. Performance there just doesn't seem to correlate well with real-world performance.
- Too 1y agoWhat does +75 arbitrary points mean in practice? Can we come up with units that relate to something in the real world.
- wahnfrieden 1y agoIt is at least much cheaper and seems faster. They also announced gpt-5-pro but I haven't seen benchmarks on that yet.
- doctoboggan 1y agoI am hoping there is a "One more thing" that shows the pro version with great benchmark scores
- Workaccount2 1y agoSam said maybe two years ago that they want to avoid "mic drop" releases, and instead want to stick to incremental steps. This is day one, so there is probably another 10-20% in optimizations that can be squeezed out of it in the coming months.
- bigmadshoe 1y agoThen why increment the version number here? This is clearly styled like a "mic drop" release but without the numbers to back it up. It's a really bad look when comparing the crazy jump from GPT3 to GPT4 to this slight improvement with GPT5.
- Workaccount2 1y agoBecause it is a 100x training compute model over 4. GPT5.5 will be a 10X compute jump. 4.5 was 10x over 4.
- bigmadshoe 1y agoEven worse optics. They scaled the training compute by 100x and got <1% improvement on several benchmarks.
- reasonableklout 1y agoIs 1% relative to more recent models like o3, or the (old and obsolete at this point) GPT-4?
- bigmadshoe 1y agoIt was relative to the number the comment I replied to included. I would assume GPT-5 is nowhere near 100x the parameters of o3. My point is that if this release isn't notable because of parameter count, nor (importantly) performance, what is it notable for? I guess it unifies the thinking and non-thinking models, but this is more of a product improvement, not a model improvement.
- 827a 1y agoI have a suspicion that while the major AI companies have been pretty samey and competing in the same space for a while now, the market is going to force them to differentiate a bit, and we're going to see OpenAI begin to lose the race toward extremely high levels of intelligence instead choosing to focus on justifying their valuations by optimizing cost and for conversational/normal intelligence/personal assistant use-cases. After all, most of their users just want to use it to cheat at school, get relationship advice, and write business emails. They also have Ive's company to continue investing in. Meanwhile, Anthropic & Google have more room in their P/S ratios to continue to spend effort on logarithmic intelligence gains. Doesn't mean we won't see more and more intelligent models out of OpenAI, especially in the o-series, but at some point you have to make payroll and reality hits.
- juped 1y agoI think this is pretty much what we've already seen happening, in fact.
- anyg 1y agoAlso, the code demos are all using GPT-5 MAX on Cursor. Most of us will not be able to use it like that all the time. They should have showed it without MAX mode as well
- hodgehog11 1y agoThe hallucination benchmarks did show major improvement. We know existing benchmarks are nearly useless at this point. It's reliability that matters more.
- jama211 1y agoI’m more worried about how they still confidently reason through things incorrectly all the time, which isn’t quite the same as hallucination, but it’s in a similar vein.
- CamperBob2 1y agoYeah, people never do that. Or at least I don't. I don't know about you.
- jama211 1y agoThere’s a name for the fallacy you’re using.
- davidhs 1y ago> I know they seemed to be trying to temper our expectations leading up to this Before the release of the model Sam Altman tweeted a picture of the Death Star appearing over the horizon of a planet.
- blitzar 1y agoIs he suggesting his company is designed with a womp rat sized opening that if you shoot a bullet into makes the whole thing explode?
- karaterobot 1y agoYou know, I used to bullseye small thermal exhaust ports in my T16 back home, they're not much smaller than womp rats.
- scubakid 1y agoYou know, I used to bullseye T16s in my womp rat back home, they're not much bigger than thermal exhaust ports.
- artursapek 1y agolol
- softwaredoug 1y agoHe also said he had an existential crisis that he was completely useless now at work.
- shafyy 1y agoGood that he finally came to the realization lol
- diogolsq 1y agoLaw of diminishing returns. We’re talking about less than a 10% performance gain, for a shitload of data, time, and money investment.
- illiac786 1y agoYeah I think that throwing more and more compute at the same training data produces smaller and smaller gains. Maybe quantum compute would be significant enough of a computing leap to meaningfully move the needle again.
- stoneyhrm1 1y agoWhat exactly is being moved? It's trained on human data, you can't make code more perfect than what is written out there by a human.
- illiac786 1y agoSome think it’s possible, I don’t, we agree actually.
- gwd 1y agoI'm not sure what "10% performance gain" is supposed to mean here; but moving from "It does a decent job 95% of the time but screws it up 5%" to "It does a decent job 98% of the time and screws it up 2%" to "It does a decent job 99.5% of the time and only screws it up 0.5%" are major qualitative improvements.