4 ms·
The marketing copy and the current livestream appear tautological: "it's better because it's better." Not much explanation yet why GPT-5 warrants a major versi
by minimaxir 1y ago
The marketing copy and the current livestream appear tautological: "it's better because it's better."
Not much explanation yet why GPT-5 warrants a major version bump. As usual, the model (and potentially OpenAI as a whole) will depend on output vibe checks.
- krat0sprakhar 1y ago> Not much explanation yet why GPT-5 warrants a major version bump Exactly. Too many videos - too little real data / benchmarks on the page. Will wait for vibe check from simonw and others
- collinmanderson 1y ago> Will wait for vibe check from simonw https://openai.com/gpt-5/?video=1108156668 https://openai.com/gpt-5/?video=1108156668 2:40 "I do like how the pelican's feet are on the pedals." "That's a rare detail that most of the other models I've tried this on have missed." 4:12 "The bicycle was flawless." 5:30 Re generating documentation: "It nailed it. It gave me the exact information I needed. It gave me full architectural overview. It was clearly very good at consuming a quarter million tokens of rust." "My trust issues are beginning to fall away" Edit: ohh he has blog post now: https://news.ycombinator.com/item?id=44828264 https://news.ycombinator.com/item?id=44828264
- dimitri-vs 1y agoThis effectively kills this benchmark.
- tuesdaynight 1y agoHonestly, I have mixed feelings about him appearing there. His blog posts are a nice way to be updated about what's going on, and he deserves the recognition, but he's now part of their marketing content. I hope that doesn't make him afraid of speaking his mind when talking about OpenAI's models. I still trust his opinions, though.
- croemer 1y agoYeah, even if he wasn't paid to appear there, this seems a bit too close.
- layer8 1y agoThe pelican is still a mess.
- bardak 1y agoI feel like we need to move on from using the same test on models since as time goes on the information about these specific test is out there in the training data and while i am not saying that it's happened in this case there is nothing stopping model developers from adding extra data for theses tests directly in the training data to make their models seem better than they are
- laurent_du 1y agoDamn Theo is really a handsome dude.
- anthonypasq 1y agoits >o3 performance at gpt4 price. seems pretty obvious
- thegeomaster 1y agoo3 pricing: $8/Mtok out GPT-5 pricing: $10/Mtok out What am I missing?
- anthonypasq 1y agopretty sure reduced cache input pricing is a pretty big deal for reasoning models, but im not positive
- thegeomaster 1y agoIt just matches the 90% discount that Claude models have had for quite a while. I don't see anything groundbreaking...
- throwaway0123_5 1y agoIt seems like you might need less output tokens for the same quality of response though. One of their plots shows o3 needing ~14k tokens to get 69% on SWE-bench Verified, but GPT-5 needing only ~4k.
- mitkebes 1y agoO3 has had some major price cuts since Gemini 2.5 Pro came out. At the time, o3 cost $10/Mtok in and $40/Mtok out. The big deal with Gemini 2.5 Pro was it had comparable quality to o3 at a fraction of the cost. I'm not sure when they slashed the o3 pricing, but the GPT-5 pricing looks like they set it to be identical to Gemini 2.5 Pro. If you scroll down on this page you can see what different models cost when 2.5 Pro was released: https://deepmind.google/models/gemini/pro/ https://deepmind.google/models/gemini/pro/
- joshmlewis 1y agoIt's more efficient with tools for one and the input cost is cheaper (which is where a lot of the cost is). See comparison between GPT-5, 4.1, and o3 tool calling here: https://promptslice.com/share/b-2ap_rfjeJgIQsG https://promptslice.com/share/b-2ap_rfjeJgIQsG.
- deleted 1y ago[deleted]
- deleted 1y ago[deleted]
- pram 1y agoWe’re at the audiophile stage of LLMs where people are talking about the improved soundstage, tonality, reduced sibilance etc
- javchz 1y agoI can already see LLMs Sommeliers: Yes, the mouthfeel and punch of GPT-5 it's comparable to the one of Grok 4, but it's tenderness lacks the crunch from Gemini 2.5 Pro.
- 0x7cfe 1y agoIsn't it exactly what the typical LLM discourse is about? People are just throwing anecdotes and stay with their opinion. A is better than B because C, and that's basically it. And whoever tries to actually bench them gets called out because all benches are gamed. Go figure.
- jaredcwhite 1y agoNote GPT-5's subtle mouthfeel reminiscent of cranberries with a touch of bourbon.
- __loam 1y agoEvery bourbon tastes the same unless it's Weller, King's County Peated, or Pappy (or Jim Beam for the wrong reasons lol)
- alephnerd 1y agoTbh, a mid-shelf Four Roses gets you 90% of the way to a upper shelf Weller.
- __loam 1y agoI'm being hyperbolic but yeah four roses is probably the best deal next to Buffalo trace. All their stuff is fairly priced. If you want something like Weller though, you should get another wheated bourbon like Maker's Mark French oaked.
- scosman 1y agoThere's a bunch of benchmarks on the intro page including AIME 2025 without tools, SWE-bench Verified, Aider Polyglot, MMMU, and HealthBench Hard (not familiar with this one): https://openai.com/index/introducing-gpt-5/ https://openai.com/index/introducing-gpt-5/ Pretty par for course evals at launch setup.
- WD-42 1y agoIt has the last ~6 months worth of flavor of the month Javascript libraries in it's training set now, so it's "better at coding". How is this sustainable.
- some-guy 1y agoAs someone who tries to push the limits of hard coding tasks (mainly refactoring old codebases) to LLMs with not much improvement since the last round of models, I'm finding that we are hitting the reduction of rate of improvement on the S-curve of quality. Obviously getting the same quality cheaper would be huge, but the quality of the output day to day isn't noticeable to me.
- camdenreslink 1y agoI find it struggles to even refactor codebases that aren't that large. If you have a somewhat complicated change that spans the full stack, and has some sort of wrinkle that makes it slightly more complicated than adding a data field, then even the most modern LLMs seem to trip on themselves. Even when I tell it to create a plan for implementation and write it to a markdown file and then step through those steps in a separate prompt. Not that it makes it useless, just that we seem to not "be there" yet for the standard tasks software engineers do every day.
- teaearlgraycold 1y agoI haven’t used GPT5 yet, but even on a 1000 line code base I found Opus 4, o3, etc. to be very hit or miss. The trouble is I can’t seem to predict when these models will hit. So the misses cost time, reducing their overall utility.
- rapind 1y agoI'm exclusively using sonnet via claude-code on their max plan (opting to specify sonnet so that opus isn't used). I just wasn't pleased with the opus output, but maybe I just need to use it differently. I haven't bothered with 4.1 yet. Another thing I noticed is opus would eat up my caps super quick, whereas using sonnet exclusively I never hit a cap. I'd really just love incremental improvements over sonnet. Increasing the context window on sonnet would be a game changer for me. After auto-compact the quality may fall off a cliff and I need to spend some time bringing it back up to speed. When I need a bit more punch for more reasoning / architecture type evaluations, I have it talk to gemini pro via zen mcp and OpenRouter. I've been considering setting up a subagent for architecture / system design decisions that would use the latest opus to see if it's better than gemini pro (so far I have no complaints though).
- nicetryguy 1y agoYeah. We're entered the Smartphone stage: "You want the new one because it's the new one."
- sbinnee 1y agoWhen they were about to release gpt4 I remember the hype was so high there were a lot of AGI debates. But then was quickly out-shadowed by more advanced models. People knew that gpt5 wouldn’t be an AGI or even close to that. It’s just an updated version. GptN would become more or leas like an annual release.
- blablablerg 1y agoFor fun, I asked it how much better it is than GPT-4. It started a rap battle against itself :P https://chatgpt.com/share/6895d5da-8884-8003-bf9d-1e191b11d3d1 https://chatgpt.com/share/6895d5da-8884-8003-bf9d-1e191b11d3...
- jennyholzer 1y agoI didn't think GPT-4 warranted a major version bump. I do not believe that Open AI's benchmarks are legitimate and I don't think they have been for quite some time, if ever.
- ttroyr 1y agoI think the biggest tell for me was having the leader of Cursor up vouching for the model, who has been a big proponent of Claude in Cursor for the last year. Doesn't seem like a light statement.