13 ms·
GPT-5: Key characteristics, pricing and system card
System card: https://cdn.openai.com/pdf/8124a3ce-ab78-4f06-96eb-49ea29ffb52f/gpt5-system-card-aug7.pdf https://cdn.openai.com/pdf/8124a3ce-ab78-4f06-96eb-49ea29ffb...
- ks2048 1y agoSo, "system card" now means what used to be a "paper", but without lots of the details?
- kaoD 1y agoNope. System card is a sales thing. I think we generally call that "product sheet" in other markets.
- simonw 1y agoAI labs tend to use "system cards" to describe their evaluation and safety research processes. They used to be more about the training process itself, but that's increasingly secretive these days.
- matusp 1y agoDespite the fact that their models are used in hiring, business, education, etc this multibillion company uses one benchmark with very artificial questions (BBQ) to evaluate how fair their model is. I am a little bit disappointed.
- xmorse 1y agoIt's because these industries don't create their own benchmarks. The only ones creating evals are the AI company themselves or open source software engineers
- Leary 1y agoMETR of only 2 hours and 15 minutes. Fast takeoff less likely.
- qsort 1y agoIsn't that pretty much in line with what people were expecting? Is it surprising?
- dingnuts 1y agoIt's not surprising to AI critics but go back to 2022 and open r/singularity and then answer: what "people" were expecting? Which people? SamA has been promising AGI next year for three years like Musk has been promising FSD next year for the last ten years. IDK what "people" are expecting but with the amount of hype I'd have to guess they were expecting more than we've gotten so far. The fact that "fast takeoff" is a term I recognize indicates that some people believed OpenAI when they said this technology (transformers) would lead to sci fi style AI and that is most certainly not happening
- falcor84 1y agoI would say that there are quite a lot of roles where you need to do a lot of planning to effectively manage an ~8 hour shift, but then there are good protocols for handing over to the next person. So once AIs get to that level (in 2027?), we'll be much closer to AIs taking on "economically valuable work".
- ToValueFunfetti 1y ago>SamA has been promising AGI next year for three years like Musk has been promising FSD next year for the last ten years. Has he said anything about it since last September: >It is possible that we will have superintelligence in a few thousand days (!); it may take longer, but I’m confident we’ll get there. This is, at an absolute minimum, 2000 days = 5 years. And he says it may take longer. Did he even say AGI next year any time before this? It looks like his predictions were all pointing at the late 2020s, and now he's thinking early 2030s. Which you could still make fun of, but it just doesn't match up with your characterization at all.
- usaar333 1y agoNo, this is below expectations on both Manifold and lesswrong (https://www.lesswrong.com/posts/FG54euEAesRkSZuJN/ryan_greenblatt-s-shortform?commentId=6ue8BPWrcoa2eGJdP https://www.lesswrong.com/posts/FG54euEAesRkSZuJN/ryan_green...). Median was ~2.75 hours on both (which already represented a bearish slowdown). Not massively off -- manifold yesterday implied odds this low were ~35%. 30% before Claude Opus 4.1 came out which updated expected agentic coding abilities downward.
- nickthegreek 1y agoThis new naming conventions, while not perfect are alot clearer and I am sure will help my coworkers.
- anyg 1y agoGood to know - > Knowledge cut-off is September 30th 2024 for GPT-5 and May 30th 2024 for GPT-5 mini and nano
- falcor84 1y agoOh wow, so essentially a full year of post-training and testing. Or was it ready and there was a sufficiently good business strategy decision to postpone the release?
- thorum 1y agoThe Information’s report from earlier this month claimed that GPT-5 was only developed in the last 1-2 months, after some sort of breakthrough in training methodology. > As recently as June, the technical problems meant none of OpenAI’s models under development seemed good enough to be labeled GPT-5, according to a person who has worked on it. But it could be that this refers to post-training and the base model was developed earlier. https://www.theinformation.com/articles/inside-openais-rocky-path-gpt-5 https://www.theinformation.com/articles/inside-openais-rocky... https://archive.ph/d72B4 https://archive.ph/d72B4
- simonw 1y agoMy understanding is that training data cut-offs and dates at which the model were trained are independent things. AI labs gather training data and then do a ton of work to process it, filter it etc. Model training teams run different parameters and techniques against that processed training data. It wouldn't surprise me to hear that OpenAI had collected data up to September 2024, dumped that data in a data warehouse of some sort, then spent months experimenting with ways to filter and process it and different training parameters to run against it.
- ncallaway 1y agoIs the filtering and processing very specific to the data set? I'd kind of assume that they would dump data into the data warehouse on September 2024, then in parallel continue data collection and do the months of work to determine how to best filter, process it, and select training parameters, etc. Then once that was locked in to do a final update to the say, December 2024 data warehouse for the final training. Do the filtering, processing, and training parameters need to be fairly fine-tuned to the specific data set?
- cco 1y agoOnly a third cheaper than Sonnet 4? Incrementally better I suppose. > and minimizing sycophancy Now we're talking about a good feature! Actually one of my biggest annoyances with Cursor (that mostly uses Sonnet). "You're absolutely right!" I mean not really Cursor, but ok. I'll be super excited if we can get rid of these sycophancy tokens.
- logicchains 1y ago>Only a third cheaper than Sonnet 4? The price should be compared to Opus, not Sonnet.
- cco 1y agoWow, if so, 7x cheaper. Crazy if true.
- nosefurhairdo 1y agoIn my early testing gpt5 is significantly less annoying in this regard. Gives a strong vibe of just doing what it's told without any fluff.
- bdcdo 1y ago"GPT-5 in the API is simpler: it’s available as three models—regular, mini and nano—which can each be run at one of four reasoning levels: minimal (a new level not previously available for other OpenAI reasoning models), low, medium or high." Is it actually simpler? For those who are currently using GPT 4.1, we're going from 3 options (4.1, 4.1 mini and 4.1 nano) to at least 8, if we don't consider gpt 5 regular - we now will have to choose between gpt 5 mini minimal, gpt 5 mini low, gpt 5 mini medium, gpt 5 mini high, gpt 5 nano minimal, gpt 5 nano low, gpt 5 nano medium and gpt 5 nano high. And, while choosing between all these options, we'll always have to wonder: should I try adjusting the prompt that I'm using, or simply change the gpt 5 version or its reasoning level?
- impossiblefork 1y agoYes, I think so. It's n=1,2,3 m=0,1,2,3. There's structure and you know that each parameter goes up and in which direction.
- makeramen 1y agoBut given the option, do you choose bigger models or more reasoning? Or medium of both?
- namibj 1y agoDepends on what you're doing.
- addaon 1y ago> Depends on what you're doing. Trying to get an accurate answer (best correlated with objective truth) on a topic I don't already know the answer to (or why would I ask?). This is, to me, the challenge with the "it depends, tune it" answers that always come up in how to use these tools -- it requires the tools to not be useful for you (because there's already a solution) to be able to do the tuning.
- diggan 1y ago> but for the moment here’s the pelican I got from GPT-5 running at its default “medium” reasoning effort: Would been interesting to see a comparison between low, medium and high reasoning_effort pelicans :) When I've played around with GPT-OSS-120b recently, seems the difference in the final answer is huge, where "low" is essentially "no reasoning" and with "high" it can spend seemingly endless amount of tokens. I'm guessing the difference with GPT-5 will be similar?
- simonw 1y ago> Would been interesting to see a comparison between low, medium and high reasoning_effort pelicans Yeah, I'm working on that - expect dozens of more pelicans in a later post.
- meatmanek 1y agoWould also be interesting to see how well they can do with a loop of: write SVG, render SVG, feed SVG back to LLM for review, iterate. Sorta like how a human would actually compose an SVG of a pelican.
- zaronymous1 1y agoCan anyone explain to me why they've removed parameter controls for temperature and top-p in reasoning models, including gpt-5? It strikes me that it makes it harder to build with these to do small tasks requiring high-levels of consistency, and in the API, I really value the ability to set certain tasks to a low temp.
- Der_Einzige 1y agoIt's because all forms of sampler settings destroy safety/alignment. That's why top_p/top_k are still used and not tfs, min_p, top_n sigma, etc, why temperature is locked to 0-2 arbitrary range, etc Open source is years ahead of these guys on samplers. It's why their models being so good is that much more impressive.
- oblio 1y agoTemperature is the response variation control?
- AH4oFVbPT4f8 1y agoYes, it controls variability or probability of the next token or text to be selected.
- hodgehog11 1y agoThe aggressive pricing here seems unusual for OpenAI. If they had a large moat, they wouldn't need to do this. Competition is fierce indeed.
- 0x00cl 1y agoMaybe the need/want data.
- dr_dshiv 1y agoAnd it’s a massive distillation of the mother model, so the costs of inference are likely low.
- impure 1y agoOpenAI and most AI companies do not train on data submitted to a paid API.
- WhereIsTheTruth 1y agoThey also do not train using copyrighted material /s
- simonw 1y agoThat's different. They train on scrapes of the web. They don't train on data submitted to their API by their paying customers.
- johnnyanmac 1y agoIf they're bold enough to say they train on data they do not own, I am not optimistic when they say they don't train on data people willingly submit to them.
- simonw 1y agoI don't understand your logic there. They have confessed to doing a bad thing - training on copyrighted data without permission. Why does that indicate they would lie about a worse thing?
- onehair 1y ago> Definitely recognizable as a pelican right :-D
- pancakemouse 1y agoPractically the first thing I do after a new model release is try to upgrade `llm`. Thank you, @simonw !
- efavdb 1y agosame, looks like he hasn't added 5.0 to the package yet but assume imminent. https://llm.datasette.io/en/stable/openai-models.html https://llm.datasette.io/en/stable/openai-models.html
- simonw 1y agoWorking on that now! https://github.com/simonw/llm/issues/1229 https://github.com/simonw/llm/issues/1229
- pancakemouse 1y ago> I want to build a tool that can render SVG to an image, then return that image so the model can see what it has drawn. For reasons. :D
- isoprophlex 1y agoWhoa this looks good. And cheap! How do you hack a proxy together so you can run Claude Code on gpt-5?!
- dalberto 1y agoConsider: https://github.com/musistudio/claude-code-router https://github.com/musistudio/claude-code-router or even: https://github.com/sst/opencode https://github.com/sst/opencode Not affiliated with either one of these, but they look promising.
- morleytj 1y agoIt's cool and I'm glad it sounds like it's getting more reliable, but given the types of things people have been saying GPT-5 would be for the last two years you'd expect GPT-5 to be a world-shattering release rather than incremental and stable improvement. It does sort of give me the vibe that the pure scaling maximalism really is dying off though. If the approach is on writing better routers, tooling, comboing specialized submodels on tasks, then it feels like there's a search for new ways to improve performance(and lower cost), suggesting the other established approaches weren't working. I could totally be wrong, but I feel like if just throwing more compute at the problem was working OpenAI probably wouldn't be spending much time on optimizing the user routing on currently existing strategies to get marginal improvements on average user interactions. I've been pretty negative on the thesis of only needing more data/compute to achieve AGI with current techniques though, so perhaps I'm overly biased against it. If there's one thing that bothers me in general about the situation though, it's that it feels like we really have no clue what the actual status of these models is because of how closed off all the industry labs have become + the feeling of not being able to expect anything other than marketing language from the presentations. I suppose that's inevitable with the massive investments though. Maybe they've got some massive earthshattering model release coming out next, who knows.
- jstummbillig 1y agoThings have moved differently than what we thought would happen 2 years ago, but lest we forget what has happened in the meanwhile (4o, o1 + thinking paradigm, o3) So yeah, maybe we are getting more incremental improvements. But that to me seems like a good thing, because more good things earlier. I will take that over world-shattering any day – but if we were to consider everything that has happened since the first release of gpt-4, I would argue the total amount is actually very much world-shattering.
- GaggiX 1y agoCompared to GPT-4, it is on a completely different level given that it is a reasoning model so on that regard it does delivers and it's not just scaling, but for this I guess the revolution was o1 and GPT-5 is just a much more mature version of the technology.
- 1y ago
- ilaksh 1y agoThis is key info from the article for me: > ------------------------------- "reasoning": {"summary": "auto"} }' Here’s the response from that API call. https://gist.github.com/simonw/1d1013ba059af76461153722005a039d https://gist.github.com/simonw/1d1013ba059af76461153722005a0... Without that option the API will often provide a lengthy delay while the model burns through thinking tokens until you start getting back visible tokens for the final response.
- justusthane 1y ago> a real-time router that quickly decides which model to use based on conversation type, complexity, tool needs, and explicit intent This is sort of interesting to me. It strikes me that so far we've had more or less direct access to the underlying model (apart from the system prompt and guardrails), but I wonder if going forward there's going to be more and more infrastructure between us and the model.
- hirako2000 1y agoConsider it a low level routing. Keeping in mind it allows the other non active parts to not be in memory. Mistral afaik came up with this concept, quite a while back.
- ItsHarper 1y agoIt's actually just a high-level routing between the reasoning and non-reasoning models that only applies to ChatGPT.
- ItsHarper 1y agoThat only applies to ChatGPT. The API has direct access to specific models.
- willy_k 1y agoThe router seems to only apply to the ChatGPT version, not the API, so it’s not really anything new. Gemini already has functionally dynamic reasoning effort.
- techpression 1y ago"They claim impressive reductions in hallucinations. In my own usage I’ve not spotted a single hallucination yet, but that’s been true for me for Claude 4 and o3 recently as well—hallucination is so much less of a problem with this year’s models." This has me so confused, Claude 4 (Sonnet and Opus) hallucinates daily for me, on both simple and hard things. And this is for small isolated questions at that.
- simonw 1y agoWhat kind of hallucinations are you seeing?
- OtherShrezzing 1y agoI rewrote a 4 page document from first to third person a couple of weeks back. I gave Claude Sonnet 4 the document after editing, so it was entirely written in the third person. I asked it to review & highlight places where it was still in the first person. >Looking through the document, I can identify several instances where it's written in the first person: And it went on to show a series of "they/them" statements. I asked it to clarify if "they" is "first person" and it responded >No, "they" is not first person - it's third person. I made an error in my analysis. First person would be: I, we, me, us, our, my. Second person would be: you, your. Third person would be: he, she, it, they, them, their. Looking back at the document more carefully, it appears to be written entirely in third person. Even the good models are still failing at real-world use cases which should be right in their wheelhouse.
- simonw 1y agoThat doesn't quite fit the definition I use for "hallucination" - it's clearly a dumb error, but the model didn't confidently state something that's not true (like naming the wrong team who won the Super Bowl).
- OtherShrezzing 1y ago>"They claim impressive reductions in hallucinations. In my own usage I’ve not spotted a single hallucination yet, but that’s been true for me for Claude 4 and o3 recently as well—hallucination is so much less of a problem with this year’s models." Could you give an estimate of how many "dumb errors" you've encountered, as opposed to hallucinations? I think many of your readers might read "hallucination" and assume you mean "hallucinations and dumb errors".
- drumhead 1y ago"Are you GPT5" - No I'm 4o, 5 hasnt been released yet. "It was released today". Oh you're right, Im GPT5. You have reached the limit of the free usage of 4o
- nonhaver 1y agohaha brutal. maybe tomorrow
- cchance 1y agoIts basically opus 4.1 ... but cheaper?
- gwd 1y agoCheaper is an understatement... it's less than 1/10 for input and nearly 1/8 for output. Part of me wonders if they're using their massive new investment to sell API below-cost and drive out the competitor. If they're really getting Opus 4.1 performance for half of Sonnet compute cost, they've done really well.
- diggan 1y agoI'm not sure I'd be surprised, I've been playing around with GPT-OSS last few days, and the architecture seems really fast for the accuracy/quality of responses, way better than most local weights I've tried for the last two years or so. And since they released that architecture publicly, I'd imagine they're sitting on something even better privately.
- bravesoul2 1y agoWith the unlimited demand I can't see that strategy working. It is not like taxis where you may do a trip or two a day but if it cheap enough you'd do 100 a day. But with AI you would totally 100x.
- aliljet 1y agoI'm curious what platform people are using to test GPT-5? I'm so deep into the claude code world that I'm actually unsure what the best option is outside of claude code...
- cainxinth 1y agoIt’s fascinating and hilarious that pelican on a bicycle in SVG is still such a challenge.
- muglug 1y agoHow easy is it for you to create an SVG of a pelican riding a bicycle in a text editor by hand?
- jopsen 1y agoWithout looking at the rendered output :)
- freediver 1y agoAnd without ever seeing a pelican on a bicycle :)
- reassess_blind 1y agoIt should Google it like the rest of us.
- cainxinth 1y agoI didn't mean to imply it was simple, just that it's funny because I can't really evaluate evals like Humanity's Last Exam, but I can see the progress of these models in a pelican.
- SomewhatLikely 1y agoNobody's preventing them from rendering it and refining. That's certainly what we'd expect an AGI to do.
- throwaway422432 1y agoI'm surprised they haven't all tried to game this test by now, or at least added it to their internal testing knowing they will be judged by it.
- joshmlewis 1y agoIt seems to be trained to use tools effectively to gather context. In this example against 4.1 and o3 it used 6 in the first turn in a pretty cool way (fetching different categories that could be relevant). Token use increases with that kind of tool calling but the aggressive pricing should make that moot. You could probably get it to not be so tool happy with prompting as well. https://promptslice.com/share/b-2ap_rfjeJgIQsG https://promptslice.com/share/b-2ap_rfjeJgIQsG
- tomrod 1y agoSimon, as always, I appreciate your succinct and dedicated writeup. This really helps to land the results.
- moralestapia 1y agoBasically repeats what it's been out through the usual PR channels, just paraphrased. No mention about the (missing) elephant on the room, where are the benchmarks? @simonw has been compromised. Sad.
- simonw 1y agoI'm sorry I didn't say "independent benchmarks are not yet available" in my post, I say that so often on model launches I guess I took it as read this time.
- kevink23 1y agoI was excited for GPT-5, but honestly, it feels worse than GPT-4 for coding.
- simonw 1y agoGPT-4 or GPT-4o?
- globular-toast 1y agoThis "system card" thing seems to have suddenly come out of nowhere. Signs of a cult forming. Is it just what we'd normally call a technical write up?
- dragonwriter 1y agoIt’s a variation on “model card”, which has become a standard thing with AI models, but with the name changed because the wroteup covers toolchain as well as model information. But a PDF of the size of the document at issue is very much not the kind of concise document model cards are, its more the kind of technical report that a much more concise card would reference.
- coldtea 1y agoThe improved bicycling pelican of course could be overfitting / benchmark-cheating...
- ozgung 1y agoOne key element missing from all these model cards are the model size/number of parameters. Without that info we are in the dark. We can't predict the future of AI. How does the intelligence scale with the increasing #parameters? Is there a limit? Should we attribute incrementally better metrics to larger model size or other techniques? Do they announce the full model they trained or a smaller version that is economically viable for the market conditions? If they double the model size will it be a Professor-level intelligence, a super-human level intelligence or a couple of phds level intelligence?
- diamond559 1y agoWe're supposed to get AGI next year and this is all they have. We are not getting real AI from LLMs people, wake up, we're in a bubble.
- noisy_boy 1y agoI wonder if there are people who find Sonnet 3.5 to be faster and good enough for programming instead of moving on to reasoning models (sometimes I do go to Gemini 2.5 pro if Sonnet doesn't cut it).
- azyc 1y ago[dead]