11 ms·

GPT-5 for Developers

470 points by 6thbit 1y ago

t1amat 1y ago
The problem with OpenAI models is the lack of a Max-like subscription for a good agentic harness. Maybe OpenAI or Microsoft could fix this. I just went through the agony of provisioning my team with new Claude Code 5x subs 2 weeks ago after reviewing all of the options available at that time. Since then, the major changes include a Cerebras sub for Qwen3 Coder 480B, and now GPT-5. I’m still not sure I made the right choice, but hey, I’m not married to it either. If you plan on using this much at all then the primary thing to avoid is API-based pay per use. It’s prohibitively costly to use regularly. And even for less important changes it never feels appropriate to use a lower quality model when the product counts. Claude Code won primarily because of the sub and that they have a top tier agentic harness and models that know how to use it. Opus and Sonnet are fantastic agents and very good at our use case, and were our preferred API-based models anyways. We can use Claude Code basically all day with at least Sonnet after using our Opus limits up. Worth nothing that Cline built a Claude Code provider that the derivatives aped which is great but I’ve found Claude Code to be as good or better anyways. The CLI interface is actually a bonus for ease of sharing state via copy/paste. I’ll probably change over to Gemini Code Assist next, as it’s half the price and more context length, but I’m waiting for a better Gemini 2.5 Pro and the gemini-cli/Code Assist extensions to have first party planning support, which you can get some form of third party through custom extensions with the cli, but as an agent harness they are incomplete without. The Cerebras + Qwen3 Coder 480B with qwen3-cli is seriously tempting. Crazy generation speed. Theres some question about how long big the rate limit really is but it’s half the cost of Claude Code 5x. I haven’t checked but I know qwen3-cli, which was introduced along side the model, is a fork of gemini-cli with Qwen-focused updates; wonder if they landed a planning tool? I don’t really consider Cursor, Windsurf, Cline, Roo, Kilo et al as they can’t provide a flat rate service with the kind of rate limits you can get with the aforementioned. GitHub Copilot could be a great offering if they were willing to really compete with a good unlimited premium plan but so far their best offering has less premium requests than I make in a week, possibly even in a few days. Would love to hear if I missed anything, or somehow missed some dynamic here worth considering. But as far as I can tell, given heavy use, you only have 3 options today: Claude Max, Gemini Code Assist, Cerebras Code.
- energy123 1y ago
  > If you plan on using this much at all then the primary thing to avoid is API-based pay per use. I find there's a niche where API pay-per-use is cost effective. It's for problems that require (i) small context and (ii) not much reasoning. Coding problems with 100k-200k context violates (i). Math problems violate (ii) because they generate long reasoning streams. Coding problems with 10k-20k context are well suited, because they generate only ~5k output tokens. That's $0.03-$0.04 per prompt to GPT-5 under flex pricing. The convenience is worth it, unless you're relying on a particular agentic harness that you don't control (I am not). For large context questions, I send them to a chat subscription, which gives me a budget of N prompts instead of N tokens. So naturally, all the 100k-400k token questions go there.
- NullifyNAN 1y ago
  OpenAI has answered your prayers. 16 hours ago the readme for codex CLI was updated. Now codex cli supports openai login like claude does, no API credits. From the readme: After you run codex select Sign in with ChatGPT. You'll need a Plus, Pro, or Team ChatGPT account, and will get access to our latest models, including gpt-5, at no extra cost to your plan. (Enterprise is coming soon.) Important: If you've used the Codex CLI before, you'll need to follow these steps to migrate from usage-based billing with your API key: Update the CLI with codex update and ensure codex --version is greater than 0.13 Ensure that there is no OPENAI_API_KEY environment variable set. (Check that env | grep 'OPENAI_API_KEY' returns empty) Run codex login again
  - IanCal 1y ago
    Oh that’s fantastic news, thanks!
catigula 1y ago
I thought we were going to have AGI by now.
- RS-232 1y ago
  No shot. LLMs are simple text predictors and they are too stupid to get us to real AGI. To achieve AGI, we will need to be capable of high fidelity whole brain simulations that model the brain's entire physical, chemical, and biological behavior. We won't have that kind of computational power until quantum computers are mature.
  - evantbyrne 1y ago
    It will be interesting to see if humans can manage to bioengineer human-level general intelligence into another species before computers.
  - machiaweliczny 1y ago
    [flagged]
    - bopbopbop7 1y ago
      “some twist” is doing a lot of heavy lifting in that statement.
      - AppleBananaPie 1y ago
        CS will define, design and implement human level intelligence before neuroscience has done even the first. That's what I hear when people say stuff like this anyway. Similar to CS folks throwing around physics 'theories'
  - nawgz 1y ago
    I don't really see any relationship between being able to model/simulate the brain and being able to exceed the brain in intelligence, can you explain more about that? Simulations sound like more of a computational and analytic problem with regards to having an accurate model. Maybe your point is that until we understand our own intelligence, which would be reflected in such a simulation, it would be difficult to improve upon it.
andrewmcwatters 1y ago
I wonder how good it is compared to Claude Sonnet 4, and when it's coming to GitHub Copilot. I almost exclusively wrote and released https://github.com/andrewmcwattersandco/git-fetch-file https://github.com/andrewmcwattersandco/git-fetch-file yesterday with GPT 4o and Claude Sonnet 4, and the latter's agentic behavior was quite nice. I barely had to guide it, and was able to quickly verify its output.
- fleebee 1y ago
  There is an option in GitHub Copilot settings to enable GPT-5 already.
croemer 1y ago
> GPT‑5 also excels at long-running agentic tasks—achieving SOTA results on τ2-bench telecom (96.7%), a tool-calling benchmark released just 2 months ago. Yes, but it does worse than o3 on the airline version of that benchmark. The prose is totally cherry picker.
- Fogest 1y ago
  How does the cost compare though? From my understanding o3 is pretty expensive to run. Is GPT-5 less costly? If so if the performance is close to o3 but cheaper, then it may still be a good improvement.
  - low_tech_punk 1y ago
    I find it strange that GPT-5 is cheaper than GPT-4.1 in input token and is only slightly more expensive in output token. Is it marketing or actually reflecting the underlying compute resources?
    - AS04 1y ago
      Very likely to be an actual reflection. That's probably their real achievement here and the key reason why they are actually publishing it as GPT-5. More or less the best or near to it on everything while being one model, substantially cheaper than the competition.
      - ComputerGuru 1y ago
        But it can’t do audio in/out or image out. Feels like an architectural step back.
        conradkay 1y ago
        My understanding is that image output is pretty separate and if it doesn’t seem that way, they’re just abstracting several models into one name
    - 1y ago
aliljet 1y ago
Between Opus aand GPT-5, it's not clear there's a substantial difference in software development expertise. The metric that I can't seem to get past in my attempts to use the systems is context awareness over long-running tasks. Producing a very complex, context-exceeding objective is a daily (maybe hourly) ocurrence for me. All I care about is how these systems manage context and stay on track over extended periods of time. What eval is tracking that? It seems like it's potentially the most imporatnt metric for real-world software engineering and not one-shot vibe prayers.
- realusername 1y ago
  Personally I think I'll wait for another 10x improvement for coding because with the current way it's going, they clearly need that.
  - fsloth 1y ago
    From my experience when used through IDE such as Cursor the current gen Claude model enables impressive speedruns over commodity tasks. My context is a CAD application I’ve been writing as a hobby. I used to work in that field for a decade so have a pretty good touch on how long I would expect tasks to take. I’m using mostly a similar software stack as that at previous job and am definetly getting stuff done much faster on holiday at home than at that previous work. Of course the codebase is also a lot smaller, intrinsic motivation, etc, but still.
    - BoredPositron 1y ago
      How often do you have to build the simple scaffolding though?
      - fsloth 1y ago
        At a real job? Not that often! And it's miserable in large scale architecture. However, at leas for me there is lots of "small enough context" boilerplate that the context can deal with. Clearly this is not a tool in the sense it's predictable.
coconut08 1y ago
over the last week or so I have put probably close to 70 hours into playing around with cursor and claude code and a few other tools (its become my new obsession). I've been blown away by how good and reliable it is now. That said the reality is in my experience the only models that actually work in any sort of reliable way are claude models. I dont care what any benchmark says because the only thing that actually matters is actual use. I'm really hoping that this new gpt model actually works for this usecase because competition is great and the price is also great.
- deleted 1y ago
  [deleted]
- ralfd 1y ago
  Just replying to ask you next week what your assessment on GPT5 is.
  - coconut08 1y ago
    I've been trying it out with openai codex over the last day and a half and I have been incredibly impressed. It has been working quite well. I also had it look over some code that claude produced for me and it said that it would be better to approach it another way and it completely rewrote it in a way that actually was significantly better. The UX for codex is quite a bit worse than Claude Code, but the model has been good enough to justify the switch for now. I'm hopeful that cursor cli will eventually have a good enough ux such that I can switch to it and have access to all of the models rather than needing to use disparate tools for everything. I would strongly suggest you check out gpt 5 for agentic stuff if you are interested.
- throwaway_2898 1y ago
  How much of the product were you able to build to say it was good/reliable? IME, 70 hours can get you to a PoC that "works", building beyond the initial set of features — like say a first draft of all the APIs — does it do well once you start layering features?
  - 1y ago
timhigins 1y ago
I opened up the developer playground and the model selection dropdown showed GPT-5 and then it disappeared. Also I don't see it in ChatGPT Pro. What's up?
- IAmGraydon 1y ago
  Not showing in my Pro account either. As someone else mentioned, I’m sure it’s throttling due to high use right now.
- Fogest 1y ago
  It's probably being throttled due to high usage.
- brookst 1y ago
  Shipping something at the moment of announcement is always hell.
sebdufbeau 1y ago
Has the API rollout started? It's not available in our org, even if we've been verified for a few months EDIT: It's out now
- spullara 1y ago
  it is out yet. i poll the api for the models and update this GitHub hourly. https://github.com/spullara/models https://github.com/spullara/models
low_tech_punk 1y ago
The ability to specify a context-free grammar as output constraint? This blows my mind. How do you control the auto regressive sampling to guarantee the correct syntax?
- qsort 1y ago
  You sample only from tokens that could possibly result in a valid production for the grammar. It's an inference-only thing.
  - low_tech_punk 1y ago
    ah, thanks!
- evnc 1y ago
  I assume they're doing "Structured Generation" or "Guided generation", which has been possible for a while if you control the LLM itself e.g. running an OSS model, e.g. [0][1]. It's cool to see a major API provider offer it, though. The basic idea is: at each auto-regressive step (each token generation), instead of letting the model generate a probability distribution over "all tokens in the entire vocab it's ever seen" (the default), only allow the model to generate a probability distribution over "this specific set of tokens I provide". And that set can change from one sampling set to the next, according to a given grammar. E.g. if you're using a JSON grammar, and you've just generated a `{`, you can provide the model a choice of only which tokens are valid JSON immediately after a `{`, etc. [0] https://github.com/dottxt-ai/outlines https://github.com/dottxt-ai/outlines [1] https://github.com/guidance-ai/guidance https://github.com/guidance-ai/guidance
low_tech_punk 1y ago
Tried using gpt-5 family with response API and got error "gpt-5 does not exist or you don't have access to it". I guess they are not rolling out in lock step with the live stream and blog article?
- low_tech_punk 1y ago
  Can confirm that they are rolling out. It's working for me.
- diggan 1y ago
  Seems they're doing rollout over time, I'm not seeing it anywhere yet.
skepticATX 1y ago
This was really a bad release for OpenAI, if benchmarks are even somewhat indicative of how the model will perform in practice.
- robterrell 1y ago
  In what ways?
- mediaman 1y ago
  I actually don't agree. Tool use is the key to successful enterprise product integration and they have done some very good work here. This is much more important to commercialization than, for example, creative writing quality (which it reportedly is not good at).
jumploops 1y ago
If the model is as good as the benchmarks say, the pricing is fantastic: Input: $1.25 / 1M tokens (cached: $0.125/1Mtok) Output: $10 / 1M tokens For context, Claude Opus 4.1 is $15 / 1M for input tokens and $75/1M for output tokens. The big question remains: how well does it handle tools? (i.e. compared to Claude Code) Initial demos look good, but it performs worse than o3 on Tau2-bench airline, so the jury is still out.
- addaon 1y ago
  > Output: $10 / 1M tokens It's interesting that they're using flat token pricing for a "model" that is explicitly made of (at least) two underlying models, one with much lower compute costs than the other; and with use ability to at least influence (via prompt) if not choose which model is being used. I have to assume this pricing model is based on a predicted split between how often the underlying models get used; I wonder if that will hold up, if users will instead try to rouse the better model into action more than expected, or if the pricing is so padded that it doesn't matter.
  - simianwords 1y ago
    > that is explicitly made of (at least) two underlying models what do you mean?
    - addaon 1y ago
      > a smart and fast model that answers most questions, a deeper reasoning model for harder problems, and a real-time router that quickly decides which model to use based on conversation type, complexity, tool needs, and explicit intent (for example, if you say “think hard about this” in the prompt). From https://openai.com/index/gpt-5-system-card/ https://openai.com/index/gpt-5-system-card/
      - tedsanders 1y ago
        In the API, there’s no router. Developers just pick whether they use the reasoning model or non-thinking ChatGPT model.
6thbit 1y ago
Seems they have quietly increased the context window up to 400,000 https://platform.openai.com/docs/models/gpt-5 https://platform.openai.com/docs/models/gpt-5
- ralfd 1y ago
  How does that compare to Claude/GPT4?
  - 6thbit 1y ago
    4o - 128k o3 - 200k Opus 4.1 - 200k Sonnet 4 - 200k So, at least twice larger context than those
  - hrpnk 1y ago
    gpt4.1 has 1M input and 32k output, Sonnet 4 200k/64k
- simianwords 1y ago
  but is it for the model in chatgpt.com as well?
- Iwan-Zotow 1y ago
  Input plus output?
mehmetoguzderin 1y ago
Context-free grammar and regex support are exciting. I wonder what, or whether, there are differences from the Lark-like CFG of llguidance, which powers the JSON schema of the OpenAI API [^1]. [^1]: https://github.com/guidance-ai/llguidance/blob/f4592cc0c783a987fa44791fc1d702208b366613/README.md https://github.com/guidance-ai/llguidance/blob/f4592cc0c783a...
- msp26 1y ago
  Yeah that was the only exciting part of the announcement for me haha. Can't wait to play around with it. I'm already running into a bunch of issues with the structured output APIs from other companies like Google and OpenAI have been doing a great job on this front.
  - chrisweekly 1y ago
    > "I'm already running into a bunch of issues with the structured output APIs from other companies like Google and OpenAI have been doing a great job on this front." This run-on sentence swerved at the end; I really can't tell what your point is. Could you reword it for clarity?
    - petercooper 1y ago
      I read it as "... from other companies, like Google, and OpenAI have been doing a great job on this front"
      - mehmetoguzderin 1y ago
        I'm not sure if it's due to experience with the aforementioned APIs, but I also read the same, “issues with APIs like ..., and (in contrast) OpenAI have been doing a great job”
belter 1y ago
We were promised AGI and all we got was code generators...
- bmau5 1y ago
  It's a logical starting point, given there are pretty defined success/failure criteria
  - ehutch79 1y ago
    The hype is real. We were told that we'd have AGI and be out of jobs 2 years ago, let alone today.
    - brookst 1y ago
      We were also told that AGI would never happen, that it was 6 months away, that it is 20 years away. I’m not sure of the utility of being so outraged that some people made wrong predictions.
    - rowanG077 1y ago
      By whom? I don't think anyone seriously said in 2023 we have AGI in two years. Even now, no one reputable is claiming AGI in two years.
      - ehutch79 1y ago
        Random on YouTube, random here on hacker news. No, I don’t take them seriously, that was my point, which apparently I didn’t make clear enough.
        rowanG077 1y ago
        The phrasing of your comment clearly implies an authoritive person or organisation telling us we would have AGI now. There are billions of people. You have people who think the earth is flat. You can probably find any insane takes if you look for it. Best not take get told anything by them as you have seemed to have taken it to heart.
pamelafox 1y ago
I am testing out gpt-5-mini for a RAG scenario, and I'm impressed so far. I used gpt-5-mini with reasoning_effort="minimal", and that model finally resisted a hallucination that every other model generated. Screenshot in post here: https://bsky.app/profile/pamelafox.bsky.social/post/3lvtdyvbh5g2o https://bsky.app/profile/pamelafox.bsky.social/post/3lvtdyvb... I'll run formal evaluations next.
- potatolicious 1y ago
  This feels like honestly the biggest gain/difference. I work on things that do a lot of tool calling, and the model hallucinating fake tools is a huge problem. Worse, sometimes the model will hallucinate a response directly without ever generating the tool call. The new training rewards that suppress hallucinations and tool-skipping hopefully push us in the right direction.
- ralfd 1y ago
  Q: What does a product manager do? GPT4: Collaborating with engineering, sales, marketing, finance, external partners, suppliers and customers to ensure …… etc GPT5: I don't know. Upon speaking these words, AI was enlightened.
  - ComputerGuru 1y ago
    That is genuinely nice to see. What are you using for the embeddings?
    - pamelafox 1y ago
      We use text-embedding-3-large, with both quantization and MRL reduction, plus oversampling on the search to compensate for the compression.
  - siva7 1y ago
    This is huge news if we finally have a model that is able to say "I don't know".
fatty_patty89 1y ago
What the fuck? Nobody else saw the cursor ceo looking through the gpt5 generated code, mindlessly scrolling saying "this looks roughly correct, i would love to merge that" LOL You can't make this up
- isoprophlex 1y ago
  This is the ideal software engineer. You may not like it, but this is what peak software engineering looks like. /s
- siva7 1y ago
  amazing time to be alive, alone for this clown show
  - throwawaybob420 1y ago
    if you’re not using an LLM to vibe code garbage then are you really a software developer?
- arresin 1y ago
  That explains a lot.
hrpnk 1y ago
The github issue showed in the livestream is getting lots of traction: https://github.com/openai/openai-python/issues/2472 https://github.com/openai/openai-python/issues/2472 It was (attempted to be) solved by a human before, yet not merged... With all the great coding models OpenAI has access to, their SDK team still feels too small for the needs.
- Iwan-Zotow 1y ago
  They hope next model will do SDK right
te_chris 1y ago
https://platform.openai.com/docs/guides/latest-model https://platform.openai.com/docs/guides/latest-model Looks like they're trying to lock us into using the Responses API for all the good stuff.
- deleted 1y ago
  [deleted]
henriquegodoy 1y ago
I dont think there's so much difference from opus 4.1 and gpt-5, probably just the context size, waiting for the gemini 3.0
- backscratches 1y ago
  gpt5 much cheaper
- blurbleblurble 1y ago
  Claude 5 is the one I'm most excited about.
sberens 1y ago
Interesting there doesn't seem to be benchmarking on codeforces
- sigbottle 1y ago
  I'm a codeforces guy, and I've benchmarked o3 on several of my favorite problems of various difficulty and concluded that o3 really isn't suitable for true reasoning still. Mostly because it's unable to think from first principles, so if you throw a non-standard problem it will brick. I think this will be a fundamental issue with any LLM. I will say I would far more appreciate an AI that when it faces these ambiguous problems, either provides sources for further reading, or just admits it doesn't know and is, you know, actually trying to work together to find a solution instead of being trained to 1 shot everything. When generalizing these skills to say, debugging, I will often just straight up ignore the AI slop output it concluded and instead explore the sources it found. o3 is surprisingly good at this. But for hard niche debugging, the conclusions it comes to are not only wrong, but it phrases it in an arrogant way and when you push back it's actually like talking to a narcissist (phrasing objections as "you feel", being excessively stubborn, word dumping a bunch of phrases that sound correct but don't hold up to scrutiny, etc).
jaflo 1y ago
I just wish their realtime audio pricing would go down but it looks like GPT-5 does not have support for that so we’re stuck with the old models.
zaronymous1 1y ago
Can anyone explain to me why they've removed parameter controls for temperature and top-p in reasoning models, including gpt-5? It strikes me that it makes it harder to build with these to do small tasks requiring high-levels of consistency, and in the API, I really value the ability to set certain tasks to a low temp.
jngiam1 1y ago
I was a little bummed that there wasn't more about better MCP support in ChatGPT, hopefully soon.
- cheema33 1y ago
  MCP is overhyped and most MCP servers are useless. What specific MCP server do you find critical in your regular use? And what functionality is missing that you wish to see in ChatGPT?
ivape 1y ago
Musk after GPT5 launch: "OpenAI is going to eat Microsoft alive" https://x.com/elonmusk/status/1953509998233104649 https://x.com/elonmusk/status/1953509998233104649 Anyone know why he said that?
- brookst 1y ago
  He was high AF?
- darylteo 1y ago
  It's not a hard logic path to follow - If AI becomes a digital necessity for modern society to function, Microsoft's relevance shrinks while OpenAI's relevance grows. Once OpenAI breaks out of the "App" space and into the "OS" and "Device" space, Microsoft may get absorbed into the ouroboros. OpenAI's dependence on Microsoft currently is purely financial (investment) and contractual (exclusivity, azure hosting).
- tough 1y ago
  agi clause comes to mind?
- czk 1y ago
  eventually traditional operating systems will cease to exist, you'll just have a model creating dynamic UX for you on the fly for whatever experience you want
- thomasfromcdnjs 1y ago
  I understood it as that the economic relationship they have is going to make Microsoft broke somehow, be it dollars and/or just the focus of the company.
- slowmotiony 1y ago
  Probably because of a mix of ketamine, magic mushrooms, ecstasy and adderall.
deleted 1y ago
[deleted]
nadis 1y ago
"When producing frontend code for web apps, GPT‑5 is more aesthetically-minded, ambitious, and accurate. In side-by-side comparisons with o3, GPT‑5 was preferred by our testers 70% of the time." That's really interesting to me. Looking forward to trying GPT-5!
attentive 1y ago
> scoring 74.9% on SWE-bench Verified and 88% on Aider polyglot why isn't it on https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/? "last updated August 07, 2025"
- tedsanders 1y ago
  The 88% is our self-reported score on our internal implementation of Aider polyglot. The leaderboard score would come from Aider independently running GPT-5 themselves. The score should be about the same. (I work at OpenAI.)
worik 1y ago
Diminishing returns?
attentive 1y ago
"Notably, GPT‑5 with minimal reasoning is a different model than the non-reasoning model in ChatGPT, and is better tuned for developers. The non-reasoning model used in ChatGPT is available as gpt-5-chat-latest." hmm, they should call it gpt-5-chat-nonreasoning or something.
- weird-eye-issue 1y ago
  Setting "reasoning_effort" to "minimal" translates to zero reasoning tokens from what I've seen. So you can get non-reasoning from both "gpt-5" and "gpt-5-chat-latest"
  - attentive 1y ago
    fwiw, I asked chatgpt: "gpt-5-chat-latest is described by OpenAI as a non-reasoning GPT-5 variant—meaning it doesn’t engage in the extended “thinking token” process at all. gpt-5 with reasoning_effort="minimal" still uses some internal reasoning tokens—just very few—so it’s not truly zero-reasoning. The difference: "minimal" is lightweight reasoning, while non-reasoning is essentially no structured chain-of-thought beyond the basic generation loop."
    - weird-eye-issue 1y ago
      If it did any reasoning then it would be billed as part of the reasoning tokens
    - wahnfrieden 1y ago
      Never ask it about itself
jodosha 1y ago
Still no CLI like Claude Code?
- Game_Ender 1y ago
  You are looking for Codex CLI [0]. 0 - https://github.com/openai/codex https://github.com/openai/codex
  - jodosha 1y ago
    Thank you!
- mediaman 1y ago
  It works on Codex CLI, install it with npm. That's been out for a while and used their 'codex' model, but they updated it today to default to gpt-5 instead.
  - jodosha 1y ago
    Oh nice, thanks!
guybedo 1y ago
here's a summary for this discussion: https://extraakt.com/extraakts/openai-s-gpt-5-performance-cost-context-and-coding-capabilities https://extraakt.com/extraakts/openai-s-gpt-5-performance-co...
mwigdahl 1y ago
Has anyone tried connecting up GPT-5 to Claude Code using the model environment variables?
planet_1649c 1y ago
Can we use this model in a fixed plan like claude code for which we can pay 100$ / month? Doesnt look like it. Unless they add a fixed pricing, claude imo still would be better from a developer POV
- spiderice 1y ago
  I just said something similar in another comment on this thread. I'm not interested in the mental aspect of getting charged per query. I feel like when I use pay-per-token tools, it's always in the back of my mind. Even if it's a bit more expensive to pay a flat rate, it's so worth it for the peace of mind.
- celeritascelery 1y ago
  They actually have a very similar setup with their plus and pro plans. They don’t claim unlimited usage, but say it should be very high. You don’t need to pay per token. https://x.com/embirico/status/1953590991870697896 https://x.com/embirico/status/1953590991870697896
- matltc 1y ago
  Was looking for this too. That said, Anthropic nerfed those plans pretty hard a few days ago. Imagine more to come, much like gh copilot is basically useless now. Literally having it write docs for a single function in an unfamiliar codebase spent 1% of my monthly "premium requests" allowance. Agent is terrible compared to Claude Code in my experience, even when using the same model (Sonnet 4)
wewewedxfgdf 1y ago
Tried it on a tough problem. GPT-5 solved the problem - which Gemini failed to solve - then failed 6 times in a row to write the code to fix it. I then gave ChatGPT-5's problem analysis to Google Gemini and it immediately implemented the correct fix. The lesson - ChatGPT is good at analysis and code reviews, not so good at coding.
- cperkins 1y ago
  I have something that both Gemini (via GCA) and CoPilot (Claude) analyzed and came up withe the same diagnosis. Each of them made the exact same wrong solution, and when I pointed that out, got further wrong. I haven't tried Chat GPT on it yet, hoping to do so soon.
  - cperkins 1y ago
    I used Cursor and Chat GPT 5 last night for the first time. Before I could even ask Chat GPT 5 about my issue it had scanned the .cpp file in question (because it was open in the editor) and had discovered some possible issues, one of which was the issue in the code. I confirmed that and gave it more description of the error behavior. It identified the problem in the code, and suggested two different CORRECT solutions (one simple, one more complex but "perfect"). I opted for the simple one. It did it. One tiny problem remained, I pointed it out, it fixed it. This was much better than Gemini or CoPilot on the exact same issue and the exact same commit pointer in my repo. Both of them suggested the same wrong solution and got themselves further and further wrong as they went. So, I guess as of today, Chat GPT 5 leads. YMMV
- Lionga 1y ago
  The real lesson is that these are just random results and all models fail at all kinds of things all the time and other times get things right in all kind of questions. Problem is the models have zero idea wether they are right or wrong and always believe they are right. Which makes them useful for anything were either you do not care if the answer is actually right or where somehow it is hard to come up with the right answer but very easy to verify it the answer is right and kind of useless for everything else.
6thbit 1y ago
Can anyone share their experience with codex CLI? I feel like that’s not mentioned enough and gpt5 is already the default model there.
- blurbleblurble 1y ago
  Not good sadly, Claude Code seems so much better in terms of overall polish but also in how it handles context. I don't really want to through the LLM into the deep end without proper tools and context, and I get the sense that this is what was happening with in Codex.
- deleted 1y ago
  [deleted]
- ed 1y ago
  I decided to check-in on Codex after being a longtime Claude Code user. The experience was not great. GPT5 is pretty solid, however! - The permission system is broken (this is such an obvious one that I wonder if it's specific to GPT5 or my environment). If you tell Codex to ask permission before running commands, it can't ever write to files. It also runs some commands (e.g. `sed`) without asking. Once you skip sandbox mode, it's difficult to go back. - You can't paste or attach images (helpful for design iteration) - No built-in login flow so you have to mess with your shell config and export your OpenAI key to all terminal processes. - Terminal width isn't respected. Model responses always wrap at some hard-coded value. Resizing the window doesn't correctly redraw the screen. - Some keyboard shortcuts aren't supported, like option+delete to delete words (which I use often, apparently...) This is on MacOS, iTerm2, Fish shell. I guess everyone uses Cursor or Windsurf?
  - wahnfrieden 1y ago
    They said images are coming soon to codex There is a built-in login flow
    - ed 1y ago
      > There is a built-in login flow Ah this doc needs to be updated: https://help.openai.com/en/articles/11096431-openai-codex-cli-getting-started https://help.openai.com/en/articles/11096431-openai-codex-cl...
joshmlewis 1y ago
It does really well at using tool calls to gain as much context as it can to provide thoughtful answers. In this example it did 6! tool calls in the first response while 4.1 did 3 and o3 did one at a time. https://promptslice.com/share/b-2ap_rfjeJgIQsG https://promptslice.com/share/b-2ap_rfjeJgIQsG
joshmlewis 1y ago
It's free in Cursor for the next few days, you should go try it out if you haven't. I've been an agentic coding power user since the day it came out across several IDE's/CLI tools and Cursor + GPT-5 seems to be a great combo.
austinmw 1y ago
Okay so say GPT-5 is better than Claude Opus 4.1. Then is GPT-5+Cursor better than Opus 4.1 + Claude Code? And if not, what's the best way to utilize GPT-5?
- felipemesquita 1y ago
  I’m not sure yet if it’s better than Claude, but the best way to use GPT-5 it is https://github.com/charmbracelet/crush https://github.com/charmbracelet/crush
- kristo 1y ago
  Apparently there is a cursor cli now… but I love the flat pricing of Claude’s Max plan and dislike having to worry about pricing and when to use “Max” mode in cursor.
redbell 1y ago
The fact that they intentionally ignored competitors' models in benchmarks and where comparing GPT-5 only to their previous models reminds me of Apple. They never compare their latest iPhone with any other phone from other brands but only to their previous(s) iPhone.
- iamsaitam 1y ago
  The artist way
chrismccord 1y ago
I'm really bummed out by this release. I expected this to best sonnet, or at least match, given all the hype. But it has drastically under performed on agent based work for me so far, even underperforming gpt-4.1. It struggles with basic instruction following. Basic things like: - "don't nest modules'–nests 4 mods in 1 file - "don't write typespecs"–writes typespecs - "Always give the user design choices"– skips design choices. gpt-4.1 way outperforms w/ same instructions. And sonnet is a whole different league (remains my goto). gpt-5 elixir code is syntactically correct, but weird in a lot of ways, junior-esque inefficient, and just odd. e.g function arguments that aren't used, yet passed in from callers, dup if checks, dup queries in same function. I imagine their chat and multimodal stuff strikes a nice balance with leaps in some areas, but for coding agents this is way behind any other SOTA model I've tried. Seems like this release was more about striking a capability balance b/w roflscale and costs than a gpt3-4 leap.
- wsintra 1y ago
  Maybe its became so intelligent it now wants to troll people as a way to create factions among the populace.
  - deleted 1y ago
    [deleted]
- enraged_camel 1y ago
  Claude has always been noticeably better for Elixir for me. GPT very frequently outputs pure garbage, and as far as I can tell this release is not much different.
- Jackson__ 1y ago
  Thankfully OAI will fix this, by removing GPT4.1 soon!
markr1 1y ago
I really hoped GPT-5 would level up, but right now it feels like a step back, not forward.
ryukoposting 1y ago
> Custom tools support constraining by developer-supplied context-free grammars. This sounds like a really cool feature. I'm imagining giving it a grammar that can only output safe, well-constrained SQL queries. Would I actually point an LLM directly at my database in production? Hell no! It's nice to see OpenAI trying to solve that problem anyway.
deleted 1y ago
[deleted]
vivzkestrel 1y ago
would be nice if we had some model out there with a context window of 1 billion tokens. i have about 25 .UNR files made with LEAD engine (heavily modified unreal engine 2.x) within which i want the AI to search for a string. Also got another 100 .utx files. Use-case game modding
weird-eye-issue 1y ago
gpt-5-chat-latest is giving much better results for our use case compared to gpt-5. Which puts me in a tricky position since gpt-5-chat-latest is not pinned and can change at any time...
- matchagaucho 1y ago
  It also lacks tool calling :-(
raducu 1y ago
Sample size of 1 but GPT-5 seems horrendous at coding? My go to benchmark is a 3d snake game Claude does almost flawlessly (or at least in 3-4 iterations) The prompt: write a 3d snake game in js and html. you can use any libraries you want. the game still happens inside a single plane, left arrow turns the snake left, right arrow turns it right. the plane is black and there's a green grid. there are multiple rewards of random colors at a given time. each time a reward is eaten, it becomes the snake's new head. The camera follows the snake's head, it is above an a bit behind it, looking forward. When the snake moves right or left, the camera follows gradually left or right, no snap movements. write everything in a single html file. EDIT: I'm not trying to shit on GPT-5, so many people here seem to be getting very good results, am I doing something wrong with my prompt?
- cncjchsue7 1y ago
  [dead]
- Frieren 1y ago
  > My go to benchmark is a 3d snake game Claude does almost flawlessly (or at least in 3-4 iterations) If you need to know how the snake game should look to get the code then Claude is not doing the work you are.
- M4v3R 1y ago
  This is what I got from your prompt in one shot with GPT-5 Thinking: Game: https://chatgpt.com/canvas/shared/6895f722f2708191ac4a6d16454312fe https://chatgpt.com/canvas/shared/6895f722f2708191ac4a6d1645... Conversation: https://chatgpt.com/share/6895f74a-0c5c-8004-b349-69da09653154 https://chatgpt.com/share/6895f74a-0c5c-8004-b349-69da096531... The controls are inverted for some reason and it could be a bit faster, but I fixed both of these easily with one prompt and here's the corrected version: https://chatgpt.com/canvas/shared/6895f82759f88191ba41c9fcd5153b91 https://chatgpt.com/canvas/shared/6895f82759f88191ba41c9fcd5...
  - raducu 1y ago
    Thanks, the issue was indeed not using explicitly the thinking model or they changed something over the weekend -- it's at least on par with Claude now. EDIT: clearly better than Claude or any other model that I tried before. I had a bonus benchmark -- add a narrow triangle on the head of the snake that indicates the direction of movement, after a single iteration GPT-5 fixed it whereas Claude could never get the rotation of the triangle right, nor could o3 the last time I tried.
skroumpelou 1y ago
I tried it out with warp terminal (warp.dev)! It will be my coding buddy today!
energy123 1y ago
I've gotten 100% cache misses so far. Has anyone got a cache hit?
magnusga 1y ago
Are anyone else experiencing extreme lag on responses from GPT-5 inside e.g. vscode? I use Claude Code as my daily driver and it takes on avg. 4-5 seconds to respond, assuming no hard thinking or research. For the same prompt, vscode with GPT-5 is taking 4-5 minutes. It is so slow it is rendered useless for us. I am using a new MacBook Pro (M4 Pro). Just wanted to know if I am the only one seeing this.
- blurbleblurble 1y ago
  Same for me. It seems to be doing alright at what it's doing, but definitely really slow. Considering trying out Kimi K2 which is kind of a funny surprise... I didn't think the day after GPT-5 came out I'd be contemplating switching to Kimi K2.
Awesomedonut 1y ago
Used it with Cursor and I've been completely blown away! It solved a pretty complex task in under an hour

470 points by 6thbit 1y ago

The problem with OpenAI models is the lack of a Max-like subscription for a good agentic harness. Maybe OpenAI or Microsoft could fix this. I just went through the agony of provisioning my team with new Claude Code 5x subs 2 weeks ago after reviewing all of the options available at that time. Since then, the major changes include a Cerebras sub for Qwen3 Coder 480B, and now GPT-5. I’m still not sure I made the right choice, but hey, I’m not married to it either. If you plan on using this much at all then the primary thing to avoid is API-based pay per use. It’s prohibitively costly to use regularly. And even for less important changes it never feels appropriate to use a lower quality model when the product counts. Claude Code won primarily because of the sub and that they have a top tier agentic harness and models that know how to use it. Opus and Sonnet are fantastic agents and very good at our use case, and were our preferred API-based models anyways. We can use Claude Code basically all day with at least Sonnet after using our Opus limits up. Worth nothing that Cline built a Claude Code provider that the derivatives aped which is great but I’ve found Claude Code to be as good or better anyways. The CLI interface is actually a bonus for ease of sharing state via copy/paste. I’ll probably change over to Gemini Code Assist next, as it’s half the price and more context length, but I’m waiting for a better Gemini 2.5 Pro and the gemini-cli/Code Assist extensions to have first party planning support, which you can get some form of third party through custom extensions with the cli, but as an agent harness they are incomplete without. The Cerebras + Qwen3 Coder 480B with qwen3-cli is seriously tempting. Crazy generation speed. Theres some question about how long big the rate limit really is but it’s half the cost of Claude Code 5x. I haven’t checked but I know qwen3-cli, which was introduced along side the model, is a fork of gemini-cli with Qwen-focused updates; wonder if they landed a planning tool? I don’t really consider Cursor, Windsurf, Cline, Roo, Kilo et al as they can’t provide a flat rate service with the kind of rate limits you can get with the aforementioned. GitHub Copilot could be a great offering if they were willing to really compete with a good unlimited premium plan but so far their best offering has less premium requests than I make in a week, possibly even in a few days. Would love to hear if I missed anything, or somehow missed some dynamic here worth considering. But as far as I can tell, given heavy use, you only have 3 options today: Claude Max, Gemini Code Assist, Cerebras Code.

energy123 1y ago
> If you plan on using this much at all then the primary thing to avoid is API-based pay per use. I find there's a niche where API pay-per-use is cost effective. It's for problems that require (i) small context and (ii) not much reasoning. Coding problems with 100k-200k context violates (i). Math problems violate (ii) because they generate long reasoning streams. Coding problems with 10k-20k context are well suited, because they generate only ~5k output tokens. That's $0.03-$0.04 per prompt to GPT-5 under flex pricing. The convenience is worth it, unless you're relying on a particular agentic harness that you don't control (I am not). For large context questions, I send them to a chat subscription, which gives me a budget of N prompts instead of N tokens. So naturally, all the 100k-400k token questions go there.
NullifyNAN 1y ago
OpenAI has answered your prayers. 16 hours ago the readme for codex CLI was updated. Now codex cli supports openai login like claude does, no API credits. From the readme: After you run codex select Sign in with ChatGPT. You'll need a Plus, Pro, or Team ChatGPT account, and will get access to our latest models, including gpt-5, at no extra cost to your plan. (Enterprise is coming soon.) Important: If you've used the Codex CLI before, you'll need to follow these steps to migrate from usage-based billing with your API key: Update the CLI with codex update and ensure codex --version is greater than 0.13 Ensure that there is no OPENAI_API_KEY environment variable set. (Check that env | grep 'OPENAI_API_KEY' returns empty) Run codex login again
- IanCal 1y ago
  Oh that’s fantastic news, thanks!

catigula 1y ago

I thought we were going to have AGI by now.

RS-232 1y ago
No shot. LLMs are simple text predictors and they are too stupid to get us to real AGI. To achieve AGI, we will need to be capable of high fidelity whole brain simulations that model the brain's entire physical, chemical, and biological behavior. We won't have that kind of computational power until quantum computers are mature.
- evantbyrne 1y ago
  It will be interesting to see if humans can manage to bioengineer human-level general intelligence into another species before computers.
- machiaweliczny 1y ago
  [flagged]
  - bopbopbop7 1y ago
    “some twist” is doing a lot of heavy lifting in that statement.
    - AppleBananaPie 1y ago
      CS will define, design and implement human level intelligence before neuroscience has done even the first. That's what I hear when people say stuff like this anyway. Similar to CS folks throwing around physics 'theories'
- nawgz 1y ago
  I don't really see any relationship between being able to model/simulate the brain and being able to exceed the brain in intelligence, can you explain more about that? Simulations sound like more of a computational and analytic problem with regards to having an accurate model. Maybe your point is that until we understand our own intelligence, which would be reflected in such a simulation, it would be difficult to improve upon it.

andrewmcwatters 1y ago

I wonder how good it is compared to Claude Sonnet 4, and when it's coming to GitHub Copilot. I almost exclusively wrote and released https://github.com/andrewmcwattersandco/git-fetch-file https://github.com/andrewmcwattersandco/git-fetch-file yesterday with GPT 4o and Claude Sonnet 4, and the latter's agentic behavior was quite nice. I barely had to guide it, and was able to quickly verify its output.

fleebee 1y ago
There is an option in GitHub Copilot settings to enable GPT-5 already.

croemer 1y ago

> GPT‑5 also excels at long-running agentic tasks—achieving SOTA results on τ2-bench telecom (96.7%), a tool-calling benchmark released just 2 months ago. Yes, but it does worse than o3 on the airline version of that benchmark. The prose is totally cherry picker.

Fogest 1y ago
How does the cost compare though? From my understanding o3 is pretty expensive to run. Is GPT-5 less costly? If so if the performance is close to o3 but cheaper, then it may still be a good improvement.
- low_tech_punk 1y ago
  I find it strange that GPT-5 is cheaper than GPT-4.1 in input token and is only slightly more expensive in output token. Is it marketing or actually reflecting the underlying compute resources?
  - AS04 1y ago
    Very likely to be an actual reflection. That's probably their real achievement here and the key reason why they are actually publishing it as GPT-5. More or less the best or near to it on everything while being one model, substantially cheaper than the competition.
    - ComputerGuru 1y ago
      But it can’t do audio in/out or image out. Feels like an architectural step back.
      - conradkay 1y ago
        My understanding is that image output is pretty separate and if it doesn’t seem that way, they’re just abstracting several models into one name
  - 1y ago

aliljet 1y ago

Between Opus aand GPT-5, it's not clear there's a substantial difference in software development expertise. The metric that I can't seem to get past in my attempts to use the systems is context awareness over long-running tasks. Producing a very complex, context-exceeding objective is a daily (maybe hourly) ocurrence for me. All I care about is how these systems manage context and stay on track over extended periods of time. What eval is tracking that? It seems like it's potentially the most imporatnt metric for real-world software engineering and not one-shot vibe prayers.

realusername 1y ago
Personally I think I'll wait for another 10x improvement for coding because with the current way it's going, they clearly need that.
- fsloth 1y ago
  From my experience when used through IDE such as Cursor the current gen Claude model enables impressive speedruns over commodity tasks. My context is a CAD application I’ve been writing as a hobby. I used to work in that field for a decade so have a pretty good touch on how long I would expect tasks to take. I’m using mostly a similar software stack as that at previous job and am definetly getting stuff done much faster on holiday at home than at that previous work. Of course the codebase is also a lot smaller, intrinsic motivation, etc, but still.
  - BoredPositron 1y ago
    How often do you have to build the simple scaffolding though?
    - fsloth 1y ago
      At a real job? Not that often! And it's miserable in large scale architecture. However, at leas for me there is lots of "small enough context" boilerplate that the context can deal with. Clearly this is not a tool in the sense it's predictable.

coconut08 1y ago

over the last week or so I have put probably close to 70 hours into playing around with cursor and claude code and a few other tools (its become my new obsession). I've been blown away by how good and reliable it is now. That said the reality is in my experience the only models that actually work in any sort of reliable way are claude models. I dont care what any benchmark says because the only thing that actually matters is actual use. I'm really hoping that this new gpt model actually works for this usecase because competition is great and the price is also great.

deleted 1y ago
[deleted]
ralfd 1y ago
Just replying to ask you next week what your assessment on GPT5 is.
- coconut08 1y ago
  I've been trying it out with openai codex over the last day and a half and I have been incredibly impressed. It has been working quite well. I also had it look over some code that claude produced for me and it said that it would be better to approach it another way and it completely rewrote it in a way that actually was significantly better. The UX for codex is quite a bit worse than Claude Code, but the model has been good enough to justify the switch for now. I'm hopeful that cursor cli will eventually have a good enough ux such that I can switch to it and have access to all of the models rather than needing to use disparate tools for everything. I would strongly suggest you check out gpt 5 for agentic stuff if you are interested.
throwaway_2898 1y ago
How much of the product were you able to build to say it was good/reliable? IME, 70 hours can get you to a PoC that "works", building beyond the initial set of features — like say a first draft of all the APIs — does it do well once you start layering features?
- 1y ago

timhigins 1y ago

I opened up the developer playground and the model selection dropdown showed GPT-5 and then it disappeared. Also I don't see it in ChatGPT Pro. What's up?

IAmGraydon 1y ago
Not showing in my Pro account either. As someone else mentioned, I’m sure it’s throttling due to high use right now.
Fogest 1y ago
It's probably being throttled due to high usage.
brookst 1y ago
Shipping something at the moment of announcement is always hell.

sebdufbeau 1y ago

Has the API rollout started? It's not available in our org, even if we've been verified for a few months EDIT: It's out now

spullara 1y ago
it is out yet. i poll the api for the models and update this GitHub hourly. https://github.com/spullara/models https://github.com/spullara/models

low_tech_punk 1y ago

The ability to specify a context-free grammar as output constraint? This blows my mind. How do you control the auto regressive sampling to guarantee the correct syntax?

qsort 1y ago
You sample only from tokens that could possibly result in a valid production for the grammar. It's an inference-only thing.
- low_tech_punk 1y ago
  ah, thanks!
evnc 1y ago
I assume they're doing "Structured Generation" or "Guided generation", which has been possible for a while if you control the LLM itself e.g. running an OSS model, e.g. [0][1]. It's cool to see a major API provider offer it, though. The basic idea is: at each auto-regressive step (each token generation), instead of letting the model generate a probability distribution over "all tokens in the entire vocab it's ever seen" (the default), only allow the model to generate a probability distribution over "this specific set of tokens I provide". And that set can change from one sampling set to the next, according to a given grammar. E.g. if you're using a JSON grammar, and you've just generated a `{`, you can provide the model a choice of only which tokens are valid JSON immediately after a `{`, etc. [0] https://github.com/dottxt-ai/outlines https://github.com/dottxt-ai/outlines [1] https://github.com/guidance-ai/guidance https://github.com/guidance-ai/guidance

low_tech_punk 1y ago

Tried using gpt-5 family with response API and got error "gpt-5 does not exist or you don't have access to it". I guess they are not rolling out in lock step with the live stream and blog article?

low_tech_punk 1y ago
Can confirm that they are rolling out. It's working for me.
diggan 1y ago
Seems they're doing rollout over time, I'm not seeing it anywhere yet.

skepticATX 1y ago

This was really a bad release for OpenAI, if benchmarks are even somewhat indicative of how the model will perform in practice.

robterrell 1y ago
In what ways?
mediaman 1y ago
I actually don't agree. Tool use is the key to successful enterprise product integration and they have done some very good work here. This is much more important to commercialization than, for example, creative writing quality (which it reportedly is not good at).

jumploops 1y ago

If the model is as good as the benchmarks say, the pricing is fantastic: Input: $1.25 / 1M tokens (cached: $0.125/1Mtok) Output: $10 / 1M tokens For context, Claude Opus 4.1 is $15 / 1M for input tokens and $75/1M for output tokens. The big question remains: how well does it handle tools? (i.e. compared to Claude Code) Initial demos look good, but it performs worse than o3 on Tau2-bench airline, so the jury is still out.

addaon 1y ago
> Output: $10 / 1M tokens It's interesting that they're using flat token pricing for a "model" that is explicitly made of (at least) two underlying models, one with much lower compute costs than the other; and with use ability to at least influence (via prompt) if not choose which model is being used. I have to assume this pricing model is based on a predicted split between how often the underlying models get used; I wonder if that will hold up, if users will instead try to rouse the better model into action more than expected, or if the pricing is so padded that it doesn't matter.
- simianwords 1y ago
  > that is explicitly made of (at least) two underlying models what do you mean?
  - addaon 1y ago
    > a smart and fast model that answers most questions, a deeper reasoning model for harder problems, and a real-time router that quickly decides which model to use based on conversation type, complexity, tool needs, and explicit intent (for example, if you say “think hard about this” in the prompt). From https://openai.com/index/gpt-5-system-card/ https://openai.com/index/gpt-5-system-card/
    - tedsanders 1y ago
      In the API, there’s no router. Developers just pick whether they use the reasoning model or non-thinking ChatGPT model.

6thbit 1y ago

Seems they have quietly increased the context window up to 400,000 https://platform.openai.com/docs/models/gpt-5 https://platform.openai.com/docs/models/gpt-5

ralfd 1y ago
How does that compare to Claude/GPT4?
- 6thbit 1y ago
  4o - 128k o3 - 200k Opus 4.1 - 200k Sonnet 4 - 200k So, at least twice larger context than those
- hrpnk 1y ago
  gpt4.1 has 1M input and 32k output, Sonnet 4 200k/64k
simianwords 1y ago
but is it for the model in chatgpt.com as well?
Iwan-Zotow 1y ago
Input plus output?

mehmetoguzderin 1y ago

Context-free grammar and regex support are exciting. I wonder what, or whether, there are differences from the Lark-like CFG of llguidance, which powers the JSON schema of the OpenAI API [^1]. [^1]: https://github.com/guidance-ai/llguidance/blob/f4592cc0c783a987fa44791fc1d702208b366613/README.md https://github.com/guidance-ai/llguidance/blob/f4592cc0c783a...

msp26 1y ago
Yeah that was the only exciting part of the announcement for me haha. Can't wait to play around with it. I'm already running into a bunch of issues with the structured output APIs from other companies like Google and OpenAI have been doing a great job on this front.
- chrisweekly 1y ago
  > "I'm already running into a bunch of issues with the structured output APIs from other companies like Google and OpenAI have been doing a great job on this front." This run-on sentence swerved at the end; I really can't tell what your point is. Could you reword it for clarity?
  - petercooper 1y ago
    I read it as "... from other companies, like Google, and OpenAI have been doing a great job on this front"
    - mehmetoguzderin 1y ago
      I'm not sure if it's due to experience with the aforementioned APIs, but I also read the same, “issues with APIs like ..., and (in contrast) OpenAI have been doing a great job”

belter 1y ago

We were promised AGI and all we got was code generators...

bmau5 1y ago
It's a logical starting point, given there are pretty defined success/failure criteria
- ehutch79 1y ago
  The hype is real. We were told that we'd have AGI and be out of jobs 2 years ago, let alone today.
  - brookst 1y ago
    We were also told that AGI would never happen, that it was 6 months away, that it is 20 years away. I’m not sure of the utility of being so outraged that some people made wrong predictions.
  - rowanG077 1y ago
    By whom? I don't think anyone seriously said in 2023 we have AGI in two years. Even now, no one reputable is claiming AGI in two years.
    - ehutch79 1y ago
      Random on YouTube, random here on hacker news. No, I don’t take them seriously, that was my point, which apparently I didn’t make clear enough.
      - rowanG077 1y ago
        The phrasing of your comment clearly implies an authoritive person or organisation telling us we would have AGI now. There are billions of people. You have people who think the earth is flat. You can probably find any insane takes if you look for it. Best not take get told anything by them as you have seemed to have taken it to heart.

pamelafox 1y ago

I am testing out gpt-5-mini for a RAG scenario, and I'm impressed so far. I used gpt-5-mini with reasoning_effort="minimal", and that model finally resisted a hallucination that every other model generated. Screenshot in post here: https://bsky.app/profile/pamelafox.bsky.social/post/3lvtdyvbh5g2o https://bsky.app/profile/pamelafox.bsky.social/post/3lvtdyvb... I'll run formal evaluations next.

potatolicious 1y ago
This feels like honestly the biggest gain/difference. I work on things that do a lot of tool calling, and the model hallucinating fake tools is a huge problem. Worse, sometimes the model will hallucinate a response directly without ever generating the tool call. The new training rewards that suppress hallucinations and tool-skipping hopefully push us in the right direction.
ralfd 1y ago
Q: What does a product manager do? GPT4: Collaborating with engineering, sales, marketing, finance, external partners, suppliers and customers to ensure …… etc GPT5: I don't know. Upon speaking these words, AI was enlightened.
- ComputerGuru 1y ago
  That is genuinely nice to see. What are you using for the embeddings?
  - pamelafox 1y ago
    We use text-embedding-3-large, with both quantization and MRL reduction, plus oversampling on the search to compensate for the compression.
- siva7 1y ago
  This is huge news if we finally have a model that is able to say "I don't know".

fatty_patty89 1y ago

What the fuck? Nobody else saw the cursor ceo looking through the gpt5 generated code, mindlessly scrolling saying "this looks roughly correct, i would love to merge that" LOL You can't make this up

isoprophlex 1y ago
This is the ideal software engineer. You may not like it, but this is what peak software engineering looks like. /s
siva7 1y ago
amazing time to be alive, alone for this clown show
- throwawaybob420 1y ago
  if you’re not using an LLM to vibe code garbage then are you really a software developer?
arresin 1y ago
That explains a lot.

hrpnk 1y ago

The github issue showed in the livestream is getting lots of traction: https://github.com/openai/openai-python/issues/2472 https://github.com/openai/openai-python/issues/2472 It was (attempted to be) solved by a human before, yet not merged... With all the great coding models OpenAI has access to, their SDK team still feels too small for the needs.

Iwan-Zotow 1y ago
They hope next model will do SDK right

te_chris 1y ago

https://platform.openai.com/docs/guides/latest-model https://platform.openai.com/docs/guides/latest-model Looks like they're trying to lock us into using the Responses API for all the good stuff.

deleted 1y ago
[deleted]

henriquegodoy 1y ago

I dont think there's so much difference from opus 4.1 and gpt-5, probably just the context size, waiting for the gemini 3.0

backscratches 1y ago
gpt5 much cheaper
blurbleblurble 1y ago
Claude 5 is the one I'm most excited about.

sberens 1y ago

Interesting there doesn't seem to be benchmarking on codeforces

sigbottle 1y ago
I'm a codeforces guy, and I've benchmarked o3 on several of my favorite problems of various difficulty and concluded that o3 really isn't suitable for true reasoning still. Mostly because it's unable to think from first principles, so if you throw a non-standard problem it will brick. I think this will be a fundamental issue with any LLM. I will say I would far more appreciate an AI that when it faces these ambiguous problems, either provides sources for further reading, or just admits it doesn't know and is, you know, actually trying to work together to find a solution instead of being trained to 1 shot everything. When generalizing these skills to say, debugging, I will often just straight up ignore the AI slop output it concluded and instead explore the sources it found. o3 is surprisingly good at this. But for hard niche debugging, the conclusions it comes to are not only wrong, but it phrases it in an arrogant way and when you push back it's actually like talking to a narcissist (phrasing objections as "you feel", being excessively stubborn, word dumping a bunch of phrases that sound correct but don't hold up to scrutiny, etc).

jaflo 1y ago

I just wish their realtime audio pricing would go down but it looks like GPT-5 does not have support for that so we’re stuck with the old models.

zaronymous1 1y ago

Can anyone explain to me why they've removed parameter controls for temperature and top-p in reasoning models, including gpt-5? It strikes me that it makes it harder to build with these to do small tasks requiring high-levels of consistency, and in the API, I really value the ability to set certain tasks to a low temp.

jngiam1 1y ago

I was a little bummed that there wasn't more about better MCP support in ChatGPT, hopefully soon.

cheema33 1y ago
MCP is overhyped and most MCP servers are useless. What specific MCP server do you find critical in your regular use? And what functionality is missing that you wish to see in ChatGPT?

ivape 1y ago

Musk after GPT5 launch: "OpenAI is going to eat Microsoft alive" https://x.com/elonmusk/status/1953509998233104649 https://x.com/elonmusk/status/1953509998233104649 Anyone know why he said that?

brookst 1y ago
He was high AF?
darylteo 1y ago
It's not a hard logic path to follow - If AI becomes a digital necessity for modern society to function, Microsoft's relevance shrinks while OpenAI's relevance grows. Once OpenAI breaks out of the "App" space and into the "OS" and "Device" space, Microsoft may get absorbed into the ouroboros. OpenAI's dependence on Microsoft currently is purely financial (investment) and contractual (exclusivity, azure hosting).
tough 1y ago
agi clause comes to mind?
czk 1y ago
eventually traditional operating systems will cease to exist, you'll just have a model creating dynamic UX for you on the fly for whatever experience you want
thomasfromcdnjs 1y ago
I understood it as that the economic relationship they have is going to make Microsoft broke somehow, be it dollars and/or just the focus of the company.
slowmotiony 1y ago
Probably because of a mix of ketamine, magic mushrooms, ecstasy and adderall.

deleted 1y ago

[deleted]

nadis 1y ago

"When producing frontend code for web apps, GPT‑5 is more aesthetically-minded, ambitious, and accurate. In side-by-side comparisons with o3, GPT‑5 was preferred by our testers 70% of the time." That's really interesting to me. Looking forward to trying GPT-5!

attentive 1y ago

> scoring 74.9% on SWE-bench Verified and 88% on Aider polyglot why isn't it on https://aider.chat/docs/leaderboards/ https://aider.chat/docs/leaderboards/? "last updated August 07, 2025"

tedsanders 1y ago
The 88% is our self-reported score on our internal implementation of Aider polyglot. The leaderboard score would come from Aider independently running GPT-5 themselves. The score should be about the same. (I work at OpenAI.)

worik 1y ago

Diminishing returns?

attentive 1y ago

"Notably, GPT‑5 with minimal reasoning is a different model than the non-reasoning model in ChatGPT, and is better tuned for developers. The non-reasoning model used in ChatGPT is available as gpt-5-chat-latest." hmm, they should call it gpt-5-chat-nonreasoning or something.

weird-eye-issue 1y ago
Setting "reasoning_effort" to "minimal" translates to zero reasoning tokens from what I've seen. So you can get non-reasoning from both "gpt-5" and "gpt-5-chat-latest"
- attentive 1y ago
  fwiw, I asked chatgpt: "gpt-5-chat-latest is described by OpenAI as a non-reasoning GPT-5 variant—meaning it doesn’t engage in the extended “thinking token” process at all. gpt-5 with reasoning_effort="minimal" still uses some internal reasoning tokens—just very few—so it’s not truly zero-reasoning. The difference: "minimal" is lightweight reasoning, while non-reasoning is essentially no structured chain-of-thought beyond the basic generation loop."
  - weird-eye-issue 1y ago
    If it did any reasoning then it would be billed as part of the reasoning tokens
  - wahnfrieden 1y ago
    Never ask it about itself

jodosha 1y ago

Still no CLI like Claude Code?

Game_Ender 1y ago
You are looking for Codex CLI [0]. 0 - https://github.com/openai/codex https://github.com/openai/codex
- jodosha 1y ago
  Thank you!
mediaman 1y ago
It works on Codex CLI, install it with npm. That's been out for a while and used their 'codex' model, but they updated it today to default to gpt-5 instead.
- jodosha 1y ago
  Oh nice, thanks!

guybedo 1y ago

here's a summary for this discussion: https://extraakt.com/extraakts/openai-s-gpt-5-performance-cost-context-and-coding-capabilities https://extraakt.com/extraakts/openai-s-gpt-5-performance-co...

mwigdahl 1y ago

Has anyone tried connecting up GPT-5 to Claude Code using the model environment variables?

planet_1649c 1y ago

Can we use this model in a fixed plan like claude code for which we can pay 100$ / month? Doesnt look like it. Unless they add a fixed pricing, claude imo still would be better from a developer POV

spiderice 1y ago
I just said something similar in another comment on this thread. I'm not interested in the mental aspect of getting charged per query. I feel like when I use pay-per-token tools, it's always in the back of my mind. Even if it's a bit more expensive to pay a flat rate, it's so worth it for the peace of mind.
celeritascelery 1y ago
They actually have a very similar setup with their plus and pro plans. They don’t claim unlimited usage, but say it should be very high. You don’t need to pay per token. https://x.com/embirico/status/1953590991870697896 https://x.com/embirico/status/1953590991870697896
matltc 1y ago
Was looking for this too. That said, Anthropic nerfed those plans pretty hard a few days ago. Imagine more to come, much like gh copilot is basically useless now. Literally having it write docs for a single function in an unfamiliar codebase spent 1% of my monthly "premium requests" allowance. Agent is terrible compared to Claude Code in my experience, even when using the same model (Sonnet 4)

wewewedxfgdf 1y ago

Tried it on a tough problem. GPT-5 solved the problem - which Gemini failed to solve - then failed 6 times in a row to write the code to fix it. I then gave ChatGPT-5's problem analysis to Google Gemini and it immediately implemented the correct fix. The lesson - ChatGPT is good at analysis and code reviews, not so good at coding.

cperkins 1y ago
I have something that both Gemini (via GCA) and CoPilot (Claude) analyzed and came up withe the same diagnosis. Each of them made the exact same wrong solution, and when I pointed that out, got further wrong. I haven't tried Chat GPT on it yet, hoping to do so soon.
- cperkins 1y ago
  I used Cursor and Chat GPT 5 last night for the first time. Before I could even ask Chat GPT 5 about my issue it had scanned the .cpp file in question (because it was open in the editor) and had discovered some possible issues, one of which was the issue in the code. I confirmed that and gave it more description of the error behavior. It identified the problem in the code, and suggested two different CORRECT solutions (one simple, one more complex but "perfect"). I opted for the simple one. It did it. One tiny problem remained, I pointed it out, it fixed it. This was much better than Gemini or CoPilot on the exact same issue and the exact same commit pointer in my repo. Both of them suggested the same wrong solution and got themselves further and further wrong as they went. So, I guess as of today, Chat GPT 5 leads. YMMV
Lionga 1y ago
The real lesson is that these are just random results and all models fail at all kinds of things all the time and other times get things right in all kind of questions. Problem is the models have zero idea wether they are right or wrong and always believe they are right. Which makes them useful for anything were either you do not care if the answer is actually right or where somehow it is hard to come up with the right answer but very easy to verify it the answer is right and kind of useless for everything else.

6thbit 1y ago

Can anyone share their experience with codex CLI? I feel like that’s not mentioned enough and gpt5 is already the default model there.

blurbleblurble 1y ago
Not good sadly, Claude Code seems so much better in terms of overall polish but also in how it handles context. I don't really want to through the LLM into the deep end without proper tools and context, and I get the sense that this is what was happening with in Codex.
deleted 1y ago
[deleted]
ed 1y ago
I decided to check-in on Codex after being a longtime Claude Code user. The experience was not great. GPT5 is pretty solid, however! - The permission system is broken (this is such an obvious one that I wonder if it's specific to GPT5 or my environment). If you tell Codex to ask permission before running commands, it can't ever write to files. It also runs some commands (e.g. `sed`) without asking. Once you skip sandbox mode, it's difficult to go back. - You can't paste or attach images (helpful for design iteration) - No built-in login flow so you have to mess with your shell config and export your OpenAI key to all terminal processes. - Terminal width isn't respected. Model responses always wrap at some hard-coded value. Resizing the window doesn't correctly redraw the screen. - Some keyboard shortcuts aren't supported, like option+delete to delete words (which I use often, apparently...) This is on MacOS, iTerm2, Fish shell. I guess everyone uses Cursor or Windsurf?
- wahnfrieden 1y ago
  They said images are coming soon to codex There is a built-in login flow
  - ed 1y ago
    > There is a built-in login flow Ah this doc needs to be updated: https://help.openai.com/en/articles/11096431-openai-codex-cli-getting-started https://help.openai.com/en/articles/11096431-openai-codex-cl...

joshmlewis 1y ago

It does really well at using tool calls to gain as much context as it can to provide thoughtful answers. In this example it did 6! tool calls in the first response while 4.1 did 3 and o3 did one at a time. https://promptslice.com/share/b-2ap_rfjeJgIQsG https://promptslice.com/share/b-2ap_rfjeJgIQsG

joshmlewis 1y ago

It's free in Cursor for the next few days, you should go try it out if you haven't. I've been an agentic coding power user since the day it came out across several IDE's/CLI tools and Cursor + GPT-5 seems to be a great combo.

austinmw 1y ago

Okay so say GPT-5 is better than Claude Opus 4.1. Then is GPT-5+Cursor better than Opus 4.1 + Claude Code? And if not, what's the best way to utilize GPT-5?

felipemesquita 1y ago
I’m not sure yet if it’s better than Claude, but the best way to use GPT-5 it is https://github.com/charmbracelet/crush https://github.com/charmbracelet/crush
kristo 1y ago
Apparently there is a cursor cli now… but I love the flat pricing of Claude’s Max plan and dislike having to worry about pricing and when to use “Max” mode in cursor.

redbell 1y ago

The fact that they intentionally ignored competitors' models in benchmarks and where comparing GPT-5 only to their previous models reminds me of Apple. They never compare their latest iPhone with any other phone from other brands but only to their previous(s) iPhone.

iamsaitam 1y ago
The artist way

chrismccord 1y ago

I'm really bummed out by this release. I expected this to best sonnet, or at least match, given all the hype. But it has drastically under performed on agent based work for me so far, even underperforming gpt-4.1. It struggles with basic instruction following. Basic things like: - "don't nest modules'–nests 4 mods in 1 file - "don't write typespecs"–writes typespecs - "Always give the user design choices"– skips design choices. gpt-4.1 way outperforms w/ same instructions. And sonnet is a whole different league (remains my goto). gpt-5 elixir code is syntactically correct, but weird in a lot of ways, junior-esque inefficient, and just odd. e.g function arguments that aren't used, yet passed in from callers, dup if checks, dup queries in same function. I imagine their chat and multimodal stuff strikes a nice balance with leaps in some areas, but for coding agents this is way behind any other SOTA model I've tried. Seems like this release was more about striking a capability balance b/w roflscale and costs than a gpt3-4 leap.

wsintra 1y ago
Maybe its became so intelligent it now wants to troll people as a way to create factions among the populace.
- deleted 1y ago
  [deleted]
enraged_camel 1y ago
Claude has always been noticeably better for Elixir for me. GPT very frequently outputs pure garbage, and as far as I can tell this release is not much different.
Jackson__ 1y ago
Thankfully OAI will fix this, by removing GPT4.1 soon!

markr1 1y ago

I really hoped GPT-5 would level up, but right now it feels like a step back, not forward.

ryukoposting 1y ago

> Custom tools support constraining by developer-supplied context-free grammars. This sounds like a really cool feature. I'm imagining giving it a grammar that can only output safe, well-constrained SQL queries. Would I actually point an LLM directly at my database in production? Hell no! It's nice to see OpenAI trying to solve that problem anyway.

deleted 1y ago

[deleted]

vivzkestrel 1y ago

would be nice if we had some model out there with a context window of 1 billion tokens. i have about 25 .UNR files made with LEAD engine (heavily modified unreal engine 2.x) within which i want the AI to search for a string. Also got another 100 .utx files. Use-case game modding

weird-eye-issue 1y ago

gpt-5-chat-latest is giving much better results for our use case compared to gpt-5. Which puts me in a tricky position since gpt-5-chat-latest is not pinned and can change at any time...

matchagaucho 1y ago
It also lacks tool calling :-(

raducu 1y ago

Sample size of 1 but GPT-5 seems horrendous at coding? My go to benchmark is a 3d snake game Claude does almost flawlessly (or at least in 3-4 iterations) The prompt: write a 3d snake game in js and html. you can use any libraries you want. the game still happens inside a single plane, left arrow turns the snake left, right arrow turns it right. the plane is black and there's a green grid. there are multiple rewards of random colors at a given time. each time a reward is eaten, it becomes the snake's new head. The camera follows the snake's head, it is above an a bit behind it, looking forward. When the snake moves right or left, the camera follows gradually left or right, no snap movements. write everything in a single html file. EDIT: I'm not trying to shit on GPT-5, so many people here seem to be getting very good results, am I doing something wrong with my prompt?

cncjchsue7 1y ago
[dead]
Frieren 1y ago
> My go to benchmark is a 3d snake game Claude does almost flawlessly (or at least in 3-4 iterations) If you need to know how the snake game should look to get the code then Claude is not doing the work you are.
M4v3R 1y ago
This is what I got from your prompt in one shot with GPT-5 Thinking: Game: https://chatgpt.com/canvas/shared/6895f722f2708191ac4a6d16454312fe https://chatgpt.com/canvas/shared/6895f722f2708191ac4a6d1645... Conversation: https://chatgpt.com/share/6895f74a-0c5c-8004-b349-69da09653154 https://chatgpt.com/share/6895f74a-0c5c-8004-b349-69da096531... The controls are inverted for some reason and it could be a bit faster, but I fixed both of these easily with one prompt and here's the corrected version: https://chatgpt.com/canvas/shared/6895f82759f88191ba41c9fcd5153b91 https://chatgpt.com/canvas/shared/6895f82759f88191ba41c9fcd5...
- raducu 1y ago
  Thanks, the issue was indeed not using explicitly the thinking model or they changed something over the weekend -- it's at least on par with Claude now. EDIT: clearly better than Claude or any other model that I tried before. I had a bonus benchmark -- add a narrow triangle on the head of the snake that indicates the direction of movement, after a single iteration GPT-5 fixed it whereas Claude could never get the rotation of the triangle right, nor could o3 the last time I tried.

skroumpelou 1y ago

I tried it out with warp terminal (warp.dev)! It will be my coding buddy today!

energy123 1y ago

I've gotten 100% cache misses so far. Has anyone got a cache hit?

magnusga 1y ago

Are anyone else experiencing extreme lag on responses from GPT-5 inside e.g. vscode? I use Claude Code as my daily driver and it takes on avg. 4-5 seconds to respond, assuming no hard thinking or research. For the same prompt, vscode with GPT-5 is taking 4-5 minutes. It is so slow it is rendered useless for us. I am using a new MacBook Pro (M4 Pro). Just wanted to know if I am the only one seeing this.

blurbleblurble 1y ago
Same for me. It seems to be doing alright at what it's doing, but definitely really slow. Considering trying out Kimi K2 which is kind of a funny surprise... I didn't think the day after GPT-5 came out I'd be contemplating switching to Kimi K2.

Awesomedonut 1y ago

Used it with Cursor and I've been completely blown away! It solved a pretty complex task in under an hour