5 ms·
GPT-5.5 Price Increase: What It Costs
- jsnell 6mo agoThis doesn't seem to be controlling for the number of turns in any way. Am I missing something? Stronger models needing fewer turns to achieve a task feels like a prime source of efficiency gains for agentic coding, more so than individual responses being shorter.
- sarjann 6mo agoI think it should be tested on goals. E.g. Crack this puzzle, fix this code so these tests pass. (A human can verify it doesn't cheese things).
- jfim 6mo agoThey also don't mention what their sample size is, or anything about the distribution of input and response lengths. It'd be interesting to see the distributions if the author actually plotted the data, so we could see if their analysis holds water or not. A plot of the input lengths using ggplot2 geom_density with color and fill by model, 0.1 alpha, and an appropriate bandwidth adjustment would allow us to see if the input data distribution looks similar across the two, and using the same for the output length distributions, faceted by the input length bins would give us an idea if those look the same too. Edit: Or even a faceted plot using input bins of output length/input length.
- crazylogger 6mo agoOpenRouter may see you fire hundreds of requests at them, but they have no idea that "these 50 requests here at 4PM are for task A", "those 100 requests there does task B", etc. So it's a shallow analysis at the "overall request shape" level.
- WhereIsTheTruth 6mo ago[dead]
- i_think_so 6mo agoHas any enterprising hacker here yet graphed price vs "output" over time since 2023, taking "quality" into account? That's got to be a very tricky analysis given how subjective quality is. But I'm sure there are people trying to pin it down.
- helloplanets 6mo agoQuality would be performance against different given benchmarks, I assume? There's multiple open weight models you can run on a pretty standard computer at home, which match the quality of GPT 4. I guess that would also change the equation.
- coalhouse 6mo agoanything that compares proprietary models will be very miscalibrated and may not be indicative, there have been too many model changes in both chat and the api where model providers did not even say the word before it got too noticable
- lukewarm707 6mo agoartificial analysis has an intelligence benchmark
- iceKirin 6mo agoI feel that the recent iterations of LLM haven't provided an intuitive qualitative leap. Have they entered a bottleneck period so quickly?
- gchamonlive 6mo agoMy take is that demand is also increasing, so maybe they are making incremental improvements to model quality while focusing on improving inference costs. Prices are increasing though because even if they achieve a very efficient model, they are still selling at a loss.
- olao99 6mo agoFor what is worth I find GPT 5.5 qualitatively different than 5.4 and 5.3 If I had to collapse the nature of the difference in one sentence it'd be that the 5.5 does more what I'm asking it to do versus doing a small aspect of what I'm asking then stopping. 5.4 required a lot of "continue" encouragement. 5.5 just "gets it" a bit more What is boils down to for me is that even though it's more expensive I would much rather use 5.5 on low then 5.4/5.3 on high/medium
- benjiro3000 6mo ago[dead]
- varispeed 6mo agoThey must have changed something recently as when 5.5 first dropped I was unable to make it do anything. It would say it will implement, but would never actually do it, no matter how many times I tried to tell it what it needs to do. It would acknowledge what needs to be done, even create step by step plan and then ask if it should do it. I would confirm and then it will just go around reiterating the plan and that this time it will start. Annoying and funny. Now it doesn't seem to be doing that anymore.
- Wingy 6mo agoI think that's a failure mode of using the legacy completions API rather than the new responses API. With the responses API, the agent actually goes and does the things it's supposed to do.
- coalhouse 6mo agoit does seem like a step change in token efficiency, though based on the earlier artificial analysis reporting it's also quite the cost lottery and i'm not sure i am comfortable with that
- XCSme 6mo ago~3.5x more expensive to run my benchmarks[0]. [0]: https://aibenchy.com/compare/openai-gpt-5-4-medium/openai-gpt-5-5-medium/ https://aibenchy.com/compare/openai-gpt-5-4-medium/openai-gp...
- iosjunkie 6mo agoSure, but it did better on the test, which matches OpenAI's claim. More bang for more buck. Interestingly, using your tests as a comparison, 5.5 low beats 5.4 medium at a 82% of the cost.[0] [0]: https://aibenchy.com/compare/openai-gpt-5-4-medium/openai-gpt-5-5-low/ https://aibenchy.com/compare/openai-gpt-5-4-medium/openai-gp...
- XCSme 6mo ago5.5 is definitely smarter indeed. And yes, 5.5 low seems to be as good as 5.4 medium, but a lot cheaper. In my experience that also holds up in real-life n8n agentic usage, so using 5.5 low is the best way to go atm.
- polski-g 6mo agoInteresting test results. GLM-5 is ranked better than GLM-5.1
- XCSme 6mo agoI think it's because the tests are more for general intelligence, but 5.1 is likely aimed more towards agentic coding and tool usage, so it lost a bit of intelligence in exchange for consistency and more focus coding use-case.
- languid-photic 6mo agoWe track performance vs. the all-in cost of completing real engineering tasks, rather than cost per token. [1] Cost per token is a bit misleading because, as others have noted, different models use tokens in different ways. (Aside - This is also why TPS isn't a great metric). We found that 5.5 is about 1.5-2x more expensive overall. On a "Pareto" basis, we only find 5.5 xhigh worth it. At the lower reasoning levels, 5.4 still edges it out on cost/perf. We take a spec-driven approach and mostly work in TS (on product development), so if you use a more steer-y approach, or work in a different domain, YMMV. [1] https://voratiq.com/leaderboard?x=cost https://voratiq.com/leaderboard?x=cost
- digdugdirk 6mo agoInteresting! I've been thinking about how to create a similar type of evaluation system for myself. How do you handle tweaks to agentic tasks? Say that a model gets pretty close to what you want, so you just need a quick follow up prompt to the original response?
- languid-photic 6mo agoYes! It depends on the extent of changes needed. If the changes needed are small, I'll apply the best implementation as a foundation and then just iterate directly. If the changes needed are drastic, it usually signals that there was sth wrong/ambiguous/etc in the spec (or the ensemble was too weak, which is rarely the case). In cases like this, I improve the spec and then rerun. If it's in the middle, I'll usually apply the best and write a follow on spec.
- digdugdirk 6mo agoHow does that get integrated into the scoring system? I'm imagining a scenario where a cheaper model may get close, but only needs a small follow up to get the desired result. How would this score in comparison to a larger model that got it right the first time - even if it may have been much more expensive overall?
- tyuerd 6mo ago[dead]
- gertlabs 6mo agoWe observed slightly smaller outputs over long horizon agentic coding for GPT 5.5, at a significant improvement in overall response scores. For one-shot coding responses, GPT 5.5 was actually more verbose than GPT 5.4, but again, the responses were significantly stronger. The expected cost increases reported by OpenRouter seem reasonably accurate (perhaps a bit optimistic), but in my opinion, highly worth it. GPT 5.5 has a pretty wide lead on the #2 model for understanding complex scenarios. Rankings at https://gertlabs.com/rankings?mode=agentic_coding https://gertlabs.com/rankings?mode=agentic_coding. See the efficiency chart at the bottom.
- degutemesgen 6mo agoI do think recent models are too expensive to be used for customer-facing agentic workflows.
- boh 6mo agoNew model releases are now like new iPhones--mostly imperceivable improvements with a higher price tag. That's one of the major benefits to open source: you can "freeze" what model you're using. Often it's the model that you know that wins over the one that is different enough that you have to start from scratch with every major update. Most businesses require cost control and predictability over a cutting edge with limited evidence of profitable output outside of tech.
- 32dsfa 6mo agoReally bad comparison. If I skip 2 models of iPhone upgrades, there is definitely a difference between how the thing feels - and it feels its worth the money. If I skip 2 models of upgrades of the frontier models now, I highly doubt I can discern what the difference is and what exactly I'd be paying more for.
- binugeorge 6mo ago[flagged]
- foresightlab 6mo ago[dead]
- DeathArrow 6mo agoIn terms of work done per dollar, new models from OpenAI and Anthropic are worse than the older models. They are trying to squeeze the customers. For personal use I switched to coding plans containing GLM 5.1, Kimi K2.6 and Xiaomi MiMo V2.5 Pro and I never been happier. I said goodbye to both Claude Max and Cursor.
- spprashant 6mo agoI feel very lost in these threads. A lot of people talk about getting bad results from gpt 5.5 xH or Opus 4.7 xH. And here I am daily driving Sonnet 4.6 with medium or high thinking. I actually am thoroughly satisfied with the work it does. Perhaps it has to do with the bite sized pieces of work I give it, that fits better with my workflow.