3 ms·

I'm suspect on how much of a coding advance it will be. Seems odd that their announcement has zero coding benchmarks, with the closest related thing being term

I'm suspect on how much of a coding advance it will be. Seems odd that their announcement has zero coding benchmarks, with the closest related thing being terminal bench.

artursapek 4mo ago
They claim extreme performance on ExploitBench, which Mythos was touted as being incredible at. https://x.com/OpenAI/status/2070555278576439306 https://x.com/OpenAI/status/2070555278576439306
- andriy_koval 4mo ago
  On graph, they are still slightly bellow Mythos. Maybe enough to not be prohibited by US government?
- HarHarVeryFunny 4mo ago
  My guess is that it's same base model as 5.5, but with additional post-training to improve and benchmaxx on a few things like that. If they really thought it was competitive with Mythos/Fable across the board, then why wouldn't they release a broader set of benchmarks, and why price it day 1 at 1/2 the cost of Fable?
  - famouswaffles 4mo ago
    >and why price it day 1 at 1/2 the cost of Fable? Why would they price it the same as Fable it it doesn't cost the same as Fable ?
    - HarHarVeryFunny 4mo ago
      That's half my point - Anthropic's remarks suggest that is Fable significantly bigger (hence more costly to run) than Opus, so it is priced accordingly, but GPT 5.6 priced the same as 5.5 is one datapoint that suggests they are the same size.
      - famouswaffles 4mo ago
        Yeah...and why would being the same size mean anything ? This is par the course for Open AI. They've always been cheaper and likely smaller than Opus models even when they weren't much if any worse.
        HarHarVeryFunny 4mo ago
        > why would being the same size mean anything Because these companies are still 100% scale-pilled, and each new generation of model is bigger than the previous one, even if active parameters (MoE) is growing slower than total parameters. There is also scaling in inference-time compute which adds to the cost just the same. Fable is rumored to use variable compute which will increase the cost for inputs that get routed for more compute. Maybe as you suggest the OpenAI pricing is unrelated to size/cost, but if so it seems a very aggressive move given that models that cost more to train and run need to generate more revenue to be profitable.
        famouswaffles 4mo ago
        I'm not saying Open AI pricing is entirely unrelated to size/cost. I'm saying why are we assuming that OpenAI is serving say OAI-Opus but at half the price of Anthropic when they could just be serving GPT-5.x which is genuinely near half the cost of Opus at scale. The official API output tokens cost of GLM-5.2 is like a third of Gemini-3.1-Pro. The model is Open weights so we know it's not just a ploy to grab users at the cost of bleeding money. You can actually serve the model profitably at similar prices. They have near a billion consumer users every week. Compute efficiency at scale would be at the forefront of any training effort. It makes a lot more sense to me that they have more compute efficient models (even with the scaling) than Anthropic rather than just serving Opus/Fable at half the costs Anthropic are incurring.
jdw64 4mo ago
Maybe I'll know once I try it? Honestly, for small functions or methods, I don't think there's a huge difference between models. But the larger the code gets, the more noticeable the difference seems to be. Personally, I think this kind of coding experience varies from person to person
- kolinko 4mo ago
  Not the size of function but conplexity.
vanuatu 4mo ago
sadly with all the labs benchmaxxing I feel like you just have to try the model for a while to really evaluate how good it is, especially for each individual use case
hereme888 4mo ago
Tracking model performance on Artificial Analysis makes me think these models are constantly optimized/tuned in some way or another. GPT 5.5 was scoring in the mid 60's when it was first released, now it's almost 10 points higher.
MangoCoffee 4mo ago
>zero coding benchmarks "What gets measured gets managed"

I'm suspect on how much of a coding advance it will be. Seems odd that their announcement has zero coding benchmarks, with the closest related thing being terminal bench.

artursapek 4mo ago

They claim extreme performance on ExploitBench, which Mythos was touted as being incredible at. https://x.com/OpenAI/status/2070555278576439306 https://x.com/OpenAI/status/2070555278576439306

andriy_koval 4mo ago
On graph, they are still slightly bellow Mythos. Maybe enough to not be prohibited by US government?
HarHarVeryFunny 4mo ago
My guess is that it's same base model as 5.5, but with additional post-training to improve and benchmaxx on a few things like that. If they really thought it was competitive with Mythos/Fable across the board, then why wouldn't they release a broader set of benchmarks, and why price it day 1 at 1/2 the cost of Fable?
- famouswaffles 4mo ago
  >and why price it day 1 at 1/2 the cost of Fable? Why would they price it the same as Fable it it doesn't cost the same as Fable ?
  - HarHarVeryFunny 4mo ago
    That's half my point - Anthropic's remarks suggest that is Fable significantly bigger (hence more costly to run) than Opus, so it is priced accordingly, but GPT 5.6 priced the same as 5.5 is one datapoint that suggests they are the same size.
    - famouswaffles 4mo ago
      Yeah...and why would being the same size mean anything ? This is par the course for Open AI. They've always been cheaper and likely smaller than Opus models even when they weren't much if any worse.
      - HarHarVeryFunny 4mo ago
        > why would being the same size mean anything Because these companies are still 100% scale-pilled, and each new generation of model is bigger than the previous one, even if active parameters (MoE) is growing slower than total parameters. There is also scaling in inference-time compute which adds to the cost just the same. Fable is rumored to use variable compute which will increase the cost for inputs that get routed for more compute. Maybe as you suggest the OpenAI pricing is unrelated to size/cost, but if so it seems a very aggressive move given that models that cost more to train and run need to generate more revenue to be profitable.
        famouswaffles 4mo ago
        I'm not saying Open AI pricing is entirely unrelated to size/cost. I'm saying why are we assuming that OpenAI is serving say OAI-Opus but at half the price of Anthropic when they could just be serving GPT-5.x which is genuinely near half the cost of Opus at scale. The official API output tokens cost of GLM-5.2 is like a third of Gemini-3.1-Pro. The model is Open weights so we know it's not just a ploy to grab users at the cost of bleeding money. You can actually serve the model profitably at similar prices. They have near a billion consumer users every week. Compute efficiency at scale would be at the forefront of any training effort. It makes a lot more sense to me that they have more compute efficient models (even with the scaling) than Anthropic rather than just serving Opus/Fable at half the costs Anthropic are incurring.

jdw64 4mo ago

Maybe I'll know once I try it? Honestly, for small functions or methods, I don't think there's a huge difference between models. But the larger the code gets, the more noticeable the difference seems to be. Personally, I think this kind of coding experience varies from person to person

kolinko 4mo ago
Not the size of function but conplexity.

vanuatu 4mo ago

sadly with all the labs benchmaxxing I feel like you just have to try the model for a while to really evaluate how good it is, especially for each individual use case

hereme888 4mo ago

Tracking model performance on Artificial Analysis makes me think these models are constantly optimized/tuned in some way or another. GPT 5.5 was scoring in the mid 60's when it was first released, now it's almost 10 points higher.

MangoCoffee 4mo ago

>zero coding benchmarks "What gets measured gets managed"