6 ms·

Nope, not what we’re doing. o3 is still o3 (no nerfing) and o3-pro is new and better than o3. If we were lying about this, it would be really easy to catch us

Nope, not what we’re doing. o3 is still o3 (no nerfing) and o3-pro is new and better than o3. If we were lying about this, it would be really easy to catch us - just run evals. (I work at OpenAI.)

arresin 1y ago
Not quantized?
- tedsanders 1y ago
  Not quantized. Weights are the same. If we did change the model, we'd release it as a new model with a new name in the API (e.g., o3-turbo-2025-06-10). It would be very annoying to API customers if we ever silently changed models, so we never do this [1]. [1] `chatgpt-4o-latest` being an explicit exception
  - ant6n 1y ago
    It was definitely annoying when o1 disappeared over night, my impression is that was better at some tasks than o3.
  - thegeomaster 1y ago
    Google could at least learn something from this attitude, given their recent 03-25 -> 05-06 model alias switcharoo with 0 notice :)
    - johnb231 1y ago
      That is a preview / beta model with no expectation of stability. Google did nothing wrong there. No one should be using a preview model in production.
      - thegeomaster 1y ago
        Hard disagree. Of course technically they didn't do anything explicitly against the public guidance (the checks and balances would never let them), but naming a model with a date very strongly implies immutability. It's the same logic of why UB in C/C++ isn't a license to do whatever the compiler wants. We're humans and we operate on implications, common-sense assumptions and trust.
        johnb231 1y ago
        The model is labelled as Preview. There are no guarantees of stability or availability for Preview models. Not intended for production workloads. https://cloud.google.com/products?hl=en#product-launch-stages https://cloud.google.com/products?hl=en#product-launch-stage... "At Preview, products or features are ready for testing by customers. Preview offerings are often publicly announced, but are not necessarily feature-complete, and no SLAs or technical support commitments are provided for these. Unless stated otherwise by Google, Preview offerings are intended for use in test environments only. The average Preview stage lasts about six months."
        refulgentis 1y ago
        There hasn't been a non-preview Gemini since...November? The previews are the same as everyone else's release cadance, "preview" is just a magic wand that meant the Launchcal (google's internal signoff tool, i.e. "wave will never happen again) needs less signoffs. Then it got to the point date-pinned models were getting swapped in, in the name of doing us a favor, and it's a...novel idea, we can both agree at the least. I bet someone at Google would be a bit surprised to see someone jumping to legalese to act like this...novelty...is inherently due to the preview status, and based on anything more than a sense that there's no net harm done to us if it costs the same and is better. I'm not sure they're wrong. But it also leads to a sort of "nobody knows how anything works because we have 2^N configs and 5 bits" - for instance, 05-06 was also upgraded to 06-05. Except it wasn't, if you sent variable thinking to 05-06 after upgrade it'd fail. (and don't get me started on the 5 different thinking configurations for Gemini 2.5 flash thinking vs. gemini 05-06 vs. 06-05 and 0 thinking)
  - linsomniac 1y ago
    >we'd release it as a new model with a new name Speaking of a new name. I'll donate the API credits to run a "choose a naming scheme for AI models that isn't confusing AF" for OpenAI.
MattDaEskimo 1y ago
What's with the dropped benchmark performance compared to the original o3 release? It was disappointing to not see o4-mini on it as well
- refulgentis 1y ago
  What dropped benchmark performance?
  - MattDaEskimo 1y ago
    o3 scores noticeably worse on benchmarks compared to its original announcement benchmarks
    - refulgentis 1y ago
      Any link / source / anything? You got quite an opportunity here, OpenAI employee claiming there's no difference and you got something that shows there is.
      - MattDaEskimo 1y ago
        Yes, the original announcement for o3 and o4-mini: https://openai.com/index/introducing-o3-and-o4-mini/ https://openai.com/index/introducing-o3-and-o4-mini/ o3 scored 91.6 on AIME 2024. 83.3 on GPQA o4-mini scored 93.4, 81.4 GPQA Then, the new announcement https://help.openai.com/en/articles/6825453-chatgpt-release-notes#h_6c98aab643 https://help.openai.com/en/articles/6825453-chatgpt-release-... o3 scored 90 on AIME 2024, 81 GPQA o4-mini wasn't measured --- Codeforces is the same, but they have a footnote that they're using a different dataset due to saturation, but still have no grounding model to compare with
        tedsanders 1y ago
        First post measures o3 at high reasoning effort. Second measures at medium reasoning effort. It’s the same model, then and now.
energy123 1y ago
Is it o3 (low), o3 (medium) or o3 (high)? Different model names have crept into the various benchmarks over the last few months.
- tedsanders 1y ago
  o3 is a model, and reasoning effort (high/medium/low) is a parameter that goes into the model. o3 pro is a different thing - it’s not just o3 with maximum remaining effort.
  - fragmede 1y ago
    Could someone there maybe possibly use, oh I dunno, ChatGPT and come up with some better product names?
  - tauntz 1y ago
    Why's it called o3 then if it's a different thing? There's already a rather extreme amount of confusion with the model names and it's not clear _at all_ which model would be "the best" in terms of response quality. Here's the current state with version numbers as far as I can piece it together (using my best guess at naming of each component of the version identifier. Might be totally wrong tho): 1) prefix (optional): "gpt-", "chatgpt-" 2) family (required): o1, o3, o4, 4o, 3.5, 4, 4.1, 4.5, 3) quality? (optional): "nano", "mini", "pro", "turbo" 4) type (optional): "audio", "search" 5) lifecycle (optional): "preview", "latest" 6) date (optional): 2025-04-14, 2024-05-13, 1106, 0613, 0125, etc (I assume the last ones are a date without a year for 2024?) 7) size (optional): "16k" Some final combinations of these version number components are as small as 1 ("o3") or as large as 6 ("gpt-4o-mini-search-preview-2024-12-17"). Given this mess, I can't blame people assuming that the "best" model is the one with the "biggest" number, which would rank the model families as: 4.5 (best) > 4.1 > 4 > 4o > o4 > 3.5 > o3 > o1 (worst).
    - rat9988 1y ago
      o3 and o3-pro aren't the same thing still makes sense though.
csomar 1y ago
I think the parent-parent poster has explained why we can't trust you (and work on OpenAI doesn't help they way you think it does). I didn't read the ToS, like everyone else, but my guess is that degrading model performance at peak times will be one of the things that can slip through. We are not suggesting you are running a different model but that you are quantizing it so that you can support more people. This can't happen with Open weight models where you put the model, allocate the memory and run the thing. With OpenAI/Claude, we don't know the model running, how large it is, what it is running on, etc... None of that is provided and there is only one reason that I can think of: to be able to reduce resources unnoticed.
- rfoo 1y ago
  An (arbitrarily) quantized model is a totally different model, compared to the original.
- Reubachi 1y ago
  I'm not totally sure how you at this point in your online presence associate someone stating their job as a "brag" and not what it really is, providing transparency/disclosure before stating their thoughts. This is HN and not reddit. "I didn't read the ToS, like everyone else, but my guess..." Ah, there it is.
fastball 1y ago
Anecdotal, but about a week ago I noticed a sharp drop in o3 performance. For many tasks I will compare Gemini 2.5 Pro with o3, running the same prompt in both. Generally for my personal use o3 and G2.5P have been neck-and neck over the last months, with responses I have been very happy with. However starting from a week ago, the o3 responses became noticeably worse, with G2.5P staying about the same (in terms of what I've come to expect from the two models). This alongside the news that you guys have decreased the price of o3 by 80% does really make it feel like you've quantized the model or knee-capped thinking or something. If you say it is wholly unchanged I'll believe you, but not sure how else to explain the (admittedly subjective) performance drop I've experienced.
- IanCal 1y ago
  Are you sure you're using the same models? G2.5P updated almost exactly a week ago.
  - fastball 1y ago
    G2.5P might've updated, but that's not the model I noticed a difference. o3 seemed noticeably dumber in isolation, not just compared to G2.5P. But yes, perhaps the answer is that about a week ago I started asking subconsciously harder questions, and G2.5P handled them better because it had just been improved, while o3 had not so it seemed worse. Or perhaps G2.5P has always had more capacity than o3, and I wasn't asking hard enough questions to notice a difference before.
fny 1y ago
Unrelated: Can you all come up with a better naming scheme for your models? I feel like this is a huge UX miss. o4-mini-high o4-mini o3 o3-pro gpt-4o Oy.
meta_ai_x 1y ago
Just because you work at openAI doesn't mean you know everything about openAI especially as strategic as nerfing models to save costs

Nope, not what we’re doing. o3 is still o3 (no nerfing) and o3-pro is new and better than o3. If we were lying about this, it would be really easy to catch us - just run evals. (I work at OpenAI.)

arresin 1y ago

Not quantized?

tedsanders 1y ago
Not quantized. Weights are the same. If we did change the model, we'd release it as a new model with a new name in the API (e.g., o3-turbo-2025-06-10). It would be very annoying to API customers if we ever silently changed models, so we never do this [1]. [1] `chatgpt-4o-latest` being an explicit exception
- ant6n 1y ago
  It was definitely annoying when o1 disappeared over night, my impression is that was better at some tasks than o3.
- thegeomaster 1y ago
  Google could at least learn something from this attitude, given their recent 03-25 -> 05-06 model alias switcharoo with 0 notice :)
  - johnb231 1y ago
    That is a preview / beta model with no expectation of stability. Google did nothing wrong there. No one should be using a preview model in production.
    - thegeomaster 1y ago
      Hard disagree. Of course technically they didn't do anything explicitly against the public guidance (the checks and balances would never let them), but naming a model with a date very strongly implies immutability. It's the same logic of why UB in C/C++ isn't a license to do whatever the compiler wants. We're humans and we operate on implications, common-sense assumptions and trust.
      - johnb231 1y ago
        The model is labelled as Preview. There are no guarantees of stability or availability for Preview models. Not intended for production workloads. https://cloud.google.com/products?hl=en#product-launch-stages https://cloud.google.com/products?hl=en#product-launch-stage... "At Preview, products or features are ready for testing by customers. Preview offerings are often publicly announced, but are not necessarily feature-complete, and no SLAs or technical support commitments are provided for these. Unless stated otherwise by Google, Preview offerings are intended for use in test environments only. The average Preview stage lasts about six months."
        refulgentis 1y ago
        There hasn't been a non-preview Gemini since...November? The previews are the same as everyone else's release cadance, "preview" is just a magic wand that meant the Launchcal (google's internal signoff tool, i.e. "wave will never happen again) needs less signoffs. Then it got to the point date-pinned models were getting swapped in, in the name of doing us a favor, and it's a...novel idea, we can both agree at the least. I bet someone at Google would be a bit surprised to see someone jumping to legalese to act like this...novelty...is inherently due to the preview status, and based on anything more than a sense that there's no net harm done to us if it costs the same and is better. I'm not sure they're wrong. But it also leads to a sort of "nobody knows how anything works because we have 2^N configs and 5 bits" - for instance, 05-06 was also upgraded to 06-05. Except it wasn't, if you sent variable thinking to 05-06 after upgrade it'd fail. (and don't get me started on the 5 different thinking configurations for Gemini 2.5 flash thinking vs. gemini 05-06 vs. 06-05 and 0 thinking)
- linsomniac 1y ago
  >we'd release it as a new model with a new name Speaking of a new name. I'll donate the API credits to run a "choose a naming scheme for AI models that isn't confusing AF" for OpenAI.

MattDaEskimo 1y ago

What's with the dropped benchmark performance compared to the original o3 release? It was disappointing to not see o4-mini on it as well

refulgentis 1y ago
What dropped benchmark performance?
- MattDaEskimo 1y ago
  o3 scores noticeably worse on benchmarks compared to its original announcement benchmarks
  - refulgentis 1y ago
    Any link / source / anything? You got quite an opportunity here, OpenAI employee claiming there's no difference and you got something that shows there is.
    - MattDaEskimo 1y ago
      Yes, the original announcement for o3 and o4-mini: https://openai.com/index/introducing-o3-and-o4-mini/ https://openai.com/index/introducing-o3-and-o4-mini/ o3 scored 91.6 on AIME 2024. 83.3 on GPQA o4-mini scored 93.4, 81.4 GPQA Then, the new announcement https://help.openai.com/en/articles/6825453-chatgpt-release-notes#h_6c98aab643 https://help.openai.com/en/articles/6825453-chatgpt-release-... o3 scored 90 on AIME 2024, 81 GPQA o4-mini wasn't measured --- Codeforces is the same, but they have a footnote that they're using a different dataset due to saturation, but still have no grounding model to compare with
      - tedsanders 1y ago
        First post measures o3 at high reasoning effort. Second measures at medium reasoning effort. It’s the same model, then and now.

energy123 1y ago

Is it o3 (low), o3 (medium) or o3 (high)? Different model names have crept into the various benchmarks over the last few months.

tedsanders 1y ago
o3 is a model, and reasoning effort (high/medium/low) is a parameter that goes into the model. o3 pro is a different thing - it’s not just o3 with maximum remaining effort.
- fragmede 1y ago
  Could someone there maybe possibly use, oh I dunno, ChatGPT and come up with some better product names?
- tauntz 1y ago
  Why's it called o3 then if it's a different thing? There's already a rather extreme amount of confusion with the model names and it's not clear _at all_ which model would be "the best" in terms of response quality. Here's the current state with version numbers as far as I can piece it together (using my best guess at naming of each component of the version identifier. Might be totally wrong tho): 1) prefix (optional): "gpt-", "chatgpt-" 2) family (required): o1, o3, o4, 4o, 3.5, 4, 4.1, 4.5, 3) quality? (optional): "nano", "mini", "pro", "turbo" 4) type (optional): "audio", "search" 5) lifecycle (optional): "preview", "latest" 6) date (optional): 2025-04-14, 2024-05-13, 1106, 0613, 0125, etc (I assume the last ones are a date without a year for 2024?) 7) size (optional): "16k" Some final combinations of these version number components are as small as 1 ("o3") or as large as 6 ("gpt-4o-mini-search-preview-2024-12-17"). Given this mess, I can't blame people assuming that the "best" model is the one with the "biggest" number, which would rank the model families as: 4.5 (best) > 4.1 > 4 > 4o > o4 > 3.5 > o3 > o1 (worst).
  - rat9988 1y ago
    o3 and o3-pro aren't the same thing still makes sense though.

csomar 1y ago

I think the parent-parent poster has explained why we can't trust you (and work on OpenAI doesn't help they way you think it does). I didn't read the ToS, like everyone else, but my guess is that degrading model performance at peak times will be one of the things that can slip through. We are not suggesting you are running a different model but that you are quantizing it so that you can support more people. This can't happen with Open weight models where you put the model, allocate the memory and run the thing. With OpenAI/Claude, we don't know the model running, how large it is, what it is running on, etc... None of that is provided and there is only one reason that I can think of: to be able to reduce resources unnoticed.

rfoo 1y ago
An (arbitrarily) quantized model is a totally different model, compared to the original.
Reubachi 1y ago
I'm not totally sure how you at this point in your online presence associate someone stating their job as a "brag" and not what it really is, providing transparency/disclosure before stating their thoughts. This is HN and not reddit. "I didn't read the ToS, like everyone else, but my guess..." Ah, there it is.

fastball 1y ago

Anecdotal, but about a week ago I noticed a sharp drop in o3 performance. For many tasks I will compare Gemini 2.5 Pro with o3, running the same prompt in both. Generally for my personal use o3 and G2.5P have been neck-and neck over the last months, with responses I have been very happy with. However starting from a week ago, the o3 responses became noticeably worse, with G2.5P staying about the same (in terms of what I've come to expect from the two models). This alongside the news that you guys have decreased the price of o3 by 80% does really make it feel like you've quantized the model or knee-capped thinking or something. If you say it is wholly unchanged I'll believe you, but not sure how else to explain the (admittedly subjective) performance drop I've experienced.

IanCal 1y ago
Are you sure you're using the same models? G2.5P updated almost exactly a week ago.
- fastball 1y ago
  G2.5P might've updated, but that's not the model I noticed a difference. o3 seemed noticeably dumber in isolation, not just compared to G2.5P. But yes, perhaps the answer is that about a week ago I started asking subconsciously harder questions, and G2.5P handled them better because it had just been improved, while o3 had not so it seemed worse. Or perhaps G2.5P has always had more capacity than o3, and I wasn't asking hard enough questions to notice a difference before.

fny 1y ago

Unrelated: Can you all come up with a better naming scheme for your models? I feel like this is a huge UX miss. o4-mini-high o4-mini o3 o3-pro gpt-4o Oy.

meta_ai_x 1y ago

Just because you work at openAI doesn't mean you know everything about openAI especially as strategic as nerfing models to save costs