8 ms·

So at this point OpenAI has 6 reasoning models, 4 flagship chat models, and 7 cost optimized models. So that's 17 models in total and that's not even counting t

So at this point OpenAI has 6 reasoning models, 4 flagship chat models, and 7 cost optimized models. So that's 17 models in total and that's not even counting their older models and more specialized ones. Compare this with Anthropic that has 7 models in total and 2 main ones that they promote. This is just getting to be a bit much, seems like they are trying to cover for the fact that they haven't actually done much. All these models feel like they took the exact same base model, tweaked a few things and released it as an entirely new model rather than updating the existing ones. In fact based on some of the other comments here it sounds like these are just updates to their existing model, but they release them as new models to create more media buzz.

amarcheschi 2y ago
The old Chinese strategy of having 7343 different phone models with almost the same specs to confuse the customer better
- MoonGhost 2y ago
  not only that. filling search lists on eBay with your products is old sellers' tactics. Try to search for used Dell workstation or server and you will see pages and pages from the same seller.
- kylehotchkiss 2y ago
  This sounds like recent Dell and Lenovo strategies
  - whalesalad 2y ago
    recent? they've been doing this for decades. person a: "I just got an new macbook pro!" person b: "Nice! I just got a Lenovo YogaPilates Flipfold XR 3299 T92 Thinkbookpad model number SRE44939293X3321" ... person a: "does that have oled?" person b: "Lol no silly that is model SRE44939293XB3321". Notice the B in the middle?!?! That is for OLED.
    - anizan 2y ago
      They should launch o999 and count backwards for each release till they hit oagi
- deleted 2y ago
  [deleted]
deleted 2y ago
[deleted]
kristofferR 2y ago
To use that criticism for this release ain't really fair, as these will replace the old models (o3 will replace o1, o4-mini will replace o3-mini). On a more general level - sure, but they aren't planning to use this release to add a larger number of models, it's just that deprecating/killing the old models can't be done overnight.
- drcongo 2y ago
  As someone who doesn't use anything OpenAI (for all the reasons), I have to agree with the GP. It's all baffling. Why is there an o3-mini and an o4-mini? Why on earth are there so many models? Once you get to this point you're putting the paradox of choice on the user - I used to use a particular brand toothpaste for years until it got to the point where I'd be in the supermarket looking at a wall of toothpaste all by the same brand with no discernible difference between the products. Why is one of them called "whitening"? Do the others not do that? Why is this one called "complete" and that one called "complete ultra"? That would suggest that the "complete" one wasn't actually complete. I stopped using that brand of toothpaste as it become impossible to know which was the right product within the brand. If I was assessing the AI landscape today, where the leading models are largely indistinguishable in day to day use, I'd look at OpenAI's wall of toothpaste and immediately discount them.
  - mkozlows 2y ago
    They keep a lot of models around for backward compatibility for API users. This is confusing, but not inherently a bad idea.
  - petesergeant 2y ago
    > Why is there an o3-mini and an o4-mini? Why on earth are there so many models? Because if they removed access to o3-mini — which I have tested, costed, and built around — I would be very angry. I will probably switch to o4-mini when the time is right.
    - TuxSH 2y ago
      They just did that, at least for chat
leesec 2y ago
"haven't actually done much" being popularizing the chat llm and absolutely dwarfing the competition in paid usage
- amarcheschi 2y ago
  I guess it was related to the last period, rather than the full picture
  - flkenosad 2y ago
    What are people expecting here honestly? This thread is ridiculous.
- iLoveOncall 2y ago
  ChatGPT was released in 2022, so OP's point stands perfectly well.
  - echelon 2y ago
    They're rumored to be working on a social network to rival X with the focus being on image generations. https://techcrunch.com/2025/04/15/openai-is-reportedly-developing-its-own-x-like-social-media-platform/ https://techcrunch.com/2025/04/15/openai-is-reportedly-devel... The play now seems to be less AGI, more "too big to fail" / use all the capital to morph into a FAANG bigtech. My bet is that they'll develop a suite of office tools that leverage their model, chat/communication tools, a browser, and perhaps a device. They're going to try to turn into Google (with maybe a bit of Apple and Meta) before Google turns into them. Near-term, I don't see late stage investors as recouping their investment. But in time, this may work out well for them. There's a tremendous amount of inefficiency and lack of competition amongst the big tech players. They've been so large that nobody else could effectively challenge them. Now there's a "startup" with enough capital to start eating into big tech's more profitable business lines.
    - refulgentis 2y ago
      I don't know how anyone could look at any of this and say ponderously: it's basically the same as Nov 2022 ChatGPT. Thus strategically they're pivoting to social to become too big to fail.
wilg 2y ago
There are 9 models in the ChatGPT model picker and they have stated that it's their goal to get rid of the model picker because everyone finds it annoying.
shmatt 2y ago
Im old enough to remember the mystery and hype before o*/o1/strawberry that was supposed to be essentially AGI. We had serious news outlets write about senior people at OpenAI quitting because o1 was SkyNet Now we're up to o4, AGI is still not even in near site (depending on your definition, I know). And OpenAI is up to about 5000 employees. I'd think even before AGI a new model would be able to cover for at least 4500 of those employees being fired, is that not the case?
- pants2 2y ago
  Remember that Docusign has 7,000 employees. I think OpenAI is pretty lean for what they're accomplishing.
  - throwanem 2y ago
    [flagged]
    - ein0p 2y ago
      The closest Elon ever came to anything Hague-worthy is allowing Starlink to be used in Ukrainian attacks on Russian civilian infrastructure. I don't think the Hague would be interested in anything like that. And if his life is worthless, then what would you say about your own? Nonetheless, I commend you on your complete lack of hinges. /s
      - throwanem 2y ago
        Oh, I'm thinking more in the sense of the special one-off kinds of trials, the sort Gustave Gilbert so ably observed. The venue is a matter of convenience, nothing more. To the rest I would say the worth of my life is no more mine to judge than anyone else is competent to do the same for themselves, or indeed other than foolish to pursue the attempt.
    - andrewinardeer 2y ago
      The US is not a signatory to the International Criminal Court so you won't see Musk on trial there.
whalesalad 2y ago
Model fatigue is a real thing - Particularly with their billing model that is wildly different from model to model and gives you more headroom as you spend more. We spend a lot of time and effort running tests across many models to balance for that cost/performance ratio. When you can run 300k tokens per min on a shittier model, or 10k tokens per min on a better model - you want to use the cheaper model but if the performance isn't there then you gotta pivot. Can I use tools here? Can I use function calling here? Do I use the chat API, the chat completions API, or the responses API? Do either of those work with the model I want to use, or only with other models? I almost wonder if this is intentional ... because when you create a quagmire of insane inter-dependent billing scenarios you end up with a product like AWS that can generate substantial amounts of revenue from sheer ignorance or confusion. Then you can hire special consultants to come in and offer solutions to your customers in order to wade through the muck on your behalf. Dealing with OpenAI's API's is a straight up nightmare.
crowcroft 2y ago
Most industries, or categories go through cycles of fragmentation and consolidation. AI is currently in a high growth expansion phase. The leads to rapid iteration and fragmentation because getting things released is the most important thing. When the models start to plateau or the demands on the industry are for profit you will see consolidation start.
- airstrike 2y ago
  having many models from the same company in some haphazard strategy doesn't equate to "industry fragmentation". it's just confusion
  - crowcroft 2y ago
    OpenAI's continued growth and press coverage relative to their peers leads to me to believe it isn't *just* confusion, even if it is confusing.
    - airstrike 2y ago
      I'd attribute that more to first mover advantage than a benefit from poor naming choices, though I do think they are likely to misattribute that to a causal relationship so that they keep doing the latter
paxys 2y ago
OpenAI isn't selling GPT-4 or o1 or o4-mini or turbo or whatever else to the general public. These announcements may as well be them releasing GPT v12.582.599385. No one outside of a small group of nerds cares. The end consumer is going to chatgpt.com and typing things in the box.
- astrange 2y ago
  They have an enterprise business too. I think it's relevant for that.
  - M4v3R 2y ago
    And that’s exactly why their model naming and release process looks like this right now.
- MoonGhost 2y ago
  $20 Plus subscription give access to o1 and Deep Research (10 uses/month). I'm pretty sure general public can get access through API as well.
  - swat535 2y ago
    Right and most people are not going to spend 200$+ /mo on ChatGPT.. Maybe businesses will but at this point they have too many choices.
resters 2y ago
They do this because people like to have predictability. A new model may behave quite differently on something that’s important for a use case. Also, there are a lot of cases where very small models are just fine and others where they are not. It would always make sense to have the smallest highest performing models available.
- t-writescode 2y ago
  I have *no idea* why you're being downvoted on this. If I want to take advantage of a new model, I must validate that the structured queries I've made to the older models still work on the new models. The last time I did a validation and update. Their Responses. Had. Changed. API users need dependability, which means they need older models to keep being usable.
  - resters 2y ago
    > I have no idea why you're being downvoted on this. I probably offended someone at YC and my account is being punished.
irthomasthomas 2y ago
That would explain why they all have a knowledge cutoff (likely training date) of ~August 2023.
onlyrealcuzzo 2y ago
> All these models feel like they took the exact same base model, tweaked a few things and released it as an entirely new model rather than updating the existing ones. That's not a problem in and of itself. It's only a problem if the models aren't good enough. Judging by ChatGPT's adoption, people seem to think they're doing just fine.
w10-1 2y ago
> This is just getting to be a bit much, seems like they are > trying to cover for the fact that they haven't actually done much Or perhaps they're trying to make some important customers happy by showing movement on areas the customers care about. Subjectively, customers get locked in by feeling they have the inside track, and these small tweaks prove that. Objectively, the small change might make a real difference to the customer's use case. Similarly, it's important to force development teams to actually ship, and shipping more frequently reduces risk, so this could reflect internal discipline. As for media buzz, OpenAI is probably trying to tamp that down; they have plenty of first-mover advantage. More puffery just makes their competitors seem more important, and the risk to their reputation of a flop is a lot larger than the reward of the next increment. As for "a bit much", before 2023 I was thinking I could meaningfully track progress and trade-off's in selecting tech, but now the cat is not only out of the bag, it's had more litters than I can count. So, yeah - a bit much!
- sksxihve 2y ago
  > Or perhaps they're trying to make some important customers happy by showing movement on areas the customers care about Or make important investors happy, they need to justify the latest $40 billion round
vunderba 2y ago
> This is just getting to be a bit much, seems like they are trying to cover for the fact that they haven't actually done much. Did you miss the 4o image generation announcement from roughly three week ago? https://news.ycombinator.com/item?id=43474112 https://news.ycombinator.com/item?id=43474112 Combining a multimodal LLM+ImageGen puts them pretty significantly ahead of the curve at least in that domain. Demonstration of the capabilities: https://mordenstar.com/blog/chatgpt-4o-images https://mordenstar.com/blog/chatgpt-4o-images
mrcwinn 2y ago
Well, in fairness, Anthropic has less because 1) they started later, 2) could learn from competitors' mistakes, 3) focused on enterprise and not consumer, 4) have fewer resources. The point is taken — and OpenAI agrees. They have said they are actively working on simplifying the offering. I just think it's a bit unfair. We have perfect hindsight today here on HackerNews and also did zero of the work to produce the product.
jstummbillig 2y ago
I can not believe that we feel that this is what's most worth talking about here (by visibility). At this point I truly wonder if AI is what will make HN side with the luddites.
- flkenosad 2y ago
  It's giving "they took our jerbs"
- siva7 2y ago
  Is there some new HN with more insightful discussions?
nopinsight 2y ago
Research by METR suggests that frontier LLMs can perform software tasks over exponentially longer time horizon required for human engineers, with ~7-month for each doubling. o3 is above the trend line. https://x.com/METR_Evals/status/1912594122176958939 https://x.com/METR_Evals/status/1912594122176958939 —- The AlexNet paper which kickstarted the deep learning era in 2012 was ahead of the 2nd-best entry by 11%. Many published AI papers then advanced SOTA by just a couple percentage points. o3 high is about 9% ahead of o1 high on livebench.ai and there are also quite a few testimonials of their differences. Yes, AlexNet made major strides in other aspects as well but it’s been just 7 months since o1-preview, the first publicly available reasoning model, which is a seminal advance beyond previous LLMs. It seems some people have become desensitized to how rapidly things are moving in AI, despite its largely unprecedented pace of progress. Ref: - https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf https://proceedings.neurips.cc/paper_files/paper/2012/file/c... - https://livebench.ai/#/ https://livebench.ai/#/
- kadushka 2y ago
  Imagenet had improved the error rate by 100*11/25=44%. o1 to o3 error rate went from 28 to 19, so 100*9/28=32%. But these are meaningless comparisons because it’s typically harder to improve already good results.
ren_engineer 2y ago
you'd think they could use AI to interpret the best model for your use case so you don't even have to think about it. Run the first few API calls in parallel, grade the result, and then send the rest to whatever works best
jasondigitized 2y ago
If there are incremental gains in each release, why would they hold them back? The amount of exhaust coming off of each release is gold for the internal teams. The naming convention is bad, and the CPO just admitted as much on Lenny's podcast, but I am not sure why incremental releases is a bad thing.
danielmarkbruce 2y ago
Think for 30 seconds about why they might in good faith do what they do. Do you use any of them? Are you a developer? Just because a model is non-deterministic it doesn't mean developers don't want some level of consistency, whether it be about capabilities, cost, latency, call structure etc.
kgeist 2y ago
> This is just getting to be a bit much, seems like they are trying to cover for the fact that they haven't actually done much. All these models feel like they took the exact same base model, tweaked a few things and released it as an entirely new model OpenAI's progress lately: 2024 December - first reasoning model (official release) 2025 February - deep search 2025 March - true multi-modal image generation 2025 April - reasoning model with tools I'm not sure why people say they haven't done much. We couldn't even dream of stuff like this five years ago, and now releasing groundbreaking/novel features every month is considered "meh"... I think we're spoiled and can't appreciate anything anymore :)
Seattle3503 2y ago
This seems like a perfect use case for "agentic" AI. OpenAI can enrich the context window with the strengths and weakness of each model, and when a user prompts for something the model can say "Hey, I'm gonna switch to another model that is better at answering this sort of question." and the user can accept or reject.

amarcheschi 2y ago

The old Chinese strategy of having 7343 different phone models with almost the same specs to confuse the customer better

MoonGhost 2y ago
not only that. filling search lists on eBay with your products is old sellers' tactics. Try to search for used Dell workstation or server and you will see pages and pages from the same seller.
kylehotchkiss 2y ago
This sounds like recent Dell and Lenovo strategies
- whalesalad 2y ago
  recent? they've been doing this for decades. person a: "I just got an new macbook pro!" person b: "Nice! I just got a Lenovo YogaPilates Flipfold XR 3299 T92 Thinkbookpad model number SRE44939293X3321" ... person a: "does that have oled?" person b: "Lol no silly that is model SRE44939293XB3321". Notice the B in the middle?!?! That is for OLED.
  - anizan 2y ago
    They should launch o999 and count backwards for each release till they hit oagi
deleted 2y ago
[deleted]

deleted 2y ago

[deleted]

kristofferR 2y ago

To use that criticism for this release ain't really fair, as these will replace the old models (o3 will replace o1, o4-mini will replace o3-mini). On a more general level - sure, but they aren't planning to use this release to add a larger number of models, it's just that deprecating/killing the old models can't be done overnight.

drcongo 2y ago
As someone who doesn't use anything OpenAI (for all the reasons), I have to agree with the GP. It's all baffling. Why is there an o3-mini and an o4-mini? Why on earth are there so many models? Once you get to this point you're putting the paradox of choice on the user - I used to use a particular brand toothpaste for years until it got to the point where I'd be in the supermarket looking at a wall of toothpaste all by the same brand with no discernible difference between the products. Why is one of them called "whitening"? Do the others not do that? Why is this one called "complete" and that one called "complete ultra"? That would suggest that the "complete" one wasn't actually complete. I stopped using that brand of toothpaste as it become impossible to know which was the right product within the brand. If I was assessing the AI landscape today, where the leading models are largely indistinguishable in day to day use, I'd look at OpenAI's wall of toothpaste and immediately discount them.
- mkozlows 2y ago
  They keep a lot of models around for backward compatibility for API users. This is confusing, but not inherently a bad idea.
- petesergeant 2y ago
  > Why is there an o3-mini and an o4-mini? Why on earth are there so many models? Because if they removed access to o3-mini — which I have tested, costed, and built around — I would be very angry. I will probably switch to o4-mini when the time is right.
  - TuxSH 2y ago
    They just did that, at least for chat

leesec 2y ago

"haven't actually done much" being popularizing the chat llm and absolutely dwarfing the competition in paid usage

amarcheschi 2y ago
I guess it was related to the last period, rather than the full picture
- flkenosad 2y ago
  What are people expecting here honestly? This thread is ridiculous.
iLoveOncall 2y ago
ChatGPT was released in 2022, so OP's point stands perfectly well.
- echelon 2y ago
  They're rumored to be working on a social network to rival X with the focus being on image generations. https://techcrunch.com/2025/04/15/openai-is-reportedly-developing-its-own-x-like-social-media-platform/ https://techcrunch.com/2025/04/15/openai-is-reportedly-devel... The play now seems to be less AGI, more "too big to fail" / use all the capital to morph into a FAANG bigtech. My bet is that they'll develop a suite of office tools that leverage their model, chat/communication tools, a browser, and perhaps a device. They're going to try to turn into Google (with maybe a bit of Apple and Meta) before Google turns into them. Near-term, I don't see late stage investors as recouping their investment. But in time, this may work out well for them. There's a tremendous amount of inefficiency and lack of competition amongst the big tech players. They've been so large that nobody else could effectively challenge them. Now there's a "startup" with enough capital to start eating into big tech's more profitable business lines.
  - refulgentis 2y ago
    I don't know how anyone could look at any of this and say ponderously: it's basically the same as Nov 2022 ChatGPT. Thus strategically they're pivoting to social to become too big to fail.

wilg 2y ago

There are 9 models in the ChatGPT model picker and they have stated that it's their goal to get rid of the model picker because everyone finds it annoying.

shmatt 2y ago

Im old enough to remember the mystery and hype before o*/o1/strawberry that was supposed to be essentially AGI. We had serious news outlets write about senior people at OpenAI quitting because o1 was SkyNet Now we're up to o4, AGI is still not even in near site (depending on your definition, I know). And OpenAI is up to about 5000 employees. I'd think even before AGI a new model would be able to cover for at least 4500 of those employees being fired, is that not the case?

pants2 2y ago
Remember that Docusign has 7,000 employees. I think OpenAI is pretty lean for what they're accomplishing.
- throwanem 2y ago
  [flagged]
  - ein0p 2y ago
    The closest Elon ever came to anything Hague-worthy is allowing Starlink to be used in Ukrainian attacks on Russian civilian infrastructure. I don't think the Hague would be interested in anything like that. And if his life is worthless, then what would you say about your own? Nonetheless, I commend you on your complete lack of hinges. /s
    - throwanem 2y ago
      Oh, I'm thinking more in the sense of the special one-off kinds of trials, the sort Gustave Gilbert so ably observed. The venue is a matter of convenience, nothing more. To the rest I would say the worth of my life is no more mine to judge than anyone else is competent to do the same for themselves, or indeed other than foolish to pursue the attempt.
  - andrewinardeer 2y ago
    The US is not a signatory to the International Criminal Court so you won't see Musk on trial there.

whalesalad 2y ago

Model fatigue is a real thing - Particularly with their billing model that is wildly different from model to model and gives you more headroom as you spend more. We spend a lot of time and effort running tests across many models to balance for that cost/performance ratio. When you can run 300k tokens per min on a shittier model, or 10k tokens per min on a better model - you want to use the cheaper model but if the performance isn't there then you gotta pivot. Can I use tools here? Can I use function calling here? Do I use the chat API, the chat completions API, or the responses API? Do either of those work with the model I want to use, or only with other models? I almost wonder if this is intentional ... because when you create a quagmire of insane inter-dependent billing scenarios you end up with a product like AWS that can generate substantial amounts of revenue from sheer ignorance or confusion. Then you can hire special consultants to come in and offer solutions to your customers in order to wade through the muck on your behalf. Dealing with OpenAI's API's is a straight up nightmare.

crowcroft 2y ago

Most industries, or categories go through cycles of fragmentation and consolidation. AI is currently in a high growth expansion phase. The leads to rapid iteration and fragmentation because getting things released is the most important thing. When the models start to plateau or the demands on the industry are for profit you will see consolidation start.

airstrike 2y ago
having many models from the same company in some haphazard strategy doesn't equate to "industry fragmentation". it's just confusion
- crowcroft 2y ago
  OpenAI's continued growth and press coverage relative to their peers leads to me to believe it isn't *just* confusion, even if it is confusing.
  - airstrike 2y ago
    I'd attribute that more to first mover advantage than a benefit from poor naming choices, though I do think they are likely to misattribute that to a causal relationship so that they keep doing the latter

paxys 2y ago

OpenAI isn't selling GPT-4 or o1 or o4-mini or turbo or whatever else to the general public. These announcements may as well be them releasing GPT v12.582.599385. No one outside of a small group of nerds cares. The end consumer is going to chatgpt.com and typing things in the box.

astrange 2y ago
They have an enterprise business too. I think it's relevant for that.
- M4v3R 2y ago
  And that’s exactly why their model naming and release process looks like this right now.
MoonGhost 2y ago
$20 Plus subscription give access to o1 and Deep Research (10 uses/month). I'm pretty sure general public can get access through API as well.
- swat535 2y ago
  Right and most people are not going to spend 200$+ /mo on ChatGPT.. Maybe businesses will but at this point they have too many choices.

resters 2y ago

They do this because people like to have predictability. A new model may behave quite differently on something that’s important for a use case. Also, there are a lot of cases where very small models are just fine and others where they are not. It would always make sense to have the smallest highest performing models available.

t-writescode 2y ago
I have *no idea* why you're being downvoted on this. If I want to take advantage of a new model, I must validate that the structured queries I've made to the older models still work on the new models. The last time I did a validation and update. Their Responses. Had. Changed. API users need dependability, which means they need older models to keep being usable.
- resters 2y ago
  > I have no idea why you're being downvoted on this. I probably offended someone at YC and my account is being punished.

irthomasthomas 2y ago

That would explain why they all have a knowledge cutoff (likely training date) of ~August 2023.

onlyrealcuzzo 2y ago

> All these models feel like they took the exact same base model, tweaked a few things and released it as an entirely new model rather than updating the existing ones. That's not a problem in and of itself. It's only a problem if the models aren't good enough. Judging by ChatGPT's adoption, people seem to think they're doing just fine.

w10-1 2y ago

> This is just getting to be a bit much, seems like they are > trying to cover for the fact that they haven't actually done much Or perhaps they're trying to make some important customers happy by showing movement on areas the customers care about. Subjectively, customers get locked in by feeling they have the inside track, and these small tweaks prove that. Objectively, the small change might make a real difference to the customer's use case. Similarly, it's important to force development teams to actually ship, and shipping more frequently reduces risk, so this could reflect internal discipline. As for media buzz, OpenAI is probably trying to tamp that down; they have plenty of first-mover advantage. More puffery just makes their competitors seem more important, and the risk to their reputation of a flop is a lot larger than the reward of the next increment. As for "a bit much", before 2023 I was thinking I could meaningfully track progress and trade-off's in selecting tech, but now the cat is not only out of the bag, it's had more litters than I can count. So, yeah - a bit much!

sksxihve 2y ago
> Or perhaps they're trying to make some important customers happy by showing movement on areas the customers care about Or make important investors happy, they need to justify the latest $40 billion round

vunderba 2y ago

> This is just getting to be a bit much, seems like they are trying to cover for the fact that they haven't actually done much. Did you miss the 4o image generation announcement from roughly three week ago? https://news.ycombinator.com/item?id=43474112 https://news.ycombinator.com/item?id=43474112 Combining a multimodal LLM+ImageGen puts them pretty significantly ahead of the curve at least in that domain. Demonstration of the capabilities: https://mordenstar.com/blog/chatgpt-4o-images https://mordenstar.com/blog/chatgpt-4o-images

mrcwinn 2y ago

Well, in fairness, Anthropic has less because 1) they started later, 2) could learn from competitors' mistakes, 3) focused on enterprise and not consumer, 4) have fewer resources. The point is taken — and OpenAI agrees. They have said they are actively working on simplifying the offering. I just think it's a bit unfair. We have perfect hindsight today here on HackerNews and also did zero of the work to produce the product.

jstummbillig 2y ago

I can not believe that we feel that this is what's most worth talking about here (by visibility). At this point I truly wonder if AI is what will make HN side with the luddites.

flkenosad 2y ago
It's giving "they took our jerbs"
siva7 2y ago
Is there some new HN with more insightful discussions?

nopinsight 2y ago

Research by METR suggests that frontier LLMs can perform software tasks over exponentially longer time horizon required for human engineers, with ~7-month for each doubling. o3 is above the trend line. https://x.com/METR_Evals/status/1912594122176958939 https://x.com/METR_Evals/status/1912594122176958939 —- The AlexNet paper which kickstarted the deep learning era in 2012 was ahead of the 2nd-best entry by 11%. Many published AI papers then advanced SOTA by just a couple percentage points. o3 high is about 9% ahead of o1 high on livebench.ai and there are also quite a few testimonials of their differences. Yes, AlexNet made major strides in other aspects as well but it’s been just 7 months since o1-preview, the first publicly available reasoning model, which is a seminal advance beyond previous LLMs. It seems some people have become desensitized to how rapidly things are moving in AI, despite its largely unprecedented pace of progress. Ref: - https://proceedings.neurips.cc/paper_files/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf https://proceedings.neurips.cc/paper_files/paper/2012/file/c... - https://livebench.ai/#/ https://livebench.ai/#/

kadushka 2y ago
Imagenet had improved the error rate by 100*11/25=44%. o1 to o3 error rate went from 28 to 19, so 100*9/28=32%. But these are meaningless comparisons because it’s typically harder to improve already good results.

ren_engineer 2y ago

you'd think they could use AI to interpret the best model for your use case so you don't even have to think about it. Run the first few API calls in parallel, grade the result, and then send the rest to whatever works best

jasondigitized 2y ago

If there are incremental gains in each release, why would they hold them back? The amount of exhaust coming off of each release is gold for the internal teams. The naming convention is bad, and the CPO just admitted as much on Lenny's podcast, but I am not sure why incremental releases is a bad thing.

danielmarkbruce 2y ago

Think for 30 seconds about why they might in good faith do what they do. Do you use any of them? Are you a developer? Just because a model is non-deterministic it doesn't mean developers don't want some level of consistency, whether it be about capabilities, cost, latency, call structure etc.

kgeist 2y ago

> This is just getting to be a bit much, seems like they are trying to cover for the fact that they haven't actually done much. All these models feel like they took the exact same base model, tweaked a few things and released it as an entirely new model OpenAI's progress lately: 2024 December - first reasoning model (official release) 2025 February - deep search 2025 March - true multi-modal image generation 2025 April - reasoning model with tools I'm not sure why people say they haven't done much. We couldn't even dream of stuff like this five years ago, and now releasing groundbreaking/novel features every month is considered "meh"... I think we're spoiled and can't appreciate anything anymore :)

Seattle3503 2y ago

This seems like a perfect use case for "agentic" AI. OpenAI can enrich the context window with the strengths and weakness of each model, and when a user prompts for something the model can say "Hey, I'm gonna switch to another model that is better at answering this sort of question." and the user can accept or reject.