10 ms·

GPT-4.5: "Not a frontier model"?

166 points by pama 2y ago

tsunego 2y ago
GPT-4.5 feels like OpenAI's way of discovering just how much we'll pay for diminishing returns. The leap from GPT-4o to 4.5 isn't a leap—it's an expensive tiptoe toward incremental improvements, priced like a luxury item without the luxury payoff. With pricing at 15x GPT-4o, they're practically daring us not to use it. Given this, I wouldn't be surprised if GPT-4.5 quietly disappears from the API once OpenAI finishes squeezing insights (and cash) out of this experiment.
- hooverd 2y ago
  They should have called it "ChatGPT Enterprise".
  - tsunego 2y ago
    Exactly! designed specifically for people who love burning corporate budgets.
    - numba888 2y ago
      OpenAI is going to add it to Plus subscriptions. I.e. available for many at no additional cost. Likely with restrictions line N prompts/hour. As for API price, when it matters businesses and people are willing to pay much more for just a bit better results. OpenAI doesn't take the other options away. So we don't lose anything.
  - fodkodrasz 2y ago
    IMO the 4o output is lot more Enterprise-compatible, the 4.5 being straight to the point and more natural is quite the opposite. Pricing-wise your point stands. Disclaimer: have not tried 4.5 yet, just skimmed through the announcement, using 4o regularly.
- TZubiri 2y ago
  Time to enter the tick cycle. I ask chatgpt to give me a map highlighting all spanish speaking countries, gives me stable diffusion trash. Just gotta do the grunt work, add a tool with a map api. Integrate with google maps for transit stuff. It's a good LLM model already it doesn't need to be einstein and solve aerospatial equations. We just need to wait until they realize their limits and find the humility to build yet another useful product that won't conquer the world.
  - Willingham 2y ago
    I’ve thought of LLM’s as google 2.0 for some time now. Truly a world changing technology similar to how google changed the world, likely to have an even larger impact than google had as we create highly specialized Implementations of the technology in the coming decade…but it’s not energy positive nuclear fusion, or a polynomial time NP solver, it’s just google 2.0
    - dingnuts 2y ago
      Google 2.0 where you have to check every answer it gives you because it's authoritative about nothing. Works great when the output is small enough to unit test or immediately try in situations with no possible negative outcomes. Anything larger? Skip the LLM slop and go to the source. You have to go to the source, anyway.
      - CamperBob2 2y ago
        You have to go to the source, anyway. Yeah, and then check that. I don't get this argument at all. People who uncritically swallow the first answer or two they get from Google have a name... but that would just derail the thread into politics.
        LPisGood 2y ago
        There is something to be said for trusting people’s (or systems of people’s) authority. For example, have you ever personally verified that humans went to the moon? Have you ever done the experiments to prove the Earth is round?
- zamadatix 2y ago
  Even this is a bit overly complicated/optimistic to me. Why not something as simple as: OpenAI has been building larger and larger models to great success for a long time. As a result, they were excited this one was going to be so much larger=so much better that the price to run it would be well worth the huge jump they were planning to get from it. What really happened is this method of scaling hit a wall and they were left with an expensive dud they won't get much out of but they have to release something for now otherwise they start falling well behind on the boards the next few months. Meanwhile they scramble focus to find other means of scaling like "chain of thought + runtime" provided.
  - hn_throwaway_99 2y ago
    Thank you so much for this comment. I don't really understand the need for people to go straight to semi-conspiratorial hypotheses, when the simpler explanation makes so much more sense. All the evidence is that this model is much larger than previous ones, so they must charge a lot more for inference because it costs so much more to run. OpenAI were the OGs when it came to scaling, so it's not surprising they went this route and eventually hit a wall. I don't at all blame OpenAI for going down this path (indeed, I laud them for making expensive bets), but I do blame all the quote-un-quote "thought leaders" who were writing breathless posts about how AGI was just around the corner because things would just scale linearly forever. It was classic "based on historical data, this 10 year old will be 20 feet tall by the time he's 30" thinking, and lots of people called them out on this, and they either just ignored it or responded with "oh, simple not-in-the-know peons" dismissiveness.
    - kruxigt 2y ago
      [dead]
    - bee_rider 2y ago
      It is weird because this is a board for working programmers for the most part. So like, who’s seen a gram conspiracy actually be accomplished? Probably now many. A lackluster product that gets released even though it sucks because too many people are highly motivated not to notice that it sucks? Everybody has experienced that, right?
- Kerbonut 2y ago
  Apparently, OpenAI API “credits” expire after a year. I stupidly put another $20 and trying to blow through them, 4.5 is the easiest way considering recent 4o has fallen out of favor for other models and I don’t want to just let them expire again. An expiry after only one year is asinine.
  - Chance-Device 2y ago
    Yes. I also discovered this, and was also forced to blow through my credits in a rush. Terrible policy.
    - glenstein 2y ago
      I'm learning this for the first time now. I don't appreciate having to anticipate how many credits I'll use like its an FSA account.
    - heed 2y ago
      >Terrible policy. And unfortunately one not exclusive to OpenAI. Anthropic credits also expire after 1 year.
      - gloosx 2y ago
        Not sure how it's with OpenAI, but Anthropic is so money-hungry, they won't even let you remove your debit card data from your account without a week-long support encounter.
  - nickfromseattle 2y ago
    I worked at a company that expired credits after 365 days. My layman understanding is the credits sit as a liability until consumed, and the alternative is having potentially millions accounts with a combined hundreds of millions of dollars in 'liabilities' on their books. More info from ChatGPT here: https://chatgpt.com/share/67c5ad36-2f84-8001-a61f-11d4e1713516 https://chatgpt.com/share/67c5ad36-2f84-8001-a61f-11d4e17135...
- jstummbillig 2y ago
  This is how pricing on human labour works. Nobody expects an employee that costs twice as much to produce twice the output for any given task. All that is expected is that they can do a narrow set of things, that another person can't.
assanineass 2y ago
[dead]
sampton 2y ago
Too much money not enough new ideas.
siva7 2y ago
It's marketed to be slightly better at "creative writing". This isn't the problem most businesses have with current-generation LLMs. On the other side; Anthropic releases nearly at the same time a new model which solves more practical problems for businesses to the point that for coding many insiders don't use OpenAI models for such tasks anymore.
- dingnuts 2y ago
  I think it should be illegal to trick humans into reading "creative" machine output. It strikes me as a form of fraud that steals my most precious resources: time and attention. I read creative writing to feel a human connection to the author. If the author is a machine and this is not disclosed, that's a huge lie. It should be required that publishers label AI generated content.
  - CuriouslyC 2y ago
    I'm pretty sure you read for pleasure, and feeling a human connection is one way that you derive pleasure. If it's the only way that you derive pleasure from reading, my condolences.
    - becquerel 2y ago
      Pretty much where my thoughts on this are. I rarely feel any particular sense of connection to the actual author when I read their books. And I have taken great pleasure from some AI stories (to the degree I put them up on my personal website as a way to keep them around).
  - Philpax 2y ago
    Under the dingnuts regime, Dwarf Fortress will be illegal. Actually, any game with a procedural story? You better believe that's a crime: we can't have a machine generate text a human might enjoy.
    - glenneroo 2y ago
buyucu 2y ago
OpenAI has been irrelevant for a while now. All of the new and exciting developments on AI are coming from other places. ClosedAI is no longer the driver of change and innovation.
- nickthegreek 2y ago
  The other models are literally distilling OpenAI’s models into theirs.
  - demosthanos 2y ago
    So it's been claimed, but has it been proven yet? I'm not even sure what is being alleged there—o1's reasoning tokens are kept secret precisely to avoid the kind of distillation that's being alleged. How can you distill a reasoning process given only the final output?
    - ipaddr 2y ago
      The outputting that they are chatGPT from deepseek is a big clue.
      - TheKGB 2y ago
        [dead]
    - Nevermark 2y ago
      You really can distill a model to a smaller one just by training on the first model's behavior. The first model has a much harder problem. It must figure out a way to iron out the differences between all the different models (i.e. multiple humans, or noise in the dataset) it is trained on, to come up with its single viewpoint of relationships. It must then be tuned to optimally fit its purpose, instead of simply operating like a (composite) rando human. The second model then has these advantages: 1. Trained on a single model, whose behavior comes out of consistent learned relationships, as apposed to many inconsistent models with noise (lots of different humans operating differently at any given time for reasons beyond what the data can show). 2. Trained on post-tuned model behavior directly. One-step training is a considerably simpler problem than two-step. It doesn't learning anything it will need to unlearn or modify. 3. These simplifications both reduce the necessary size of the second model. And with these combined advantages (a simpler more consistent single-system to model, smaller size, and fewer training steps), it can be trained much faster. There, of course, can be disadvantages. The distilled model may not be as versatile, if the data used to prompt the original model isn't as diverse as the first models data. So prompt data for training still matters.
yimby2001 2y ago
It seems like there’s a misunderstanding as why this happened. They’ve been baking this model for months. long before deep seek came out with fundamental new ways of distilling models. and even given that it’s not great it’s its large form, they’re going to distil from this going forward .. so it likely makes sense for them to periodically train these very large models as a basis.
- lhl 2y ago
  I think this framing isn't quite right either. DeepSeek's R1 isn't very different from what OpenAI has already been doing with o1 (and that other groups have been doing as well). As for distilling - the R1 "distilled" models they released aren't even proper (logit) distillations, but just SFTs, not fundamentally new at all. But it's great that they published their full recipes and it's also great to see that it's effective. In fact we've seen now with LIMO, s1/s1.1, that even as few as 1K reasoning traces can get most LLMs to near SOTA math benchmarks. This mirrors the "Alpaca" moment in a lot of ways (and you could even directly mirror say LIMO w/ LIMA). I think the main takeaway of GPT4.5 (Orion) is that it basically gives a perspective to all the "hit a wall" talk from the end of last year. Here we have a model that has been trained on by many accounts 10-100X the compute of GPT4, is likely several times larger in parameter count, but is only... subtly better, certainly not super-intelligent. I've been playing around w/ it a lot the past few days, both with several million tokens worth of non-standard benchmarks and talking to it and it is better than previous GPTs (in particular, it makes a big jump in humor), but I think it's clear that the "easy" gains in the near future are going to be figuring out how as many domains as possible can be approximately verified/RL'd. As for the release? I suppose they could just have kept it internally for distillation/knowledge transfer, so I'm actually happy that they released it, even if it ends up not being a really "useful" model.
modeless 2y ago
I've been using 4.5 instead of 4o for quick questions. I don't mind the slowness for short answers. I feel like it is less likely to hallucinate than other models.
kubb 2y ago
Seems like we're hitting the limits of the technology...
- xmichael909 2y ago
  Yes, I believe the sprint is over, now its doing to be slow cycles maybe 18 months to see a 5% increase an ability and even that 5% increase will be highly subjective. Claude's new release is about the same 3.7 is arguably worse at some things than 3.5 and better at others. Based on the previous pace of release in about 6 months or so - if the next release from any of the leaders is about the same "kinda better kinda worse" then we'll know. Imagine how much money is going to evaporate from the stock market if this is the limit!!!
  - ANewFormation 2y ago
    You can keep getting rich off shovels long after the gold has run dry.
  - deleted 2y ago
    [deleted]
  - borgdefenser 2y ago
    To say 3.7 is worse is completely insane.
- TIPSIO 2y ago
  I also hate waiting on reasoning. I much would prefer a super lightning fast model that is cheaper but the same quality as these frontier models. Let me query these things to death.
  - ljlolel 2y ago
    try groq (hyperfast chips) https://groq.com/ https://groq.com/
- apwell23 2y ago
  does it mean we get a reprieve from "this is just the beginning" comments.
retskrad 2y ago
Sam Altman views Steve Jobs as one of his inspirations (he called the iPhone the greatest product of all time). So if you look at OpenAI in the lens of Apple, where you think about making the product enjoyable to use at all costs, then it makes perfect sense why you’d spend so much money to go from 4o to 4.5 which brings such subtle differences to power users. The vast majority of users, which are over 300 million weekly, will mainly use 4o and whatever is the default. In the future they’ll use 4.5 and think it’s most human like and less robotic.
- bravura 2y ago
  Yes but Steve Jobs also understood the paradox of choice, and the importance of having incredibly clear delineation between every different product in your line.
  - ipaddr 2y ago
    Do models matter to the regular user over brand? People talk about using chatGPT over Google's AI or Deepseek not 4o-mini vs gemini 2. OpenAI has done a good job of making the model less important and the domain gptGPT.com more important. Most of the time the model rarely matters. When you find something incorrect you may switch models but that rarely fixes the problem. Rewording a prompt has more value than changing a model.
    - esafak 2y ago
      If the model did not matter they would be spending their money on marketing or sales instead of improving the model.
      - ipaddr 2y ago
        Spending or saying they are spending is marketing but when people use their product the model doesn't matter.
pzo 2y ago
Long term it might be hard to monetise those infrastructure considering their competition: 1) For coding (API) most probably will stick to Claude 3.5 / 3.7 - big market but still small comparing to all world wide problems 2) For non-coding API IMHO gemini 2.0 flash is the winner - dirty cheap (cheaper than 4o-mini), good enough and even better than gpt-4o, cheap audio and image input. 3) For subscription app ChatGPT is probably still the best but only slightly - they have the best advanced voice audio conversation but Grok will be probably eating their lunch here
- anukin 2y ago
  Sesame model for voice audio imo is better than ChatGPT voice audio conversation. They are going to open source it as well.
  - bckr 2y ago
    Sure but is there an app I can talk to / work with? It seems they're a voice synthesis model company, not a chatbot app / tool company.
  - OsrsNeedsf2P 2y ago
    > They are going to open source it as well. Means nothing until they do
- ipaddr 2y ago
  For the rest of us using free tiers ChatGPT is hands down the winner allowing limited image generation, unlimited usage of some model and limited usage of 4o. Claude is still stuck at 10 messages per day and gemini is less accurate/useful.
  - dingnuts 2y ago
    10 messages a day? How are people "vibe coding" with that?
    - irishloop 2y ago
HarHarVeryFunny 2y ago
GPT 4.5 also has a knowledge cutoff date of 10-2023. https://www.reddit.com/r/singularity/comments/1izpb8t/gpt45_knowledge_cutoff_is_still_october_2023/ https://www.reddit.com/r/singularity/comments/1izpb8t/gpt45_... I'm guessing that this model was finished pre-training at least a year ago (it's been 2 years since GPT 4.0 was released) and they just didn't see the hoped-for performance gains to think it warranted releasing at the time, and so put all their effort into the Q-star/strawberry = eventual O1 reasoning effort instead. It seems that OpenAI's reasoning model lead isn't perhaps what they thought it was, and the recent slew of strong non-reasoning models (Gemini 2.0 Flash, Grok 3, Sonnet 3.7) made them feel the need to release something themselves for appearances sake, so they dusted off this model, perhaps did a bit of post-training on it for EQ, and here we are. The price is a bit of a mystery - perhaps just a reflection of an older model without all the latest efficiency tricks to make it cheaper. Maybe it's dense rather than MoE - who knows.
- sigmoid10 2y ago
  Rumors said that GPT4.5 is an order of magnitude larger. Around 12 trillion parameters total (compared to GPT4's 1.2 trillion). It's almost certainly MoE as well, just a scaled up version. That would explain the cost. OpenAI also said that this is what they originally developed as "Omni" - the model supposed to succeed GPT4 but which fell behind expectations. So they renamed it 4.5 and shoehorned it in to remain in the news among all those competitor releases.
  - ljlolel 2y ago
    the gpt-4o ("omni") is probably a distilled 4.5; hence why not much quality difference
    - sigmoid10 2y ago
      4o has been out since May last year, while omni (now rechristened as 4.5) only finished training in October/November.
      - 2y ago
phillipcarter 2y ago
My take from using it a bit is that they seem to have genuinely innovated on: - Not writing things that go off in weird directions / staying grounded in "reality" - Responding very well to tone preferences and catching nuance in what I say It seems like it's less that it has a great "personality" like Claude, but that it's capable of adapting towards being the "personality" I want and "understanding" what I'm saying in ways that other models haven't been able to do for me.
- XenophileJKO 2y ago
  So this kind of mirrors my feelings after using GPT-4.5 on general conversation and song writing. GPT picked up on unspecified requirements almost instantly. It is subtle (and may be undesirable in some contexts). For example in my songs, I have to bracket the section headings, it picked up on that from my original input. All the other frontier models generally have to be reminded. Additionally, I separately asked for an edit to a music style description. When I asked GPT-4.5 to write a song all by itself, it included a music style description. No other model I have worked with has done this. These are subtle differences, but in aggregate the model just generally needs less nudging to create what is required.
  - deleted 2y ago
    [deleted]
  - torginus 2y ago
    I haven't used 4.5 but have some experience using Claude for creative writing, and in my experience it sometimes has the uncanny ability to get to the core of my ideas, rephrasing my paragraph long descriptions into just a sentence or two, or both improving and concretizing my vague ideas into something that's insightful and tasteful. Other times it locks itself into a dull style and ignores what I ask of it and just produces boring generic garbage, and I have to wrangle it hard to get some of the spark back. I have no idea what's going on inside, but just like with Stable Diffusion, it's fairly easy to make something that has the spark of genius, and is very close to being perfect, but getting the last 10% there, and maintaining the quality seems almost impossible. It's a very weird feeling, it's hard to put into words what is exactly going on, and probably even harder to make it into a benchmark, but it makes me constantly flip-flop between scared of being how good the AI is, and questioning why I ever bothered with using it in the first place, as I would've progress much faster without it.
neom 2y ago
4.5 can extremely quickly distill and work with what I at least consider, complex nuanced thought. 4.5 is night and day better than every other AI for my work, it's quite clever and I like it. Very quick mvp comparison for the show me what you mean crew: https://chatgpt.com/share/67c48fcc-db24-800f-865b-c0485efd7f93 https://chatgpt.com/share/67c48fcc-db24-800f-865b-c0485efd7f... & https://chatgpt.com/share/67c48fe2-0830-800f-a370-7a18586e8b21 https://chatgpt.com/share/67c48fe2-0830-800f-a370-7a18586e8b... (~30 seconds vs ~3 minutes)
- ttul 2y ago
  I believe 4.5 is a very large and rich model. The price is high because it's costly to inference; however, the bigger reason is to ensure that others don't distill from it. Big models have a rich latent space, but it takes time to squeeze the juice out.
  - esafak 2y ago
    That also means people won't use it. Way to shoot yourself in the foot. The irony of a company that has distilled the word's information complaining about another company distilling their model...
    - cscurmudgeon 2y ago
      My assumption: There will be use cases where cost of using this will be smaller than the gain from it. Data from this will make the next version better and cheaper.
    - ttul 2y ago
      The small number of use cases that do pay are providing gross margins as well as feedback that helps OpenAI in various ways. I don’t think it’s a stupid move at all.
- deleted 2y ago
  [deleted]
- nyrikki 2y ago
EcommerceFlow 2y ago
I've found 4.5 to be quite good at "business decisions", much better than other models. It does have some magic to it, similar to Grok 3, but maybe a bit smarter?
sunami-ai 2y ago
Meanwhile all GPT4o models on Azure are set to be deprecated in May and there are no alternative models yet. We should start moving to Anthropic? DS too slow, melting under its own success. Anyone on GPT4o/Azure has any idea when they'll release the next "o" model?
- Uvix 2y ago
  Only an older version of GPT-4o has been deprecated and will be removed in May. The newest version will be supported through at least 20 November 2025. https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/model-retirements#current-models https://learn.microsoft.com/en-us/azure/ai-services/openai/c...
  - sunami-ai 2y ago
    The Nov 2024 release, which is due to be deprecated in Nov 2025, I was told has degraded performance compared to the Aug 2024 release. In fact, OpenAI Models page says their current GPT4o API is serving the Aug release. https://platform.openai.com/docs/models#gpt-4o https://platform.openai.com/docs/models#gpt-4o So I'm still on the Aug 24 release, which, with your reminding me, is not to be deprecated till Aug 2025, but that's less than 5 months from now, and we're skipping the Nov 2024 release just as OpenAI themselves have chosen to do.
ein0p 2y ago
I have access to it. It is better, but not where most techies would care. It knows more, it writes better, it's more pleasant to talk to. I think they might have studied the traffic their hundreds of millions of users generate and realized where they need to improve, then did exactly that for their _non thinking_ model. They understand that a non-thinking model is not going to blow the doors off on coding no matter what they do, but it can do writing and "associative memory" tasks quite well, and having a lot more weights helps there. I also predict that they will fine tune their future distilled, thinking models for coding, based on the same logic, distilling from 4.5 this time. Those models have to be fast, and therefore they have to be smaller.
ghostly_s 2y ago
I don't get it. Aren't these two sentences in the same paragraph contradictory? >"Scaling to this size of model did NOT make a clear jump in capabilities we are measuring." > "The jump from GPT-4o (where we are now) to GPT-4.5 made the models go from great to really great."
- XenophileJKO 2y ago
  No, it means that it got better on things orthogonal to what we have mostly been measuring. On the last few rounds, we have been mostly focusing on reasoning, not as much on knowledge, "creativity", or emotional resonance.
  - johnecheck 2y ago
    "It's better. We can't measure it, but we're pretty sure it's better. We also desperately need it to be better because we just spent a boat-load of money on it."
Artgodma 2y ago
No general model is the frontier. Thousands of small, specific models are infinitely more efficient than a general one. The more narrowed the task - the better algorithms work. That's obvious. Why are general models pushed so hard by its creators? Their enormous valuations are based on total control over user experience. This total control is justified by computational requirements. Users can't run general models locally. Giant data centers for billions are the moat for Model creators and corporations behind.
- azan_ 2y ago
  It's neither obvious nor true, generalist models outperform specialized ones all the time (so frequently that it even has its own name - the bitter lesson)
  - Artgodma 2y ago
    So you think that one general model can outperform thousands of specific ones in their specific areas?
- maleldil 2y ago
  Certain desirable capabilities are available only in bigger models because it takes a certain size for some behaviours emerge.
mvkel 2y ago
I think this release is for the researchers who worked on it and would quit if it never saw daylight
TheKGB 2y ago
[dead]
mirekrusin 2y ago
Is somebody actually looking at those last percentages on benchmarks? Aren't we making mistake of assuming benchmarks are purely 100% correct?

166 points by pama 2y ago

GPT-4.5 feels like OpenAI's way of discovering just how much we'll pay for diminishing returns. The leap from GPT-4o to 4.5 isn't a leap—it's an expensive tiptoe toward incremental improvements, priced like a luxury item without the luxury payoff. With pricing at 15x GPT-4o, they're practically daring us not to use it. Given this, I wouldn't be surprised if GPT-4.5 quietly disappears from the API once OpenAI finishes squeezing insights (and cash) out of this experiment.

hooverd 2y ago
They should have called it "ChatGPT Enterprise".
- tsunego 2y ago
  Exactly! designed specifically for people who love burning corporate budgets.
  - numba888 2y ago
    OpenAI is going to add it to Plus subscriptions. I.e. available for many at no additional cost. Likely with restrictions line N prompts/hour. As for API price, when it matters businesses and people are willing to pay much more for just a bit better results. OpenAI doesn't take the other options away. So we don't lose anything.
- fodkodrasz 2y ago
  IMO the 4o output is lot more Enterprise-compatible, the 4.5 being straight to the point and more natural is quite the opposite. Pricing-wise your point stands. Disclaimer: have not tried 4.5 yet, just skimmed through the announcement, using 4o regularly.
TZubiri 2y ago
Time to enter the tick cycle. I ask chatgpt to give me a map highlighting all spanish speaking countries, gives me stable diffusion trash. Just gotta do the grunt work, add a tool with a map api. Integrate with google maps for transit stuff. It's a good LLM model already it doesn't need to be einstein and solve aerospatial equations. We just need to wait until they realize their limits and find the humility to build yet another useful product that won't conquer the world.
- Willingham 2y ago
  I’ve thought of LLM’s as google 2.0 for some time now. Truly a world changing technology similar to how google changed the world, likely to have an even larger impact than google had as we create highly specialized Implementations of the technology in the coming decade…but it’s not energy positive nuclear fusion, or a polynomial time NP solver, it’s just google 2.0
  - dingnuts 2y ago
    Google 2.0 where you have to check every answer it gives you because it's authoritative about nothing. Works great when the output is small enough to unit test or immediately try in situations with no possible negative outcomes. Anything larger? Skip the LLM slop and go to the source. You have to go to the source, anyway.
    - CamperBob2 2y ago
      You have to go to the source, anyway. Yeah, and then check that. I don't get this argument at all. People who uncritically swallow the first answer or two they get from Google have a name... but that would just derail the thread into politics.
      - LPisGood 2y ago
        There is something to be said for trusting people’s (or systems of people’s) authority. For example, have you ever personally verified that humans went to the moon? Have you ever done the experiments to prove the Earth is round?
zamadatix 2y ago
Even this is a bit overly complicated/optimistic to me. Why not something as simple as: OpenAI has been building larger and larger models to great success for a long time. As a result, they were excited this one was going to be so much larger=so much better that the price to run it would be well worth the huge jump they were planning to get from it. What really happened is this method of scaling hit a wall and they were left with an expensive dud they won't get much out of but they have to release something for now otherwise they start falling well behind on the boards the next few months. Meanwhile they scramble focus to find other means of scaling like "chain of thought + runtime" provided.
- hn_throwaway_99 2y ago
  Thank you so much for this comment. I don't really understand the need for people to go straight to semi-conspiratorial hypotheses, when the simpler explanation makes so much more sense. All the evidence is that this model is much larger than previous ones, so they must charge a lot more for inference because it costs so much more to run. OpenAI were the OGs when it came to scaling, so it's not surprising they went this route and eventually hit a wall. I don't at all blame OpenAI for going down this path (indeed, I laud them for making expensive bets), but I do blame all the quote-un-quote "thought leaders" who were writing breathless posts about how AGI was just around the corner because things would just scale linearly forever. It was classic "based on historical data, this 10 year old will be 20 feet tall by the time he's 30" thinking, and lots of people called them out on this, and they either just ignored it or responded with "oh, simple not-in-the-know peons" dismissiveness.
  - kruxigt 2y ago
    [dead]
  - bee_rider 2y ago
    It is weird because this is a board for working programmers for the most part. So like, who’s seen a gram conspiracy actually be accomplished? Probably now many. A lackluster product that gets released even though it sucks because too many people are highly motivated not to notice that it sucks? Everybody has experienced that, right?
Kerbonut 2y ago
Apparently, OpenAI API “credits” expire after a year. I stupidly put another $20 and trying to blow through them, 4.5 is the easiest way considering recent 4o has fallen out of favor for other models and I don’t want to just let them expire again. An expiry after only one year is asinine.
- Chance-Device 2y ago
  Yes. I also discovered this, and was also forced to blow through my credits in a rush. Terrible policy.
  - glenstein 2y ago
    I'm learning this for the first time now. I don't appreciate having to anticipate how many credits I'll use like its an FSA account.
  - heed 2y ago
    >Terrible policy. And unfortunately one not exclusive to OpenAI. Anthropic credits also expire after 1 year.
    - gloosx 2y ago
      Not sure how it's with OpenAI, but Anthropic is so money-hungry, they won't even let you remove your debit card data from your account without a week-long support encounter.
- nickfromseattle 2y ago
  I worked at a company that expired credits after 365 days. My layman understanding is the credits sit as a liability until consumed, and the alternative is having potentially millions accounts with a combined hundreds of millions of dollars in 'liabilities' on their books. More info from ChatGPT here: https://chatgpt.com/share/67c5ad36-2f84-8001-a61f-11d4e1713516 https://chatgpt.com/share/67c5ad36-2f84-8001-a61f-11d4e17135...
jstummbillig 2y ago
This is how pricing on human labour works. Nobody expects an employee that costs twice as much to produce twice the output for any given task. All that is expected is that they can do a narrow set of things, that another person can't.

assanineass 2y ago

[dead]

sampton 2y ago

Too much money not enough new ideas.

siva7 2y ago

It's marketed to be slightly better at "creative writing". This isn't the problem most businesses have with current-generation LLMs. On the other side; Anthropic releases nearly at the same time a new model which solves more practical problems for businesses to the point that for coding many insiders don't use OpenAI models for such tasks anymore.

dingnuts 2y ago
I think it should be illegal to trick humans into reading "creative" machine output. It strikes me as a form of fraud that steals my most precious resources: time and attention. I read creative writing to feel a human connection to the author. If the author is a machine and this is not disclosed, that's a huge lie. It should be required that publishers label AI generated content.
- CuriouslyC 2y ago
  I'm pretty sure you read for pleasure, and feeling a human connection is one way that you derive pleasure. If it's the only way that you derive pleasure from reading, my condolences.
  - becquerel 2y ago
    Pretty much where my thoughts on this are. I rarely feel any particular sense of connection to the actual author when I read their books. And I have taken great pleasure from some AI stories (to the degree I put them up on my personal website as a way to keep them around).
- Philpax 2y ago
  Under the dingnuts regime, Dwarf Fortress will be illegal. Actually, any game with a procedural story? You better believe that's a crime: we can't have a machine generate text a human might enjoy.
  - glenneroo 2y ago

buyucu 2y ago

OpenAI has been irrelevant for a while now. All of the new and exciting developments on AI are coming from other places. ClosedAI is no longer the driver of change and innovation.

nickthegreek 2y ago
The other models are literally distilling OpenAI’s models into theirs.
- demosthanos 2y ago
  So it's been claimed, but has it been proven yet? I'm not even sure what is being alleged there—o1's reasoning tokens are kept secret precisely to avoid the kind of distillation that's being alleged. How can you distill a reasoning process given only the final output?
  - ipaddr 2y ago
    The outputting that they are chatGPT from deepseek is a big clue.
    - TheKGB 2y ago
      [dead]
  - Nevermark 2y ago
    You really can distill a model to a smaller one just by training on the first model's behavior. The first model has a much harder problem. It must figure out a way to iron out the differences between all the different models (i.e. multiple humans, or noise in the dataset) it is trained on, to come up with its single viewpoint of relationships. It must then be tuned to optimally fit its purpose, instead of simply operating like a (composite) rando human. The second model then has these advantages: 1. Trained on a single model, whose behavior comes out of consistent learned relationships, as apposed to many inconsistent models with noise (lots of different humans operating differently at any given time for reasons beyond what the data can show). 2. Trained on post-tuned model behavior directly. One-step training is a considerably simpler problem than two-step. It doesn't learning anything it will need to unlearn or modify. 3. These simplifications both reduce the necessary size of the second model. And with these combined advantages (a simpler more consistent single-system to model, smaller size, and fewer training steps), it can be trained much faster. There, of course, can be disadvantages. The distilled model may not be as versatile, if the data used to prompt the original model isn't as diverse as the first models data. So prompt data for training still matters.

yimby2001 2y ago

It seems like there’s a misunderstanding as why this happened. They’ve been baking this model for months. long before deep seek came out with fundamental new ways of distilling models. and even given that it’s not great it’s its large form, they’re going to distil from this going forward .. so it likely makes sense for them to periodically train these very large models as a basis.

lhl 2y ago
I think this framing isn't quite right either. DeepSeek's R1 isn't very different from what OpenAI has already been doing with o1 (and that other groups have been doing as well). As for distilling - the R1 "distilled" models they released aren't even proper (logit) distillations, but just SFTs, not fundamentally new at all. But it's great that they published their full recipes and it's also great to see that it's effective. In fact we've seen now with LIMO, s1/s1.1, that even as few as 1K reasoning traces can get most LLMs to near SOTA math benchmarks. This mirrors the "Alpaca" moment in a lot of ways (and you could even directly mirror say LIMO w/ LIMA). I think the main takeaway of GPT4.5 (Orion) is that it basically gives a perspective to all the "hit a wall" talk from the end of last year. Here we have a model that has been trained on by many accounts 10-100X the compute of GPT4, is likely several times larger in parameter count, but is only... subtly better, certainly not super-intelligent. I've been playing around w/ it a lot the past few days, both with several million tokens worth of non-standard benchmarks and talking to it and it is better than previous GPTs (in particular, it makes a big jump in humor), but I think it's clear that the "easy" gains in the near future are going to be figuring out how as many domains as possible can be approximately verified/RL'd. As for the release? I suppose they could just have kept it internally for distillation/knowledge transfer, so I'm actually happy that they released it, even if it ends up not being a really "useful" model.

modeless 2y ago

I've been using 4.5 instead of 4o for quick questions. I don't mind the slowness for short answers. I feel like it is less likely to hallucinate than other models.

kubb 2y ago

Seems like we're hitting the limits of the technology...

xmichael909 2y ago
Yes, I believe the sprint is over, now its doing to be slow cycles maybe 18 months to see a 5% increase an ability and even that 5% increase will be highly subjective. Claude's new release is about the same 3.7 is arguably worse at some things than 3.5 and better at others. Based on the previous pace of release in about 6 months or so - if the next release from any of the leaders is about the same "kinda better kinda worse" then we'll know. Imagine how much money is going to evaporate from the stock market if this is the limit!!!
- ANewFormation 2y ago
  You can keep getting rich off shovels long after the gold has run dry.
- deleted 2y ago
  [deleted]
- borgdefenser 2y ago
  To say 3.7 is worse is completely insane.
TIPSIO 2y ago
I also hate waiting on reasoning. I much would prefer a super lightning fast model that is cheaper but the same quality as these frontier models. Let me query these things to death.
- ljlolel 2y ago
  try groq (hyperfast chips) https://groq.com/ https://groq.com/
apwell23 2y ago
does it mean we get a reprieve from "this is just the beginning" comments.

retskrad 2y ago

Sam Altman views Steve Jobs as one of his inspirations (he called the iPhone the greatest product of all time). So if you look at OpenAI in the lens of Apple, where you think about making the product enjoyable to use at all costs, then it makes perfect sense why you’d spend so much money to go from 4o to 4.5 which brings such subtle differences to power users. The vast majority of users, which are over 300 million weekly, will mainly use 4o and whatever is the default. In the future they’ll use 4.5 and think it’s most human like and less robotic.

bravura 2y ago
Yes but Steve Jobs also understood the paradox of choice, and the importance of having incredibly clear delineation between every different product in your line.
- ipaddr 2y ago
  Do models matter to the regular user over brand? People talk about using chatGPT over Google's AI or Deepseek not 4o-mini vs gemini 2. OpenAI has done a good job of making the model less important and the domain gptGPT.com more important. Most of the time the model rarely matters. When you find something incorrect you may switch models but that rarely fixes the problem. Rewording a prompt has more value than changing a model.
  - esafak 2y ago
    If the model did not matter they would be spending their money on marketing or sales instead of improving the model.
    - ipaddr 2y ago
      Spending or saying they are spending is marketing but when people use their product the model doesn't matter.

pzo 2y ago

Long term it might be hard to monetise those infrastructure considering their competition: 1) For coding (API) most probably will stick to Claude 3.5 / 3.7 - big market but still small comparing to all world wide problems 2) For non-coding API IMHO gemini 2.0 flash is the winner - dirty cheap (cheaper than 4o-mini), good enough and even better than gpt-4o, cheap audio and image input. 3) For subscription app ChatGPT is probably still the best but only slightly - they have the best advanced voice audio conversation but Grok will be probably eating their lunch here

anukin 2y ago
Sesame model for voice audio imo is better than ChatGPT voice audio conversation. They are going to open source it as well.
- bckr 2y ago
  Sure but is there an app I can talk to / work with? It seems they're a voice synthesis model company, not a chatbot app / tool company.
- OsrsNeedsf2P 2y ago
  > They are going to open source it as well. Means nothing until they do
ipaddr 2y ago
For the rest of us using free tiers ChatGPT is hands down the winner allowing limited image generation, unlimited usage of some model and limited usage of 4o. Claude is still stuck at 10 messages per day and gemini is less accurate/useful.
- dingnuts 2y ago
  10 messages a day? How are people "vibe coding" with that?
  - irishloop 2y ago

HarHarVeryFunny 2y ago

GPT 4.5 also has a knowledge cutoff date of 10-2023. https://www.reddit.com/r/singularity/comments/1izpb8t/gpt45_knowledge_cutoff_is_still_october_2023/ https://www.reddit.com/r/singularity/comments/1izpb8t/gpt45_... I'm guessing that this model was finished pre-training at least a year ago (it's been 2 years since GPT 4.0 was released) and they just didn't see the hoped-for performance gains to think it warranted releasing at the time, and so put all their effort into the Q-star/strawberry = eventual O1 reasoning effort instead. It seems that OpenAI's reasoning model lead isn't perhaps what they thought it was, and the recent slew of strong non-reasoning models (Gemini 2.0 Flash, Grok 3, Sonnet 3.7) made them feel the need to release something themselves for appearances sake, so they dusted off this model, perhaps did a bit of post-training on it for EQ, and here we are. The price is a bit of a mystery - perhaps just a reflection of an older model without all the latest efficiency tricks to make it cheaper. Maybe it's dense rather than MoE - who knows.

sigmoid10 2y ago
Rumors said that GPT4.5 is an order of magnitude larger. Around 12 trillion parameters total (compared to GPT4's 1.2 trillion). It's almost certainly MoE as well, just a scaled up version. That would explain the cost. OpenAI also said that this is what they originally developed as "Omni" - the model supposed to succeed GPT4 but which fell behind expectations. So they renamed it 4.5 and shoehorned it in to remain in the news among all those competitor releases.
- ljlolel 2y ago
  the gpt-4o ("omni") is probably a distilled 4.5; hence why not much quality difference
  - sigmoid10 2y ago
    4o has been out since May last year, while omni (now rechristened as 4.5) only finished training in October/November.
    - 2y ago

phillipcarter 2y ago

My take from using it a bit is that they seem to have genuinely innovated on: - Not writing things that go off in weird directions / staying grounded in "reality" - Responding very well to tone preferences and catching nuance in what I say It seems like it's less that it has a great "personality" like Claude, but that it's capable of adapting towards being the "personality" I want and "understanding" what I'm saying in ways that other models haven't been able to do for me.

XenophileJKO 2y ago
So this kind of mirrors my feelings after using GPT-4.5 on general conversation and song writing. GPT picked up on unspecified requirements almost instantly. It is subtle (and may be undesirable in some contexts). For example in my songs, I have to bracket the section headings, it picked up on that from my original input. All the other frontier models generally have to be reminded. Additionally, I separately asked for an edit to a music style description. When I asked GPT-4.5 to write a song all by itself, it included a music style description. No other model I have worked with has done this. These are subtle differences, but in aggregate the model just generally needs less nudging to create what is required.
- deleted 2y ago
  [deleted]
- torginus 2y ago
  I haven't used 4.5 but have some experience using Claude for creative writing, and in my experience it sometimes has the uncanny ability to get to the core of my ideas, rephrasing my paragraph long descriptions into just a sentence or two, or both improving and concretizing my vague ideas into something that's insightful and tasteful. Other times it locks itself into a dull style and ignores what I ask of it and just produces boring generic garbage, and I have to wrangle it hard to get some of the spark back. I have no idea what's going on inside, but just like with Stable Diffusion, it's fairly easy to make something that has the spark of genius, and is very close to being perfect, but getting the last 10% there, and maintaining the quality seems almost impossible. It's a very weird feeling, it's hard to put into words what is exactly going on, and probably even harder to make it into a benchmark, but it makes me constantly flip-flop between scared of being how good the AI is, and questioning why I ever bothered with using it in the first place, as I would've progress much faster without it.

neom 2y ago

4.5 can extremely quickly distill and work with what I at least consider, complex nuanced thought. 4.5 is night and day better than every other AI for my work, it's quite clever and I like it. Very quick mvp comparison for the show me what you mean crew: https://chatgpt.com/share/67c48fcc-db24-800f-865b-c0485efd7f93 https://chatgpt.com/share/67c48fcc-db24-800f-865b-c0485efd7f... & https://chatgpt.com/share/67c48fe2-0830-800f-a370-7a18586e8b21 https://chatgpt.com/share/67c48fe2-0830-800f-a370-7a18586e8b... (~30 seconds vs ~3 minutes)

ttul 2y ago
I believe 4.5 is a very large and rich model. The price is high because it's costly to inference; however, the bigger reason is to ensure that others don't distill from it. Big models have a rich latent space, but it takes time to squeeze the juice out.
- esafak 2y ago
  That also means people won't use it. Way to shoot yourself in the foot. The irony of a company that has distilled the word's information complaining about another company distilling their model...
  - cscurmudgeon 2y ago
    My assumption: There will be use cases where cost of using this will be smaller than the gain from it. Data from this will make the next version better and cheaper.
  - ttul 2y ago
    The small number of use cases that do pay are providing gross margins as well as feedback that helps OpenAI in various ways. I don’t think it’s a stupid move at all.
deleted 2y ago
[deleted]
nyrikki 2y ago

EcommerceFlow 2y ago

I've found 4.5 to be quite good at "business decisions", much better than other models. It does have some magic to it, similar to Grok 3, but maybe a bit smarter?

sunami-ai 2y ago

Meanwhile all GPT4o models on Azure are set to be deprecated in May and there are no alternative models yet. We should start moving to Anthropic? DS too slow, melting under its own success. Anyone on GPT4o/Azure has any idea when they'll release the next "o" model?

Uvix 2y ago
Only an older version of GPT-4o has been deprecated and will be removed in May. The newest version will be supported through at least 20 November 2025. https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/model-retirements#current-models https://learn.microsoft.com/en-us/azure/ai-services/openai/c...
- sunami-ai 2y ago
  The Nov 2024 release, which is due to be deprecated in Nov 2025, I was told has degraded performance compared to the Aug 2024 release. In fact, OpenAI Models page says their current GPT4o API is serving the Aug release. https://platform.openai.com/docs/models#gpt-4o https://platform.openai.com/docs/models#gpt-4o So I'm still on the Aug 24 release, which, with your reminding me, is not to be deprecated till Aug 2025, but that's less than 5 months from now, and we're skipping the Nov 2024 release just as OpenAI themselves have chosen to do.

ein0p 2y ago

I have access to it. It is better, but not where most techies would care. It knows more, it writes better, it's more pleasant to talk to. I think they might have studied the traffic their hundreds of millions of users generate and realized where they need to improve, then did exactly that for their _non thinking_ model. They understand that a non-thinking model is not going to blow the doors off on coding no matter what they do, but it can do writing and "associative memory" tasks quite well, and having a lot more weights helps there. I also predict that they will fine tune their future distilled, thinking models for coding, based on the same logic, distilling from 4.5 this time. Those models have to be fast, and therefore they have to be smaller.

ghostly_s 2y ago

I don't get it. Aren't these two sentences in the same paragraph contradictory? >"Scaling to this size of model did NOT make a clear jump in capabilities we are measuring." > "The jump from GPT-4o (where we are now) to GPT-4.5 made the models go from great to really great."

XenophileJKO 2y ago
No, it means that it got better on things orthogonal to what we have mostly been measuring. On the last few rounds, we have been mostly focusing on reasoning, not as much on knowledge, "creativity", or emotional resonance.
- johnecheck 2y ago
  "It's better. We can't measure it, but we're pretty sure it's better. We also desperately need it to be better because we just spent a boat-load of money on it."

Artgodma 2y ago

No general model is the frontier. Thousands of small, specific models are infinitely more efficient than a general one. The more narrowed the task - the better algorithms work. That's obvious. Why are general models pushed so hard by its creators? Their enormous valuations are based on total control over user experience. This total control is justified by computational requirements. Users can't run general models locally. Giant data centers for billions are the moat for Model creators and corporations behind.

azan_ 2y ago
It's neither obvious nor true, generalist models outperform specialized ones all the time (so frequently that it even has its own name - the bitter lesson)
- Artgodma 2y ago
  So you think that one general model can outperform thousands of specific ones in their specific areas?
maleldil 2y ago
Certain desirable capabilities are available only in bigger models because it takes a certain size for some behaviours emerge.

mvkel 2y ago

I think this release is for the researchers who worked on it and would quit if it never saw daylight

TheKGB 2y ago

[dead]

mirekrusin 2y ago

Is somebody actually looking at those last percentages on benchmarks? Aren't we making mistake of assuming benchmarks are purely 100% correct?