11 ms·

It's cool and I'm glad it sounds like it's getting more reliable, but given the types of things people have been saying GPT-5 would be for the last two years yo

It's cool and I'm glad it sounds like it's getting more reliable, but given the types of things people have been saying GPT-5 would be for the last two years you'd expect GPT-5 to be a world-shattering release rather than incremental and stable improvement. It does sort of give me the vibe that the pure scaling maximalism really is dying off though. If the approach is on writing better routers, tooling, comboing specialized submodels on tasks, then it feels like there's a search for new ways to improve performance(and lower cost), suggesting the other established approaches weren't working. I could totally be wrong, but I feel like if just throwing more compute at the problem was working OpenAI probably wouldn't be spending much time on optimizing the user routing on currently existing strategies to get marginal improvements on average user interactions. I've been pretty negative on the thesis of only needing more data/compute to achieve AGI with current techniques though, so perhaps I'm overly biased against it. If there's one thing that bothers me in general about the situation though, it's that it feels like we really have no clue what the actual status of these models is because of how closed off all the industry labs have become + the feeling of not being able to expect anything other than marketing language from the presentations. I suppose that's inevitable with the massive investments though. Maybe they've got some massive earthshattering model release coming out next, who knows.

jstummbillig 1y ago
Things have moved differently than what we thought would happen 2 years ago, but lest we forget what has happened in the meanwhile (4o, o1 + thinking paradigm, o3) So yeah, maybe we are getting more incremental improvements. But that to me seems like a good thing, because more good things earlier. I will take that over world-shattering any day – but if we were to consider everything that has happened since the first release of gpt-4, I would argue the total amount is actually very much world-shattering.
GaggiX 1y ago
Compared to GPT-4, it is on a completely different level given that it is a reasoning model so on that regard it does delivers and it's not just scaling, but for this I guess the revolution was o1 and GPT-5 is just a much more mature version of the technology.
hnuser123456 1y ago
I agree, we have now proven that GPUs can ingest information and be trained to generate content for various tasks. But to put it to work, make it useful, requires far more thought about a specific problem and how to apply the tech. If you could just ask GPT to create a startup that'll be guaranteed to be worth $1B on a $1k investment within one year, someone else would've already done it. Elbow grease still required for the foreseeable future. In the meantime, figuring out how to train them to make less of their most common mistakes is a worthwhile effort.
- morleytj 1y ago
  Certainly, yes, plenty of elbow grease required in all things that matter. The interesting point as well to me though, is that if it could create a startup that was worth $1B, that startup wouldn't be worth $1B. Why would anyone pay that much to invest in the startup if they could recreate the entire thing with the same tool that everyone would have access to?
  - RossBencina 1y ago
    If you created a $1B startup using LLMs, would you be advertising it? or would you be creating more $1B startups.
    - morleytj 1y ago
      Comment I'm replying to poses the following scenario: "If you could just ask GPT to create a startup that'll be guaranteed to be worth $1B on a $1k investment within one year" I think if the situation is that I do this by just asking it to make a startup, it seems unlikely that no one else would be aware that they could just ask it to make a startup
  - selcuka 1y ago
    > if they could recreate the entire thing with the same tool "Within one year" is the key part. The product is only part of the equation. If a startup was launched one year ago and is worth $1B today, there is no way you can launch the same startup today and achieve the same market cap in 1 day. You still need customers, which takes time. There are also IP related issues. Facebook had the resources to create an exact copy of Instagram, or WhatsApp, but they didn't. Instead, they paid billions of dollars to acquire those companies.
BoiledCabbage 1y ago
Performance is doubling roughly every 4-7 months. That trend is continuing. That's insane. If your expectations were any higher than that then, then it seems like you were caught up in hype. Doubling 2-3 times per year isn't leveling off my any means. https://metr.github.io/autonomy-evals-guide/gpt-5-report/ https://metr.github.io/autonomy-evals-guide/gpt-5-report/
- oblio 1y ago
  By "performance" I guess you mean "the length of task that can be done adequately"? It is a benchmark but I'm not very convinced it's the be-all, end-all.
  - nomel 1y ago
    > It is a benchmark but I'm not very convinced it's the be-all, end-all. Who's suggesting it is?
- morleytj 1y ago
  I wouldn't say model development and performance is "leveling off", and in fact didn't write that. I'd say that tons more funding is going into the development of many models, so one would expect performance increases unless the paradigm was completely flawed at it's core, a belief I wouldn't personally profess to. My point was moreso the following: A couple years ago it was easy to find people saying that all we needed was to add in video data, or genetic data, or some other data modality, in the exact same format that the models trained on existing language data were, and we'd see a fast takeoff scenario with no other algorithmic changes. Given that the top labs seem to be increasingly investigating alternate approaches to setting up the models beyond just adding more data sources, and have been for the last couple years(Which, I should clarify, is a good idea in my opinion), then the probability of those statements of just adding more data or more compute taking us straight to AGI being correct seems at the very least slightly lower, right? Rather than my personal opinion, I was commenting on commonly viewed opinions of people I would believe to have been caught up in hype in the past. But I do feel that although that's a benchmark, it's not necessarily the end-all of benchmarks. I'll reserve my final opinions until I test personally, of course. I will say that increasing the context window probably translates pretty well to longer context task performance, but I'm not entirely convinced it directly translates to individual end-step improvement on every class of task.
simonw 1y ago
I for one am pretty glad about this. I like LLMs that augment human abilities - tools that help people get more done and be more ambitious. The common concept for AGI seems to be much more about human replacement - the ability to complete "economically valuable tasks" better than humans can. I still don't understand what our human lives or economies would look like there. What I personally wanted from GPT-5 is exactly what I got: models that do the same stuff that existing models do, but more reliably and "better".
- morleytj 1y ago
  I'd agree on that. That's pretty much the key component these approaches have been lacking on, the reliability and consistency on the tasks they already work well on to some extent. I think there's a lot of visions of what our human lives would look like in that world that I can imagine, but your comment did make me think of one particularly interesting tautological scenario in that commonly defined version of AGI. If artificial general intelligence is defined as completed "economically valuable tasks" better than human can, it requires one to define "economically valuable." As it currently stands, something holds value in an economy relative to human beings wanting it. Houses get expensive because many people, each of whom have economic utility which they use to purchase things, want to have houses, of which there is a limited supply for a variety of reasons. If human beings are not the most effective producers of value in the system, they lose capability to trade for things, which negates that existing definition of economic value. Doesn't matter how many people would pay $5 dollars for your widget if people have no economic utility relative to AGI, meaning they cannot trade that utility for goods. In general that sort of definition of AGI being held reveals a bit of a deeper belief, which is that there is some version of economic value detached from the humans consuming it. Some sort of nebulous concept of progress, rather than the acknowledgement that for all of human history, progress and value have both been relative to the people themselves getting some form of value or progress. I suppose it generally points to the idea of an economy without consumers, which is always a pretty bizarre thing to consider, but in that case, wouldn't it just be a definition saying that "AGI is achieved when it can do things that the people who control the AI system think are useful." Since in that case, the economy would eventually largely consist of the people controlling the most economically valuable agents. I suppose that's the whole point of the various alignment studies, but I do find it kind of interesting to think about the fact that even the concept of something being "economically valuable", which sounds very rigorous and measurable to many people, is so nebulous as to be dependent on our preferences and wants as a society.
thorum 1y ago
The quiet revolution is happening in tool use and multimodal capabilities. Moderate incremental improvements on general intelligence, but dramatic improvements on multi-step tool use and ability to interact with the world (vs 1 year ago), will eventually feed back into general intelligence.
- darkhorse222 1y ago
  Completely agree. General intelligence is a building block. By chaining things together you can achieve meta programming. The trick isn't to create one perfect block but to build a variety of blocks and make one of those blocks a block-builder.
  - SecretDreams 1y ago
    > The trick isn't to create one perfect block but to build a variety of blocks and make one of those blocks a block-builder. This has some Egyptian pyramids building vibes. I hope we treat these AGIs better than the deal the pyramid slaves got.
    - z0r 1y ago
      We don't have AGI and the pyramids weren't built by slaves.
- coolKid721 1y ago
  [flagged]
  - dang 1y ago
    Can you please make your substantive points thoughtfully? Thoughtful criticism is welcome but snarky putdowns and onliners, etc., degrade the discussion for everyone. You've posted substantive comments in other threads, so this should be easy to fix. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
belter 1y ago
> Maybe they've got some massive earthshattering model release coming out next, who knows. Nothing in the current technology offers a path to AGI. These models are fixed after training completes.
- echoangle 1y ago
  Why do you think that AGI necessitates modification of the model during use? Couldn’t all the insights the model gains be contained in the context given to it?
  - belter 1y ago
    Because: https://en.wikipedia.org/wiki/Anterograde_amnesia https://en.wikipedia.org/wiki/Anterograde_amnesia
    - echoangle 1y ago
      Like I already said, the model can remember stuff as long as it’s in the context. LLMs can obviously remember stuff they were told or output themselves, even a few messages later.
      - godelski 1y ago
        > the model can remember stuff as long as it’s in the context. You would need an infinite context or compression Also you might be interested in this theorem https://en.wikipedia.org/wiki/Data_processing_inequality https://en.wikipedia.org/wiki/Data_processing_inequality
        echoangle 1y ago
        > You would need an infinite context or compression Only if AGI would require infinite knowledge, which it doesn’t.
        1y ago
cchance 1y ago
SAM is a HYPE CEO, he literally hypes his company nonstop, then the announcements come and ... they're... ok, so people aren't really upset, but they end up feeling lackluster at the hype... Until the next cycle comes around... If you want actual big moves, watch google, anthropic, qwen, deepseek. Qwen and Deepseek teams honestly seem so much better at under promising and over delivering. Cant wait to see what Gemini 3 looks like too.
brandall10 1y ago
To be fair, this is one of the pathways GPT-5 was speculated to take as far back at 6 or so months ago - simply being an incremental upgrade from a performance perspective, but a leap from a product simplification approach. At this point it's pretty much given it's a game of inches moving forward.
- ac29 1y ago
  > a leap from a product simplification approach. According to the article, GPT-5 is actually three models and they can be run at 4 levels of thinking. Thats a dozen ways you can run any given input on "GPT-5", so its hardly a simple product line up (but maybe better than before).
  - brandall10 1y ago
    It's a big improvement from an API consumer standpoint - everything is now under a single product family that is logically stratified... up until yesterday people were using o3, o4-mini, 4o, 4.1, o3, and all their variants as valid choices for new products, now those are moved off the main page as legacy or specialized options for the few things GPT-5 doesn't do. It's even more simplified for the ChatGPT plan, It's just GPT-5 thinking/non-thinking for most accounts, and then the option of Pro for the higher end accounts.
  - eru 1y ago
    A bit like Google Search uses a lot of different components under the hood?
AbstractH24 1y ago
> It's cool and I'm glad it sounds like it's getting more reliable, but given the types of things people have been saying GPT-5 would be for the last two years you'd expect GPT-5 to be a world-shattering release rather than incremental and stable improvement. Are you trying to say the curve is flattening? That advances are coming slower and slower? As long as it doesn't suggest a dot com level recession I'm good.
- morleytj 1y ago
  I suppose what I'm getting at is that if there are performance increases on a steady pace, but the investment needed to get those performance increases is on a much faster growth rate, it's not really a fair comparison in terms of a rate of progress, and could suggest diminishing returns from a particular approach. I don't really have the actual data to make a claim either way though,I think anyone would need more data to do so than is publicly accessible. But I do think the fact that we can publicly observe this reallocation of resources and emphasized aspects of the models gives us a bit of insight into what could be happening behind the scenes if we think about the reasons why those shifts could have happened, I guess.
  - Karrot_Kream 1y ago
    How are you measuring investment? If we're looking at aggregate AI investment, I would guess that a lot of it is going into applications built atop AI rather than on the LLMs themselves. That's going to be tools, MCPs, workflow builders, etc
godelski 1y ago
> It does sort of give me the vibe that the pure scaling maximalism really is dying off though I think the big question is if/when investors will start giving money to those who have been predicting this (with evidence) and trying other avenues. Really though, why put all your eggs in one basket? That's what I've been confused about for awhile. Why fund yet another LLMs to AGI startup. Space is saturated with big players and has been for years. Even if LLMs could get there that doesn't mean something else won't get there faster and for less. It also seems you'd want a backup in order to avoid popping the bubble. Technology S-Curves and all that still apply to AI Though I'm similarly biased, but so is everyone I know with a strong math and/or science background (I even mentioned it in my thesis more than a few times lol). Scaling is all you need just doesn't check out
- morleytj 1y ago
  I'm pretty curious about the same thing. I think a somewhat comparable situation is in various online game platforms now that I think about it. Investors would love to make a game like Fortnite, and get the profits that Fortnite makes. So a ton of companies try to make Fortnite. Almost all fail, and make no return whatsoever, just lose a ton of money and toss the game in the bin, shut down the servers. On the other hand, it may have been more logical for many of them to go for a less ambitious (not always online, not a game that requires a high player count and social buy-in to stay relevant) but still profitable investment (Maybe a smaller scale single player game that doesn't offer recurring revenue), yet we still see a very crowded space for trying to emulate the same business model as something like Fortnite. Another more historical example was the constant question of whether a given MMO would be the next "WoW-killer" all through the 2000's/2010's. I think part of why this arises is that there's definitely a bit of a psychological hack for humans in particular where if there's a low-probability but extremely high reward outcome, we're deeply entranced by it, and investors are the same. Even if the chances are smaller in their minds than they were before, if they can just follow the same path that seems to be working to some extent and then get lucky, they're completely set. They're not really thinking about any broader bubble that could exist, that's on the level of the society, they're thinking about the individual, who could be very very rich, famous, and powerful if their investment works. And in the mind of someone debating what path to go down, I imagine a more nebulous answer of "we probably need to come up with some fundamentally different tools for learning and research a lot of different approaches to do so" is a bit less satisfying and exciting than a pitch that says "If you just give me enough money, the curve will eventually hit the point where you get to be king of the universe and we go colonize the solar system and carve your face into the moon." I also have to acknowledge the possibility that they just have access to different information than I do! They might be getting shown much better demos than I do, I suppose.
famouswaffles 1y ago
>you'd expect GPT-5 to be a world-shattering release rather than incremental and stable improvement. Compared to the GPT-4 release which was a little over 2 years ago (less than the gap between 3 and 4), it is. The only difference is we now have multiple organizations releasing state of the art models every few months. Even if models are improving at the same rate, those same big jumps after every handful of months was never realistic. It's an incremental stable improvement over o3, which was released what? 4 months ago.
- morleytj 1y ago
  The benchmarks certainly seem to be improving from the presentation. I don't think they started training this 4 months ago though. There's gains, but the question is, how much investment for that gain? How sustainable is that investment to gain ratio? The things I'm curious about here are more about the amount of effort being put into this level of improvement, rather than the time.
outside1234 1y ago
The next step will be for OpenAI to number their releases based on year (ala what Windows did once innovation ran out)
- eru 1y ago
  Windows 95 was a big step from the previous release, wasn't it? And later, Windows reverted to version numbers; but I'm not sure they regained lots of innovation?
fastball 1y ago
My reading is more that unit economics are starting to catch up with the frontier labs, rather than "scaling maximalism is dying". Maybe that is the same thing.
- ch4s3 1y ago
  My loosely held belief is that it is the same thing, but I’m open to being proven wrong.
danenania 1y ago
Isn’t reasoning, aka test-time compute, ultimately just another form of scaling? Yes it happens at a different stage, but the equation is still 'scale total compute > more intelligence'. In that sense, combining their biggest pre-trained models with their best reasoning strategies from RL could be the most impactful scaling lever available to them at the moment.
mwigdahl 1y ago
It seems like few people are referencing the improvements in reliability and deception. If the benchmarks given generalize, what OpenAI has in GPT-5 is a cheap, powerful, _reliable_ model -- the perfect engine to generate high quality synthetic data to punch through the training data bottleneck. I'd expect that at some level of reliability this could lead to a self-improvement cycle, similar to how a powerful enough model (the Claude 4 models in Claude Code) enables iteratively converging on a solution to a problem even if it can't one-shot it. No idea if we're at that point yet, but it seems a natural use for a model with these characteristics.
maoberlehner 1y ago
I mostly use Gemini 2.5 Pro. I have a “you are my editor” prompt asking it to proofread my texts. Recently it pointed out two typos in two different words that just weren’t there. Indeed, the two words each had a typo but not the one pointed out by Gemini. The real typos were random missing letters. But the typos Gemini hallucinated were ones that are very common typos made in those words. The only thing transformer based LLMs can ever do is _faking_ intelligence. Which for many tasks is good enough. Even in my example above, the corrected text was flawless. But for a whole category of tasks, LLMs without oversight will never be good enough because there simply is no real intelligence in them.
- shmel 1y ago
  do you really think that an architecture that struggles to count r in strawberry is a good choice for proofreading? It perceives words very differently from us.
  - maoberlehner 1y ago
    Yes, actually I think it works really well for me considering that I’m not a native speaker and one thing I’m after is correcting technical correct but non-idiomatic wording.
  - simonw 1y ago
    Counting letters in words and identifying when words are misspelled are two different tasks - it can be good at one and bad at the other. Interestingly, spell checking is something models have been surprisingly bad at in the past - I remember being shocked at how bad Claude 3 was at spotting typos. This has changed with Claude 4 and o3 from what I've seen - another example of incremental model improvements swinging over a line in terms of things they can now be useful for.
    - slagsnag 1y ago
      Shill harder, Simon! Otherwise they may refuse to ask you back for their next PR chucklefest.
dotancohen 1y ago
> but given the types of things people have been saying GPT-5 would be for the last two years This is why you listen to official announcements, not "people".
eloisant 1y ago
It reminds me of the latest, most advanced steam locomotives from the beginning of the 20th century. They become extremely complex and sophisticated machines to squeeze a few more percent of efficiency compared to earlier models. Then diesel, and eventually electric locomotive arrived, much better and also much simpler than those late steam monsters. I feel like that's where we are with LLM: extremely smart engineering to marginally improve quality, while increasing cost and complexity greatly. At some point we'll need a different approach if we want a world-shattering release.
- anshumankmr 1y ago
  Just to add to this, if we see F1 cars a way to measure the cutting edge of cars being developed, we can see cars haven't become insanely faster than they were 10-20 years ago, just more "efficient",reliable and definitely safer along with quirks like DRS. Of course shaving off a second or two from laptimes is notable, but not an insane delta like say if you compared a car from post 2000s GP to 1950s GP. I feel after a while we will have specialized LLMs great for one particular task down the line as well, cut off updates, 0.something better than the SOTA on some benchmark and as compute gets better, cheaper to run at scale.
  - Anonyneko 1y ago
    To be fair, the speed of F1 cars is mostly limited by regulations that are meant to make the sport more competitive and entertaining. With fewer restrictions on engines and aerodynamics we could have had much faster cars within a year. But even setting safety issues aside, the insane aero wash would make it nearly impossible to follow another car, let alone overtake it, hence the restrictions and the big "rule resets" every few years that slow down the cars, compensating for all of the tricks the teams have found over that time. (I agree with the general thoughts on the state of LLMs though, just a bit too much into open-wheel cars going vroom vroom in circles for two hours at a time)
  - epolanski 1y ago
    > we can see cars haven't become insanely faster than they were 10-20 years ago I don't think this is a good example, because the regulations in F1 are largely focused on slowing the cars. It's an engineering miracle that 2025 cars can be competitive with previous generation cars on so many tracks.
jillesvangurp 1y ago
Year on year, progress is indeed a bit incremental. But seen over a five year period the progress is actually stupidly amazing. In practical terms, Gpt 5 is a nice upgrade over most other models. We'll no doubt get lots of subjective reports how it was wrong or right or worse than some other model for some chats. But my personal (subjective) experience so far is that it just made it possible for me to use codex on more serious projects. It still gets plenty of things wrong. But that's more because of a lack of context than hallucination issues. Context fixes are a lot easier than model improvements. But last week I didn't bother and now I'm getting decent results. I don't really care what version number they slap on things. That is indeed just marketing. And competition is quite fierce so I can understand why they are overselling what could have been just chat gpt 4.2 or whatever. Also discussions about AGI tend to bore me as they seem to escalate into low quality philosophical debates with lots of amateurs rehashing ancient argument poorly. There aren't a hell of a lot new arguments that people come up with at this point. IMHO we don't actually need an AGI to bootstrap the singularity. We just AIs to be good enough to come up with algorithmic optimizations, breakthroughs and improvements at a steady pace. We're getting quite close to that and I wouldn't be surprised to learn that OpenAI's people are already eating their own dogfood in liberal quantities. It's not necessary for AIs to be conscious in order to come up with the improvements that might eventually enable such a thing. I expect the singularity might be more of a phase than a moment. And if you know your boiling frog analogy, we might be smack down in the middle of that already and just not realize it. Five years ago, it was all very theoretical. And now I'm waiting for codex to wrap up a few pull requests that would have distracted me for a week each five years ago. It's taking too long and I'm procrastinating my gained productivity away on HN. But what else is new ;-).
tough 1y ago
there's no more data to throw compute at to 1T tokens is a lot

jstummbillig 1y ago

Things have moved differently than what we thought would happen 2 years ago, but lest we forget what has happened in the meanwhile (4o, o1 + thinking paradigm, o3) So yeah, maybe we are getting more incremental improvements. But that to me seems like a good thing, because more good things earlier. I will take that over world-shattering any day – but if we were to consider everything that has happened since the first release of gpt-4, I would argue the total amount is actually very much world-shattering.

GaggiX 1y ago

Compared to GPT-4, it is on a completely different level given that it is a reasoning model so on that regard it does delivers and it's not just scaling, but for this I guess the revolution was o1 and GPT-5 is just a much more mature version of the technology.

hnuser123456 1y ago

I agree, we have now proven that GPUs can ingest information and be trained to generate content for various tasks. But to put it to work, make it useful, requires far more thought about a specific problem and how to apply the tech. If you could just ask GPT to create a startup that'll be guaranteed to be worth $1B on a $1k investment within one year, someone else would've already done it. Elbow grease still required for the foreseeable future. In the meantime, figuring out how to train them to make less of their most common mistakes is a worthwhile effort.

morleytj 1y ago
Certainly, yes, plenty of elbow grease required in all things that matter. The interesting point as well to me though, is that if it could create a startup that was worth $1B, that startup wouldn't be worth $1B. Why would anyone pay that much to invest in the startup if they could recreate the entire thing with the same tool that everyone would have access to?
- RossBencina 1y ago
  If you created a $1B startup using LLMs, would you be advertising it? or would you be creating more $1B startups.
  - morleytj 1y ago
    Comment I'm replying to poses the following scenario: "If you could just ask GPT to create a startup that'll be guaranteed to be worth $1B on a $1k investment within one year" I think if the situation is that I do this by just asking it to make a startup, it seems unlikely that no one else would be aware that they could just ask it to make a startup
- selcuka 1y ago
  > if they could recreate the entire thing with the same tool "Within one year" is the key part. The product is only part of the equation. If a startup was launched one year ago and is worth $1B today, there is no way you can launch the same startup today and achieve the same market cap in 1 day. You still need customers, which takes time. There are also IP related issues. Facebook had the resources to create an exact copy of Instagram, or WhatsApp, but they didn't. Instead, they paid billions of dollars to acquire those companies.

BoiledCabbage 1y ago

Performance is doubling roughly every 4-7 months. That trend is continuing. That's insane. If your expectations were any higher than that then, then it seems like you were caught up in hype. Doubling 2-3 times per year isn't leveling off my any means. https://metr.github.io/autonomy-evals-guide/gpt-5-report/ https://metr.github.io/autonomy-evals-guide/gpt-5-report/

oblio 1y ago
By "performance" I guess you mean "the length of task that can be done adequately"? It is a benchmark but I'm not very convinced it's the be-all, end-all.
- nomel 1y ago
  > It is a benchmark but I'm not very convinced it's the be-all, end-all. Who's suggesting it is?
morleytj 1y ago
I wouldn't say model development and performance is "leveling off", and in fact didn't write that. I'd say that tons more funding is going into the development of many models, so one would expect performance increases unless the paradigm was completely flawed at it's core, a belief I wouldn't personally profess to. My point was moreso the following: A couple years ago it was easy to find people saying that all we needed was to add in video data, or genetic data, or some other data modality, in the exact same format that the models trained on existing language data were, and we'd see a fast takeoff scenario with no other algorithmic changes. Given that the top labs seem to be increasingly investigating alternate approaches to setting up the models beyond just adding more data sources, and have been for the last couple years(Which, I should clarify, is a good idea in my opinion), then the probability of those statements of just adding more data or more compute taking us straight to AGI being correct seems at the very least slightly lower, right? Rather than my personal opinion, I was commenting on commonly viewed opinions of people I would believe to have been caught up in hype in the past. But I do feel that although that's a benchmark, it's not necessarily the end-all of benchmarks. I'll reserve my final opinions until I test personally, of course. I will say that increasing the context window probably translates pretty well to longer context task performance, but I'm not entirely convinced it directly translates to individual end-step improvement on every class of task.

simonw 1y ago

I for one am pretty glad about this. I like LLMs that augment human abilities - tools that help people get more done and be more ambitious. The common concept for AGI seems to be much more about human replacement - the ability to complete "economically valuable tasks" better than humans can. I still don't understand what our human lives or economies would look like there. What I personally wanted from GPT-5 is exactly what I got: models that do the same stuff that existing models do, but more reliably and "better".

morleytj 1y ago
I'd agree on that. That's pretty much the key component these approaches have been lacking on, the reliability and consistency on the tasks they already work well on to some extent. I think there's a lot of visions of what our human lives would look like in that world that I can imagine, but your comment did make me think of one particularly interesting tautological scenario in that commonly defined version of AGI. If artificial general intelligence is defined as completed "economically valuable tasks" better than human can, it requires one to define "economically valuable." As it currently stands, something holds value in an economy relative to human beings wanting it. Houses get expensive because many people, each of whom have economic utility which they use to purchase things, want to have houses, of which there is a limited supply for a variety of reasons. If human beings are not the most effective producers of value in the system, they lose capability to trade for things, which negates that existing definition of economic value. Doesn't matter how many people would pay $5 dollars for your widget if people have no economic utility relative to AGI, meaning they cannot trade that utility for goods. In general that sort of definition of AGI being held reveals a bit of a deeper belief, which is that there is some version of economic value detached from the humans consuming it. Some sort of nebulous concept of progress, rather than the acknowledgement that for all of human history, progress and value have both been relative to the people themselves getting some form of value or progress. I suppose it generally points to the idea of an economy without consumers, which is always a pretty bizarre thing to consider, but in that case, wouldn't it just be a definition saying that "AGI is achieved when it can do things that the people who control the AI system think are useful." Since in that case, the economy would eventually largely consist of the people controlling the most economically valuable agents. I suppose that's the whole point of the various alignment studies, but I do find it kind of interesting to think about the fact that even the concept of something being "economically valuable", which sounds very rigorous and measurable to many people, is so nebulous as to be dependent on our preferences and wants as a society.

thorum 1y ago

The quiet revolution is happening in tool use and multimodal capabilities. Moderate incremental improvements on general intelligence, but dramatic improvements on multi-step tool use and ability to interact with the world (vs 1 year ago), will eventually feed back into general intelligence.

darkhorse222 1y ago
Completely agree. General intelligence is a building block. By chaining things together you can achieve meta programming. The trick isn't to create one perfect block but to build a variety of blocks and make one of those blocks a block-builder.
- SecretDreams 1y ago
  > The trick isn't to create one perfect block but to build a variety of blocks and make one of those blocks a block-builder. This has some Egyptian pyramids building vibes. I hope we treat these AGIs better than the deal the pyramid slaves got.
  - z0r 1y ago
    We don't have AGI and the pyramids weren't built by slaves.
coolKid721 1y ago
[flagged]
- dang 1y ago
  Can you please make your substantive points thoughtfully? Thoughtful criticism is welcome but snarky putdowns and onliners, etc., degrade the discussion for everyone. You've posted substantive comments in other threads, so this should be easy to fix. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

belter 1y ago

> Maybe they've got some massive earthshattering model release coming out next, who knows. Nothing in the current technology offers a path to AGI. These models are fixed after training completes.

echoangle 1y ago
Why do you think that AGI necessitates modification of the model during use? Couldn’t all the insights the model gains be contained in the context given to it?
- belter 1y ago
  Because: https://en.wikipedia.org/wiki/Anterograde_amnesia https://en.wikipedia.org/wiki/Anterograde_amnesia
  - echoangle 1y ago
    Like I already said, the model can remember stuff as long as it’s in the context. LLMs can obviously remember stuff they were told or output themselves, even a few messages later.
    - godelski 1y ago
      > the model can remember stuff as long as it’s in the context. You would need an infinite context or compression Also you might be interested in this theorem https://en.wikipedia.org/wiki/Data_processing_inequality https://en.wikipedia.org/wiki/Data_processing_inequality
      - echoangle 1y ago
        > You would need an infinite context or compression Only if AGI would require infinite knowledge, which it doesn’t.
        1y ago

cchance 1y ago

SAM is a HYPE CEO, he literally hypes his company nonstop, then the announcements come and ... they're... ok, so people aren't really upset, but they end up feeling lackluster at the hype... Until the next cycle comes around... If you want actual big moves, watch google, anthropic, qwen, deepseek. Qwen and Deepseek teams honestly seem so much better at under promising and over delivering. Cant wait to see what Gemini 3 looks like too.

brandall10 1y ago

To be fair, this is one of the pathways GPT-5 was speculated to take as far back at 6 or so months ago - simply being an incremental upgrade from a performance perspective, but a leap from a product simplification approach. At this point it's pretty much given it's a game of inches moving forward.

ac29 1y ago
> a leap from a product simplification approach. According to the article, GPT-5 is actually three models and they can be run at 4 levels of thinking. Thats a dozen ways you can run any given input on "GPT-5", so its hardly a simple product line up (but maybe better than before).
- brandall10 1y ago
  It's a big improvement from an API consumer standpoint - everything is now under a single product family that is logically stratified... up until yesterday people were using o3, o4-mini, 4o, 4.1, o3, and all their variants as valid choices for new products, now those are moved off the main page as legacy or specialized options for the few things GPT-5 doesn't do. It's even more simplified for the ChatGPT plan, It's just GPT-5 thinking/non-thinking for most accounts, and then the option of Pro for the higher end accounts.
- eru 1y ago
  A bit like Google Search uses a lot of different components under the hood?

AbstractH24 1y ago

> It's cool and I'm glad it sounds like it's getting more reliable, but given the types of things people have been saying GPT-5 would be for the last two years you'd expect GPT-5 to be a world-shattering release rather than incremental and stable improvement. Are you trying to say the curve is flattening? That advances are coming slower and slower? As long as it doesn't suggest a dot com level recession I'm good.

morleytj 1y ago
I suppose what I'm getting at is that if there are performance increases on a steady pace, but the investment needed to get those performance increases is on a much faster growth rate, it's not really a fair comparison in terms of a rate of progress, and could suggest diminishing returns from a particular approach. I don't really have the actual data to make a claim either way though,I think anyone would need more data to do so than is publicly accessible. But I do think the fact that we can publicly observe this reallocation of resources and emphasized aspects of the models gives us a bit of insight into what could be happening behind the scenes if we think about the reasons why those shifts could have happened, I guess.
- Karrot_Kream 1y ago
  How are you measuring investment? If we're looking at aggregate AI investment, I would guess that a lot of it is going into applications built atop AI rather than on the LLMs themselves. That's going to be tools, MCPs, workflow builders, etc

godelski 1y ago

> It does sort of give me the vibe that the pure scaling maximalism really is dying off though I think the big question is if/when investors will start giving money to those who have been predicting this (with evidence) and trying other avenues. Really though, why put all your eggs in one basket? That's what I've been confused about for awhile. Why fund yet another LLMs to AGI startup. Space is saturated with big players and has been for years. Even if LLMs could get there that doesn't mean something else won't get there faster and for less. It also seems you'd want a backup in order to avoid popping the bubble. Technology S-Curves and all that still apply to AI Though I'm similarly biased, but so is everyone I know with a strong math and/or science background (I even mentioned it in my thesis more than a few times lol). Scaling is all you need just doesn't check out

morleytj 1y ago
I'm pretty curious about the same thing. I think a somewhat comparable situation is in various online game platforms now that I think about it. Investors would love to make a game like Fortnite, and get the profits that Fortnite makes. So a ton of companies try to make Fortnite. Almost all fail, and make no return whatsoever, just lose a ton of money and toss the game in the bin, shut down the servers. On the other hand, it may have been more logical for many of them to go for a less ambitious (not always online, not a game that requires a high player count and social buy-in to stay relevant) but still profitable investment (Maybe a smaller scale single player game that doesn't offer recurring revenue), yet we still see a very crowded space for trying to emulate the same business model as something like Fortnite. Another more historical example was the constant question of whether a given MMO would be the next "WoW-killer" all through the 2000's/2010's. I think part of why this arises is that there's definitely a bit of a psychological hack for humans in particular where if there's a low-probability but extremely high reward outcome, we're deeply entranced by it, and investors are the same. Even if the chances are smaller in their minds than they were before, if they can just follow the same path that seems to be working to some extent and then get lucky, they're completely set. They're not really thinking about any broader bubble that could exist, that's on the level of the society, they're thinking about the individual, who could be very very rich, famous, and powerful if their investment works. And in the mind of someone debating what path to go down, I imagine a more nebulous answer of "we probably need to come up with some fundamentally different tools for learning and research a lot of different approaches to do so" is a bit less satisfying and exciting than a pitch that says "If you just give me enough money, the curve will eventually hit the point where you get to be king of the universe and we go colonize the solar system and carve your face into the moon." I also have to acknowledge the possibility that they just have access to different information than I do! They might be getting shown much better demos than I do, I suppose.

famouswaffles 1y ago

>you'd expect GPT-5 to be a world-shattering release rather than incremental and stable improvement. Compared to the GPT-4 release which was a little over 2 years ago (less than the gap between 3 and 4), it is. The only difference is we now have multiple organizations releasing state of the art models every few months. Even if models are improving at the same rate, those same big jumps after every handful of months was never realistic. It's an incremental stable improvement over o3, which was released what? 4 months ago.

morleytj 1y ago
The benchmarks certainly seem to be improving from the presentation. I don't think they started training this 4 months ago though. There's gains, but the question is, how much investment for that gain? How sustainable is that investment to gain ratio? The things I'm curious about here are more about the amount of effort being put into this level of improvement, rather than the time.

outside1234 1y ago

The next step will be for OpenAI to number their releases based on year (ala what Windows did once innovation ran out)

eru 1y ago
Windows 95 was a big step from the previous release, wasn't it? And later, Windows reverted to version numbers; but I'm not sure they regained lots of innovation?

fastball 1y ago

My reading is more that unit economics are starting to catch up with the frontier labs, rather than "scaling maximalism is dying". Maybe that is the same thing.

ch4s3 1y ago
My loosely held belief is that it is the same thing, but I’m open to being proven wrong.

danenania 1y ago

Isn’t reasoning, aka test-time compute, ultimately just another form of scaling? Yes it happens at a different stage, but the equation is still 'scale total compute > more intelligence'. In that sense, combining their biggest pre-trained models with their best reasoning strategies from RL could be the most impactful scaling lever available to them at the moment.

mwigdahl 1y ago

It seems like few people are referencing the improvements in reliability and deception. If the benchmarks given generalize, what OpenAI has in GPT-5 is a cheap, powerful, _reliable_ model -- the perfect engine to generate high quality synthetic data to punch through the training data bottleneck. I'd expect that at some level of reliability this could lead to a self-improvement cycle, similar to how a powerful enough model (the Claude 4 models in Claude Code) enables iteratively converging on a solution to a problem even if it can't one-shot it. No idea if we're at that point yet, but it seems a natural use for a model with these characteristics.

maoberlehner 1y ago

I mostly use Gemini 2.5 Pro. I have a “you are my editor” prompt asking it to proofread my texts. Recently it pointed out two typos in two different words that just weren’t there. Indeed, the two words each had a typo but not the one pointed out by Gemini. The real typos were random missing letters. But the typos Gemini hallucinated were ones that are very common typos made in those words. The only thing transformer based LLMs can ever do is _faking_ intelligence. Which for many tasks is good enough. Even in my example above, the corrected text was flawless. But for a whole category of tasks, LLMs without oversight will never be good enough because there simply is no real intelligence in them.

shmel 1y ago
do you really think that an architecture that struggles to count r in strawberry is a good choice for proofreading? It perceives words very differently from us.
- maoberlehner 1y ago
  Yes, actually I think it works really well for me considering that I’m not a native speaker and one thing I’m after is correcting technical correct but non-idiomatic wording.
- simonw 1y ago
  Counting letters in words and identifying when words are misspelled are two different tasks - it can be good at one and bad at the other. Interestingly, spell checking is something models have been surprisingly bad at in the past - I remember being shocked at how bad Claude 3 was at spotting typos. This has changed with Claude 4 and o3 from what I've seen - another example of incremental model improvements swinging over a line in terms of things they can now be useful for.
  - slagsnag 1y ago
    Shill harder, Simon! Otherwise they may refuse to ask you back for their next PR chucklefest.

dotancohen 1y ago

> but given the types of things people have been saying GPT-5 would be for the last two years This is why you listen to official announcements, not "people".

eloisant 1y ago

It reminds me of the latest, most advanced steam locomotives from the beginning of the 20th century. They become extremely complex and sophisticated machines to squeeze a few more percent of efficiency compared to earlier models. Then diesel, and eventually electric locomotive arrived, much better and also much simpler than those late steam monsters. I feel like that's where we are with LLM: extremely smart engineering to marginally improve quality, while increasing cost and complexity greatly. At some point we'll need a different approach if we want a world-shattering release.

anshumankmr 1y ago
Just to add to this, if we see F1 cars a way to measure the cutting edge of cars being developed, we can see cars haven't become insanely faster than they were 10-20 years ago, just more "efficient",reliable and definitely safer along with quirks like DRS. Of course shaving off a second or two from laptimes is notable, but not an insane delta like say if you compared a car from post 2000s GP to 1950s GP. I feel after a while we will have specialized LLMs great for one particular task down the line as well, cut off updates, 0.something better than the SOTA on some benchmark and as compute gets better, cheaper to run at scale.
- Anonyneko 1y ago
  To be fair, the speed of F1 cars is mostly limited by regulations that are meant to make the sport more competitive and entertaining. With fewer restrictions on engines and aerodynamics we could have had much faster cars within a year. But even setting safety issues aside, the insane aero wash would make it nearly impossible to follow another car, let alone overtake it, hence the restrictions and the big "rule resets" every few years that slow down the cars, compensating for all of the tricks the teams have found over that time. (I agree with the general thoughts on the state of LLMs though, just a bit too much into open-wheel cars going vroom vroom in circles for two hours at a time)
- epolanski 1y ago
  > we can see cars haven't become insanely faster than they were 10-20 years ago I don't think this is a good example, because the regulations in F1 are largely focused on slowing the cars. It's an engineering miracle that 2025 cars can be competitive with previous generation cars on so many tracks.

jillesvangurp 1y ago

Year on year, progress is indeed a bit incremental. But seen over a five year period the progress is actually stupidly amazing. In practical terms, Gpt 5 is a nice upgrade over most other models. We'll no doubt get lots of subjective reports how it was wrong or right or worse than some other model for some chats. But my personal (subjective) experience so far is that it just made it possible for me to use codex on more serious projects. It still gets plenty of things wrong. But that's more because of a lack of context than hallucination issues. Context fixes are a lot easier than model improvements. But last week I didn't bother and now I'm getting decent results. I don't really care what version number they slap on things. That is indeed just marketing. And competition is quite fierce so I can understand why they are overselling what could have been just chat gpt 4.2 or whatever. Also discussions about AGI tend to bore me as they seem to escalate into low quality philosophical debates with lots of amateurs rehashing ancient argument poorly. There aren't a hell of a lot new arguments that people come up with at this point. IMHO we don't actually need an AGI to bootstrap the singularity. We just AIs to be good enough to come up with algorithmic optimizations, breakthroughs and improvements at a steady pace. We're getting quite close to that and I wouldn't be surprised to learn that OpenAI's people are already eating their own dogfood in liberal quantities. It's not necessary for AIs to be conscious in order to come up with the improvements that might eventually enable such a thing. I expect the singularity might be more of a phase than a moment. And if you know your boiling frog analogy, we might be smack down in the middle of that already and just not realize it. Five years ago, it was all very theoretical. And now I'm waiting for codex to wrap up a few pull requests that would have distracted me for a week each five years ago. It's taking too long and I'm procrastinating my gained productivity away on HN. But what else is new ;-).

tough 1y ago

there's no more data to throw compute at to 1T tokens is a lot