3 ms·

This is not a really valuable article. The Apple paper was widely considered as a “well, duh” paper, GPT5 being underwhelming seems to be mostly a cost cutting

This is not a really valuable article. The Apple paper was widely considered as a “well, duh” paper, GPT5 being underwhelming seems to be mostly a cost cutting / supply can’t keep up issue, and the others are just mainly expert opinions. To be clear, I am definitely an AGI skeptic, and I very much believe that our current techniques of neural networks on GPUs is extremely inefficient, but this article doesn’t really add a lot to this discussion; it seems to self congratulate on the insights by a few others.

an0malous 1y ago
I don’t think either of your first two statements are accurate, what is your support for those claims?
p1esk 1y ago
GPT5 compared to the original GPT4 is a huge improvement. It exceeded all my expectations from 2 years ago. I really doubted most GPT4 limitations will be resolved so quickly. If they manage a similar quality jump with GPT6, it will probably meet most reasonable definitions of AGI.
- dns_snek 1y ago
  > GPT5 compared to the original GPT4 is a huge improvement. It exceeded all my expectations from 2 years ago. Cool story. In my experience they're still on the same order of magnitude of usefulness as the original Copilot. Every few months I read about these fantastic ground-breaking improvements and fire up whatever the trendiest toolchain is (most recently Claude Code and Cursor) and walk away less disappointed than last time, but ultimately still disappointed. On simple tasks it doesn't save me any time but on more complex tasks it always makes a huge mess unless I mentor it like a really junior coworker. But if I do that I'm not saving any time and I end up with lower quality, poorly organized code that contains more defects than if I did it myself.
  - p1esk 1y ago
    You probably just forgot how bad GPT4 was. It was failing half of complicated tasks or questions I asked. GPT5 has roughly 90% success rate for me.
    - dns_snek 1y ago
      "Success rate" is a useless metric unless we have a shared definition of what success means. Either we have a categorically different idea of what constitutes a "complicated" task or we have categorically different standards of minimum viable quality. Current SOTA agents are great at gluing together pre-made components but they break down as soon as I try to treat them like mid-level developers by asking them to implement a cross-cutting feature in an idiomatic way. Without junior-level hand-holding it far too often generates a half-broken implementation that looks right at first glance, supported by about 3 dozen tests, most of which are just useless bloat that doesn't test the right things. Sometimes it starts looping on failing tests until it eventually gives up, confidently concluding that the implementation is complete with production-quality code, with whopping 90% test success rate (fabricating a lazy explanation for why the failing tests are outside of its control). Even when the code does work and the tests pass it's poorly designed and poorly structured. There's very little attention paid to future maintainability, even with planning ahead of time. GPT-5, Claude 4.5 Sonnet, it's all the same. "Thinking" or no thinking.
      - p1esk 1y ago
        I don’t really use LLMs for coding - I still enjoy doing it myself. I use them for brainstorming ideas, reviewing papers (I’m a researcher), practical questions (e.g. how to replace a cooling bypass valve in my car), or curiosity questions (e.g. estimate 0-60 mph time for Starship). And I don’t really have any need for agents yet (though I’m sure this will change in a couple of years). I pay 20$/mo each for Claude, Gemini, and ChatGPT. I’ve been playing with language models since ~2014, and I estimate the quality jump from gpt4 to gpt5 to be similar to the gpt3.5 to gpt4 jump. In 5 years we gradually went from almost useless to somewhat useful but unreliable, to incredibly useful and reliable.
stogot 1y ago
What if GPT6 and Claude5 is equally a dud?
- make3 1y ago
  gpt 5 is not a dud though
- tim333 1y ago
  I'm not sure it matters much if they are underwhelming except some investors may be down a little, but such is the nature of investing. I think AGI is coming but more along the lines of what Karpathy was saying "They're cognitively lacking and it's just not working. It will take about a decade to work through all of those issues." I don't get the Marcus - GPT5 isn't AGI so it's game over type stuff. New technologies take time to develop.
- stingraycharles 1y ago
  I mean, they’re definitely not duds, and I personally foresee incremental improvements, not revolutionary. Which is fine.

an0malous 1y ago

I don’t think either of your first two statements are accurate, what is your support for those claims?

p1esk 1y ago

GPT5 compared to the original GPT4 is a huge improvement. It exceeded all my expectations from 2 years ago. I really doubted most GPT4 limitations will be resolved so quickly. If they manage a similar quality jump with GPT6, it will probably meet most reasonable definitions of AGI.

dns_snek 1y ago
> GPT5 compared to the original GPT4 is a huge improvement. It exceeded all my expectations from 2 years ago. Cool story. In my experience they're still on the same order of magnitude of usefulness as the original Copilot. Every few months I read about these fantastic ground-breaking improvements and fire up whatever the trendiest toolchain is (most recently Claude Code and Cursor) and walk away less disappointed than last time, but ultimately still disappointed. On simple tasks it doesn't save me any time but on more complex tasks it always makes a huge mess unless I mentor it like a really junior coworker. But if I do that I'm not saving any time and I end up with lower quality, poorly organized code that contains more defects than if I did it myself.
- p1esk 1y ago
  You probably just forgot how bad GPT4 was. It was failing half of complicated tasks or questions I asked. GPT5 has roughly 90% success rate for me.
  - dns_snek 1y ago
    "Success rate" is a useless metric unless we have a shared definition of what success means. Either we have a categorically different idea of what constitutes a "complicated" task or we have categorically different standards of minimum viable quality. Current SOTA agents are great at gluing together pre-made components but they break down as soon as I try to treat them like mid-level developers by asking them to implement a cross-cutting feature in an idiomatic way. Without junior-level hand-holding it far too often generates a half-broken implementation that looks right at first glance, supported by about 3 dozen tests, most of which are just useless bloat that doesn't test the right things. Sometimes it starts looping on failing tests until it eventually gives up, confidently concluding that the implementation is complete with production-quality code, with whopping 90% test success rate (fabricating a lazy explanation for why the failing tests are outside of its control). Even when the code does work and the tests pass it's poorly designed and poorly structured. There's very little attention paid to future maintainability, even with planning ahead of time. GPT-5, Claude 4.5 Sonnet, it's all the same. "Thinking" or no thinking.
    - p1esk 1y ago
      I don’t really use LLMs for coding - I still enjoy doing it myself. I use them for brainstorming ideas, reviewing papers (I’m a researcher), practical questions (e.g. how to replace a cooling bypass valve in my car), or curiosity questions (e.g. estimate 0-60 mph time for Starship). And I don’t really have any need for agents yet (though I’m sure this will change in a couple of years). I pay 20$/mo each for Claude, Gemini, and ChatGPT. I’ve been playing with language models since ~2014, and I estimate the quality jump from gpt4 to gpt5 to be similar to the gpt3.5 to gpt4 jump. In 5 years we gradually went from almost useless to somewhat useful but unreliable, to incredibly useful and reliable.

stogot 1y ago

What if GPT6 and Claude5 is equally a dud?

make3 1y ago
gpt 5 is not a dud though
tim333 1y ago
I'm not sure it matters much if they are underwhelming except some investors may be down a little, but such is the nature of investing. I think AGI is coming but more along the lines of what Karpathy was saying "They're cognitively lacking and it's just not working. It will take about a decade to work through all of those issues." I don't get the Marcus - GPT5 isn't AGI so it's game over type stuff. New technologies take time to develop.
stingraycharles 1y ago
I mean, they’re definitely not duds, and I personally foresee incremental improvements, not revolutionary. Which is fine.