4 ms·

GPT5 compared to the original GPT4 is a huge improvement. It exceeded all my expectations from 2 years ago. I really doubted most GPT4 limitations will be resol

by p1esk 1y ago

GPT5 compared to the original GPT4 is a huge improvement. It exceeded all my expectations from 2 years ago. I really doubted most GPT4 limitations will be resolved so quickly. If they manage a similar quality jump with GPT6, it will probably meet most reasonable definitions of AGI.

dns_snek 1y ago
> GPT5 compared to the original GPT4 is a huge improvement. It exceeded all my expectations from 2 years ago. Cool story. In my experience they're still on the same order of magnitude of usefulness as the original Copilot. Every few months I read about these fantastic ground-breaking improvements and fire up whatever the trendiest toolchain is (most recently Claude Code and Cursor) and walk away less disappointed than last time, but ultimately still disappointed. On simple tasks it doesn't save me any time but on more complex tasks it always makes a huge mess unless I mentor it like a really junior coworker. But if I do that I'm not saving any time and I end up with lower quality, poorly organized code that contains more defects than if I did it myself.
- p1esk 1y ago
  You probably just forgot how bad GPT4 was. It was failing half of complicated tasks or questions I asked. GPT5 has roughly 90% success rate for me.
  - dns_snek 1y ago
    "Success rate" is a useless metric unless we have a shared definition of what success means. Either we have a categorically different idea of what constitutes a "complicated" task or we have categorically different standards of minimum viable quality. Current SOTA agents are great at gluing together pre-made components but they break down as soon as I try to treat them like mid-level developers by asking them to implement a cross-cutting feature in an idiomatic way. Without junior-level hand-holding it far too often generates a half-broken implementation that looks right at first glance, supported by about 3 dozen tests, most of which are just useless bloat that doesn't test the right things. Sometimes it starts looping on failing tests until it eventually gives up, confidently concluding that the implementation is complete with production-quality code, with whopping 90% test success rate (fabricating a lazy explanation for why the failing tests are outside of its control). Even when the code does work and the tests pass it's poorly designed and poorly structured. There's very little attention paid to future maintainability, even with planning ahead of time. GPT-5, Claude 4.5 Sonnet, it's all the same. "Thinking" or no thinking.
    - p1esk 1y ago
      I don’t really use LLMs for coding - I still enjoy doing it myself. I use them for brainstorming ideas, reviewing papers (I’m a researcher), practical questions (e.g. how to replace a cooling bypass valve in my car), or curiosity questions (e.g. estimate 0-60 mph time for Starship). And I don’t really have any need for agents yet (though I’m sure this will change in a couple of years). I pay 20$/mo each for Claude, Gemini, and ChatGPT. I’ve been playing with language models since ~2014, and I estimate the quality jump from gpt4 to gpt5 to be similar to the gpt3.5 to gpt4 jump. In 5 years we gradually went from almost useless to somewhat useful but unreliable, to incredibly useful and reliable.
      - orwin 1y ago
        Is this quality caused by an improvement in the LLM, or is it caused by a better inference engine? Because i use Claude code, clearly Sonnet 4.5 is way ahead of Sonnet4 (and much better than Codex Imho), but the reason it's more helpful is that it is better at web search (which, to be clear, is not from LLMs), and has a better inference engine for sure (which avoid painfull loops), and do not silently ignore its context when it find an incoherence. It seems the improvements have nothing to do with AGI, a better inference engine is nice but i mean, i could run prolog program 30 years ago. To me, most of the improvements i've noticed do not fill me with the belief AGI is 3 years away. Worse, i now think that while LLMs will be usefull for AGI, like perceptron will be, clearly tehy won't be the base. I'm no expert to be clear: i've only worked professionnally with perceptrons for OCRs, and only 6 months, 10 years ago, so my opinion have no real value, but it's a datapoint for people who are experienced with the subject.
        int_19h 1y ago
        What do you mean by "inference engine" here?

by p1esk 1y ago

dns_snek 1y ago

> GPT5 compared to the original GPT4 is a huge improvement. It exceeded all my expectations from 2 years ago. Cool story. In my experience they're still on the same order of magnitude of usefulness as the original Copilot. Every few months I read about these fantastic ground-breaking improvements and fire up whatever the trendiest toolchain is (most recently Claude Code and Cursor) and walk away less disappointed than last time, but ultimately still disappointed. On simple tasks it doesn't save me any time but on more complex tasks it always makes a huge mess unless I mentor it like a really junior coworker. But if I do that I'm not saving any time and I end up with lower quality, poorly organized code that contains more defects than if I did it myself.

p1esk 1y ago
You probably just forgot how bad GPT4 was. It was failing half of complicated tasks or questions I asked. GPT5 has roughly 90% success rate for me.
- dns_snek 1y ago
  "Success rate" is a useless metric unless we have a shared definition of what success means. Either we have a categorically different idea of what constitutes a "complicated" task or we have categorically different standards of minimum viable quality. Current SOTA agents are great at gluing together pre-made components but they break down as soon as I try to treat them like mid-level developers by asking them to implement a cross-cutting feature in an idiomatic way. Without junior-level hand-holding it far too often generates a half-broken implementation that looks right at first glance, supported by about 3 dozen tests, most of which are just useless bloat that doesn't test the right things. Sometimes it starts looping on failing tests until it eventually gives up, confidently concluding that the implementation is complete with production-quality code, with whopping 90% test success rate (fabricating a lazy explanation for why the failing tests are outside of its control). Even when the code does work and the tests pass it's poorly designed and poorly structured. There's very little attention paid to future maintainability, even with planning ahead of time. GPT-5, Claude 4.5 Sonnet, it's all the same. "Thinking" or no thinking.
  - p1esk 1y ago
    I don’t really use LLMs for coding - I still enjoy doing it myself. I use them for brainstorming ideas, reviewing papers (I’m a researcher), practical questions (e.g. how to replace a cooling bypass valve in my car), or curiosity questions (e.g. estimate 0-60 mph time for Starship). And I don’t really have any need for agents yet (though I’m sure this will change in a couple of years). I pay 20$/mo each for Claude, Gemini, and ChatGPT. I’ve been playing with language models since ~2014, and I estimate the quality jump from gpt4 to gpt5 to be similar to the gpt3.5 to gpt4 jump. In 5 years we gradually went from almost useless to somewhat useful but unreliable, to incredibly useful and reliable.
    - orwin 1y ago
      Is this quality caused by an improvement in the LLM, or is it caused by a better inference engine? Because i use Claude code, clearly Sonnet 4.5 is way ahead of Sonnet4 (and much better than Codex Imho), but the reason it's more helpful is that it is better at web search (which, to be clear, is not from LLMs), and has a better inference engine for sure (which avoid painfull loops), and do not silently ignore its context when it find an incoherence. It seems the improvements have nothing to do with AGI, a better inference engine is nice but i mean, i could run prolog program 30 years ago. To me, most of the improvements i've noticed do not fill me with the belief AGI is 3 years away. Worse, i now think that while LLMs will be usefull for AGI, like perceptron will be, clearly tehy won't be the base. I'm no expert to be clear: i've only worked professionnally with perceptrons for OCRs, and only 6 months, 10 years ago, so my opinion have no real value, but it's a datapoint for people who are experienced with the subject.
      - int_19h 1y ago
        What do you mean by "inference engine" here?