3 ms·

You probably just forgot how bad GPT4 was. It was failing half of complicated tasks or questions I asked. GPT5 has roughly 90% success rate for me.

by p1esk 1y ago

dns_snek 1y ago
"Success rate" is a useless metric unless we have a shared definition of what success means. Either we have a categorically different idea of what constitutes a "complicated" task or we have categorically different standards of minimum viable quality. Current SOTA agents are great at gluing together pre-made components but they break down as soon as I try to treat them like mid-level developers by asking them to implement a cross-cutting feature in an idiomatic way. Without junior-level hand-holding it far too often generates a half-broken implementation that looks right at first glance, supported by about 3 dozen tests, most of which are just useless bloat that doesn't test the right things. Sometimes it starts looping on failing tests until it eventually gives up, confidently concluding that the implementation is complete with production-quality code, with whopping 90% test success rate (fabricating a lazy explanation for why the failing tests are outside of its control). Even when the code does work and the tests pass it's poorly designed and poorly structured. There's very little attention paid to future maintainability, even with planning ahead of time. GPT-5, Claude 4.5 Sonnet, it's all the same. "Thinking" or no thinking.
- p1esk 1y ago
  I don’t really use LLMs for coding - I still enjoy doing it myself. I use them for brainstorming ideas, reviewing papers (I’m a researcher), practical questions (e.g. how to replace a cooling bypass valve in my car), or curiosity questions (e.g. estimate 0-60 mph time for Starship). And I don’t really have any need for agents yet (though I’m sure this will change in a couple of years). I pay 20$/mo each for Claude, Gemini, and ChatGPT. I’ve been playing with language models since ~2014, and I estimate the quality jump from gpt4 to gpt5 to be similar to the gpt3.5 to gpt4 jump. In 5 years we gradually went from almost useless to somewhat useful but unreliable, to incredibly useful and reliable.
  - orwin 1y ago
    Is this quality caused by an improvement in the LLM, or is it caused by a better inference engine? Because i use Claude code, clearly Sonnet 4.5 is way ahead of Sonnet4 (and much better than Codex Imho), but the reason it's more helpful is that it is better at web search (which, to be clear, is not from LLMs), and has a better inference engine for sure (which avoid painfull loops), and do not silently ignore its context when it find an incoherence. It seems the improvements have nothing to do with AGI, a better inference engine is nice but i mean, i could run prolog program 30 years ago. To me, most of the improvements i've noticed do not fill me with the belief AGI is 3 years away. Worse, i now think that while LLMs will be usefull for AGI, like perceptron will be, clearly tehy won't be the base. I'm no expert to be clear: i've only worked professionnally with perceptrons for OCRs, and only 6 months, 10 years ago, so my opinion have no real value, but it's a datapoint for people who are experienced with the subject.
    - int_19h 1y ago
      What do you mean by "inference engine" here?

You probably just forgot how bad GPT4 was. It was failing half of complicated tasks or questions I asked. GPT5 has roughly 90% success rate for me.

by p1esk 1y ago

You probably just forgot how bad GPT4 was. It was failing half of complicated tasks or questions I asked. GPT5 has roughly 90% success rate for me.

dns_snek 1y ago

"Success rate" is a useless metric unless we have a shared definition of what success means. Either we have a categorically different idea of what constitutes a "complicated" task or we have categorically different standards of minimum viable quality. Current SOTA agents are great at gluing together pre-made components but they break down as soon as I try to treat them like mid-level developers by asking them to implement a cross-cutting feature in an idiomatic way. Without junior-level hand-holding it far too often generates a half-broken implementation that looks right at first glance, supported by about 3 dozen tests, most of which are just useless bloat that doesn't test the right things. Sometimes it starts looping on failing tests until it eventually gives up, confidently concluding that the implementation is complete with production-quality code, with whopping 90% test success rate (fabricating a lazy explanation for why the failing tests are outside of its control). Even when the code does work and the tests pass it's poorly designed and poorly structured. There's very little attention paid to future maintainability, even with planning ahead of time. GPT-5, Claude 4.5 Sonnet, it's all the same. "Thinking" or no thinking.

p1esk 1y ago
I don’t really use LLMs for coding - I still enjoy doing it myself. I use them for brainstorming ideas, reviewing papers (I’m a researcher), practical questions (e.g. how to replace a cooling bypass valve in my car), or curiosity questions (e.g. estimate 0-60 mph time for Starship). And I don’t really have any need for agents yet (though I’m sure this will change in a couple of years). I pay 20$/mo each for Claude, Gemini, and ChatGPT. I’ve been playing with language models since ~2014, and I estimate the quality jump from gpt4 to gpt5 to be similar to the gpt3.5 to gpt4 jump. In 5 years we gradually went from almost useless to somewhat useful but unreliable, to incredibly useful and reliable.
- orwin 1y ago
  Is this quality caused by an improvement in the LLM, or is it caused by a better inference engine? Because i use Claude code, clearly Sonnet 4.5 is way ahead of Sonnet4 (and much better than Codex Imho), but the reason it's more helpful is that it is better at web search (which, to be clear, is not from LLMs), and has a better inference engine for sure (which avoid painfull loops), and do not silently ignore its context when it find an incoherence. It seems the improvements have nothing to do with AGI, a better inference engine is nice but i mean, i could run prolog program 30 years ago. To me, most of the improvements i've noticed do not fill me with the belief AGI is 3 years away. Worse, i now think that while LLMs will be usefull for AGI, like perceptron will be, clearly tehy won't be the base. I'm no expert to be clear: i've only worked professionnally with perceptrons for OCRs, and only 6 months, 10 years ago, so my opinion have no real value, but it's a datapoint for people who are experienced with the subject.
  - int_19h 1y ago
    What do you mean by "inference engine" here?