5 ms·

>This seems to be the end of the road for LLM's This is an amazingly ignorant thing to say given the current pace of progress.

Jensson 2mo ago
There is high rate of progress in specific domains, not high rate of progress in generalness. The models haven't gotten generally smarter, for things they didn't focus on the models are just as bad as a year ago.
- naasking 2mo ago
  I don't think that's correct. Increasing parameter count increases capabilities in all domains per the scaling laws. Models are larger than they were years ago, so capabilities in all domains must necessarily be better. This doesn't even account for better training data, which has also much improved.
runarberg 2mo ago
And sales of disco records were up 400% for the year ending 1976. If these trends continue...
malfist 2mo ago
Can you quantify this rate of progress? Because someone always comes around and says there's been exponential progress in the past <short timeline> every time someone complains that models just aren't very good. Both can't be true
- Certhas 2mo ago
  In research we are still seeing massive jumps. Subjects that LLMs were completely useless for half a year ago are now definitely in scope. And there are benchmarks that cleanly separate the SOTA models: https://epoch.ai/MirrorCode https://epoch.ai/MirrorCode Saturation of benchmarks is a property of benchmarks just as much as of the models.
- jake-coworker 2mo ago
  one of these claimed is backed by data, one is backed by anecdotes, you can decide which you trust more
  - malfist 2mo ago
    So far, neither is backed by data.
    - astro1234 2mo ago
      I don’t think that’s fair to say, there are two principle sources of data that paint a fairly consistent picture - one is scaling laws, where we found years ago that pretraining validation loss scales in an almost miraculously predictable way with data volume and compute. There are apparently theoretical bases for this that I don’t quite understand but this property alone is holding at every scale we’ve ever tested. There is not just “one” scaling law but the point is there are scaling laws and they continue to faithfully predict the performance gains we see - one is benchmarks, which I always point to epoch capability index as a good summary of them in aggregate which makes it nice to plot on one curve the capability improvement over time To me either one without the other is substantially weaker, the fact that theory and empirical measurements give you a very good scaling law on a more unintuitive quantity (pretraining validation loss) that’s only indirectly related to the downstream performance you care about, benchmarks (in aggregate) are more direct measures of downstream performance but are harder to nail down clean and well motivated “laws” from theory (as far as I can tell). Nevertheless we do in fact see a clear trend that is not slowing. That doesn’t mean there aren’t a whole host of benchmark problems that don’t impact the numbers involved here (leakage from training data, fundamental flaws in the design, benchmaxxing) but they don’t change the larger story. These problems don’t plausibly explain the clean trends we see.
- NyxWulf 2mo ago
  Both can be true, because the experience depends on the skill of the user. The article the other day here on HN that LLMs reward skill is my exact experience. If you are are good at what you are trying to use it for they can be a skill amplifier, and they are definitely getting much better rapidly for the work I do with them. At the same time people are complaining that they are getting dumber. Saying that both can't be true ignores the skill requirement to use them and the completely different perspectives of people using them. Even if both aren't true, your evidence was people saying two opposing things. The truth (if there is a single objective truth on a given thing) has little bearing on whether or not different people agree on it.
  - totallygeeky 2mo ago
    "Skill" with an LLM is nonsense. It's non-deterministic, so any equal efforts are not to be given equal outcome.
    - farbklang 2mo ago
      not equal, but probabisticly better. Best example: give the agent a tight spec and it will perform better compared with a spec that leaves room for interpretation. This is true for all models, more or less. (purely anecdotal of course)
    - whatisthiseven 2mo ago
      You could say the same when applied to games with high RNG and chance, such as Slay the Spire 2, and yet those with real skill do perform far better than those without. Those with skill can clear the highest difficulties more often than those with lower skill. Something being "non-deterministic" is orthogonal to whether or not skill plays a role.
    - AbrahamParangi 2mo ago
      "Skill" at poker is nonsense. It's non-deterministic, so any equal efforts are not to be given equal outcome.
    - sdenton4 2mo ago
pasquinelli 2mo ago
unless you assume endless progress, fast pace of progress will get you quicker to the plateau.
deleted 2mo ago
[deleted]
airstrike 2mo ago
Shallow dismissals like this one are against HN guidelines because they make for very poor discourse. We all learned more from the prior comment than we did from this one.
vor_ 2mo ago
What is the current pace of progress?
827a 2mo ago
I'm very bullish on AI, but I still feel that we pretty much plateaued at Opus 4.6 and everything since then has been in the domain of "extremely verifiable and loopable tasks" (math), benchmarkmaxxing, and harness improvements. Which are fine to good things, but I think its very reasonable at this point to start asking questions about when we'll see progress in more general domains. The readability of AI output, for example, has nosedived as they've gotten more intelligent, which makes the frontier models difficult to use even for things like writing emails. In that sense, the frontier models are going to quickly blaze past any semblance of usefulness to humans, while every once in a while we get a news drop like "GPT-7 solved some crazy math problem" or "it invented some new awesome drug"; meanwhile what most people will use will be smaller, more human-specialized models, maybe distilled from those frontier models, that take much longer to iterate on because they rely on large amounts of human feedback in the domain they're specialized for. In other words, useful progress will probably slow down and become more linear starting in Q4, bounded by the rate at which the humans paying for it say "yes this is a good react website". (By the way: I earnestly do categorize "inventing a new drug" as non-useful AI progress, counter-intuitively. The drug industry has more ideas for drugs than they know what to do with; "useful progress" is, after the idea is made, validating that it works in humans and doesn't kill the human, and productionizing it. AI will help with this and does, but I have substantial doubt that we'll ever see the drug pipeline speed up to, like, a year from idea to prescription. That would be useful progress, which unfortunately many AI pilled hypermaxers conveniently forget. The invention of a promising new drug, or the solution to an arcane set theory problem, are cherries that, through the diligent labor of humans and AI, may become useful, but progress is rarely made by the lone intellect having an a-ha moment.)
- cracell 2mo ago
  You must have different use cases than me. I find even Deepseek Flash v4 0731 even outperforms Opus for me (at 10x the speed two). Using Sol, Fable, Kimi 3 and other recent models has been unbelievable for me. I didn’t think we’d get to this level for years. I’m using them for Ruby, TypeScript and Python. In large existing codebases but also lots of tiny tools.
  - 2mo ago

Jensson 2mo ago

There is high rate of progress in specific domains, not high rate of progress in generalness. The models haven't gotten generally smarter, for things they didn't focus on the models are just as bad as a year ago.

naasking 2mo ago
I don't think that's correct. Increasing parameter count increases capabilities in all domains per the scaling laws. Models are larger than they were years ago, so capabilities in all domains must necessarily be better. This doesn't even account for better training data, which has also much improved.

runarberg 2mo ago

And sales of disco records were up 400% for the year ending 1976. If these trends continue...

malfist 2mo ago

Can you quantify this rate of progress? Because someone always comes around and says there's been exponential progress in the past <short timeline> every time someone complains that models just aren't very good. Both can't be true

Certhas 2mo ago
In research we are still seeing massive jumps. Subjects that LLMs were completely useless for half a year ago are now definitely in scope. And there are benchmarks that cleanly separate the SOTA models: https://epoch.ai/MirrorCode https://epoch.ai/MirrorCode Saturation of benchmarks is a property of benchmarks just as much as of the models.
jake-coworker 2mo ago
one of these claimed is backed by data, one is backed by anecdotes, you can decide which you trust more
- malfist 2mo ago
  So far, neither is backed by data.
  - astro1234 2mo ago
    I don’t think that’s fair to say, there are two principle sources of data that paint a fairly consistent picture - one is scaling laws, where we found years ago that pretraining validation loss scales in an almost miraculously predictable way with data volume and compute. There are apparently theoretical bases for this that I don’t quite understand but this property alone is holding at every scale we’ve ever tested. There is not just “one” scaling law but the point is there are scaling laws and they continue to faithfully predict the performance gains we see - one is benchmarks, which I always point to epoch capability index as a good summary of them in aggregate which makes it nice to plot on one curve the capability improvement over time To me either one without the other is substantially weaker, the fact that theory and empirical measurements give you a very good scaling law on a more unintuitive quantity (pretraining validation loss) that’s only indirectly related to the downstream performance you care about, benchmarks (in aggregate) are more direct measures of downstream performance but are harder to nail down clean and well motivated “laws” from theory (as far as I can tell). Nevertheless we do in fact see a clear trend that is not slowing. That doesn’t mean there aren’t a whole host of benchmark problems that don’t impact the numbers involved here (leakage from training data, fundamental flaws in the design, benchmaxxing) but they don’t change the larger story. These problems don’t plausibly explain the clean trends we see.
NyxWulf 2mo ago
Both can be true, because the experience depends on the skill of the user. The article the other day here on HN that LLMs reward skill is my exact experience. If you are are good at what you are trying to use it for they can be a skill amplifier, and they are definitely getting much better rapidly for the work I do with them. At the same time people are complaining that they are getting dumber. Saying that both can't be true ignores the skill requirement to use them and the completely different perspectives of people using them. Even if both aren't true, your evidence was people saying two opposing things. The truth (if there is a single objective truth on a given thing) has little bearing on whether or not different people agree on it.
- totallygeeky 2mo ago
  "Skill" with an LLM is nonsense. It's non-deterministic, so any equal efforts are not to be given equal outcome.
  - farbklang 2mo ago
    not equal, but probabisticly better. Best example: give the agent a tight spec and it will perform better compared with a spec that leaves room for interpretation. This is true for all models, more or less. (purely anecdotal of course)
  - whatisthiseven 2mo ago
    You could say the same when applied to games with high RNG and chance, such as Slay the Spire 2, and yet those with real skill do perform far better than those without. Those with skill can clear the highest difficulties more often than those with lower skill. Something being "non-deterministic" is orthogonal to whether or not skill plays a role.
  - AbrahamParangi 2mo ago
    "Skill" at poker is nonsense. It's non-deterministic, so any equal efforts are not to be given equal outcome.
  - sdenton4 2mo ago

pasquinelli 2mo ago

unless you assume endless progress, fast pace of progress will get you quicker to the plateau.

deleted 2mo ago

[deleted]

airstrike 2mo ago

Shallow dismissals like this one are against HN guidelines because they make for very poor discourse. We all learned more from the prior comment than we did from this one.

vor_ 2mo ago

What is the current pace of progress?

827a 2mo ago

I'm very bullish on AI, but I still feel that we pretty much plateaued at Opus 4.6 and everything since then has been in the domain of "extremely verifiable and loopable tasks" (math), benchmarkmaxxing, and harness improvements. Which are fine to good things, but I think its very reasonable at this point to start asking questions about when we'll see progress in more general domains. The readability of AI output, for example, has nosedived as they've gotten more intelligent, which makes the frontier models difficult to use even for things like writing emails. In that sense, the frontier models are going to quickly blaze past any semblance of usefulness to humans, while every once in a while we get a news drop like "GPT-7 solved some crazy math problem" or "it invented some new awesome drug"; meanwhile what most people will use will be smaller, more human-specialized models, maybe distilled from those frontier models, that take much longer to iterate on because they rely on large amounts of human feedback in the domain they're specialized for. In other words, useful progress will probably slow down and become more linear starting in Q4, bounded by the rate at which the humans paying for it say "yes this is a good react website". (By the way: I earnestly do categorize "inventing a new drug" as non-useful AI progress, counter-intuitively. The drug industry has more ideas for drugs than they know what to do with; "useful progress" is, after the idea is made, validating that it works in humans and doesn't kill the human, and productionizing it. AI will help with this and does, but I have substantial doubt that we'll ever see the drug pipeline speed up to, like, a year from idea to prescription. That would be useful progress, which unfortunately many AI pilled hypermaxers conveniently forget. The invention of a promising new drug, or the solution to an arcane set theory problem, are cherries that, through the diligent labor of humans and AI, may become useful, but progress is rarely made by the lone intellect having an a-ha moment.)

cracell 2mo ago
You must have different use cases than me. I find even Deepseek Flash v4 0731 even outperforms Opus for me (at 10x the speed two). Using Sol, Fable, Kimi 3 and other recent models has been unbelievable for me. I didn’t think we’d get to this level for years. I’m using them for Ruby, TypeScript and Python. In large existing codebases but also lots of tiny tools.
- 2mo ago