4 ms·

Can you quantify this rate of progress? Because someone always comes around and says there's been exponential progress in the past <short timeline> every time s

by malfist 3mo ago

Can you quantify this rate of progress? Because someone always comes around and says there's been exponential progress in the past <short timeline> every time someone complains that models just aren't very good. Both can't be true

Certhas 3mo ago
In research we are still seeing massive jumps. Subjects that LLMs were completely useless for half a year ago are now definitely in scope. And there are benchmarks that cleanly separate the SOTA models: https://epoch.ai/MirrorCode https://epoch.ai/MirrorCode Saturation of benchmarks is a property of benchmarks just as much as of the models.
jake-coworker 3mo ago
one of these claimed is backed by data, one is backed by anecdotes, you can decide which you trust more
- malfist 3mo ago
  So far, neither is backed by data.
  - astro1234 3mo ago
    I don’t think that’s fair to say, there are two principle sources of data that paint a fairly consistent picture - one is scaling laws, where we found years ago that pretraining validation loss scales in an almost miraculously predictable way with data volume and compute. There are apparently theoretical bases for this that I don’t quite understand but this property alone is holding at every scale we’ve ever tested. There is not just “one” scaling law but the point is there are scaling laws and they continue to faithfully predict the performance gains we see - one is benchmarks, which I always point to epoch capability index as a good summary of them in aggregate which makes it nice to plot on one curve the capability improvement over time To me either one without the other is substantially weaker, the fact that theory and empirical measurements give you a very good scaling law on a more unintuitive quantity (pretraining validation loss) that’s only indirectly related to the downstream performance you care about, benchmarks (in aggregate) are more direct measures of downstream performance but are harder to nail down clean and well motivated “laws” from theory (as far as I can tell). Nevertheless we do in fact see a clear trend that is not slowing. That doesn’t mean there aren’t a whole host of benchmark problems that don’t impact the numbers involved here (leakage from training data, fundamental flaws in the design, benchmaxxing) but they don’t change the larger story. These problems don’t plausibly explain the clean trends we see.
NyxWulf 3mo ago
Both can be true, because the experience depends on the skill of the user. The article the other day here on HN that LLMs reward skill is my exact experience. If you are are good at what you are trying to use it for they can be a skill amplifier, and they are definitely getting much better rapidly for the work I do with them. At the same time people are complaining that they are getting dumber. Saying that both can't be true ignores the skill requirement to use them and the completely different perspectives of people using them. Even if both aren't true, your evidence was people saying two opposing things. The truth (if there is a single objective truth on a given thing) has little bearing on whether or not different people agree on it.
- totallygeeky 3mo ago
  "Skill" with an LLM is nonsense. It's non-deterministic, so any equal efforts are not to be given equal outcome.
  - farbklang 3mo ago
    not equal, but probabisticly better. Best example: give the agent a tight spec and it will perform better compared with a spec that leaves room for interpretation. This is true for all models, more or less. (purely anecdotal of course)
  - whatisthiseven 3mo ago
    You could say the same when applied to games with high RNG and chance, such as Slay the Spire 2, and yet those with real skill do perform far better than those without. Those with skill can clear the highest difficulties more often than those with lower skill. Something being "non-deterministic" is orthogonal to whether or not skill plays a role.
  - AbrahamParangi 3mo ago
    "Skill" at poker is nonsense. It's non-deterministic, so any equal efforts are not to be given equal outcome.
  - sdenton4 3mo ago

by malfist 3mo ago

Certhas 3mo ago

In research we are still seeing massive jumps. Subjects that LLMs were completely useless for half a year ago are now definitely in scope. And there are benchmarks that cleanly separate the SOTA models: https://epoch.ai/MirrorCode https://epoch.ai/MirrorCode Saturation of benchmarks is a property of benchmarks just as much as of the models.

jake-coworker 3mo ago

one of these claimed is backed by data, one is backed by anecdotes, you can decide which you trust more

malfist 3mo ago
So far, neither is backed by data.
- astro1234 3mo ago
  I don’t think that’s fair to say, there are two principle sources of data that paint a fairly consistent picture - one is scaling laws, where we found years ago that pretraining validation loss scales in an almost miraculously predictable way with data volume and compute. There are apparently theoretical bases for this that I don’t quite understand but this property alone is holding at every scale we’ve ever tested. There is not just “one” scaling law but the point is there are scaling laws and they continue to faithfully predict the performance gains we see - one is benchmarks, which I always point to epoch capability index as a good summary of them in aggregate which makes it nice to plot on one curve the capability improvement over time To me either one without the other is substantially weaker, the fact that theory and empirical measurements give you a very good scaling law on a more unintuitive quantity (pretraining validation loss) that’s only indirectly related to the downstream performance you care about, benchmarks (in aggregate) are more direct measures of downstream performance but are harder to nail down clean and well motivated “laws” from theory (as far as I can tell). Nevertheless we do in fact see a clear trend that is not slowing. That doesn’t mean there aren’t a whole host of benchmark problems that don’t impact the numbers involved here (leakage from training data, fundamental flaws in the design, benchmaxxing) but they don’t change the larger story. These problems don’t plausibly explain the clean trends we see.

NyxWulf 3mo ago

Both can be true, because the experience depends on the skill of the user. The article the other day here on HN that LLMs reward skill is my exact experience. If you are are good at what you are trying to use it for they can be a skill amplifier, and they are definitely getting much better rapidly for the work I do with them. At the same time people are complaining that they are getting dumber. Saying that both can't be true ignores the skill requirement to use them and the completely different perspectives of people using them. Even if both aren't true, your evidence was people saying two opposing things. The truth (if there is a single objective truth on a given thing) has little bearing on whether or not different people agree on it.

totallygeeky 3mo ago
"Skill" with an LLM is nonsense. It's non-deterministic, so any equal efforts are not to be given equal outcome.
- farbklang 3mo ago
  not equal, but probabisticly better. Best example: give the agent a tight spec and it will perform better compared with a spec that leaves room for interpretation. This is true for all models, more or less. (purely anecdotal of course)
- whatisthiseven 3mo ago
  You could say the same when applied to games with high RNG and chance, such as Slay the Spire 2, and yet those with real skill do perform far better than those without. Those with skill can clear the highest difficulties more often than those with lower skill. Something being "non-deterministic" is orthogonal to whether or not skill plays a role.
- AbrahamParangi 3mo ago
  "Skill" at poker is nonsense. It's non-deterministic, so any equal efforts are not to be given equal outcome.
- sdenton4 3mo ago