2 ms·

https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ https://metr.org/blog/2025-03-19-measuring-ai-ability-to-com... He’s wrong we sti

by Herring 11mo ago

https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ https://metr.org/blog/2025-03-19-measuring-ai-ability-to-com... He’s wrong we still scaling, boys.

epistasis 11mo ago
That blog post is eight months old. That feels like pretty old news in the age of AI. Has it held since then?
- conception 11mo ago
  It looks like it’s been updated as it has codex 5.1 max on it
- deleted 11mo ago
  [deleted]
rockinghigh 11mo ago
You should read the transcript. He's including 2025 in the age of scaling. > Maybe here’s another way to put it. Up until 2020, from 2012 to 2020, it was the age of research. Now, from 2020 to 2025, it was the age of scaling—maybe plus or minus, let’s add error bars to those years—because people say, “This is amazing. You’ve got to scale more. Keep scaling.” The one word: scaling. > But now the scale is so big. Is the belief really, “Oh, it’s so big, but if you had 100x more, everything would be so different?” It would be different, for sure. But is the belief that if you just 100x the scale, everything would be transformed? I don’t think that’s true. So it’s back to the age of research again, just with big computers.
- Herring 11mo ago
  Nope, Epoch.ai thinks we have enough to scale till 2030 at least. https://epoch.ai/blog/can-ai-scaling-continue-through-2030 https://epoch.ai/blog/can-ai-scaling-continue-through-2030 ^ /_\ ***
  - deleted 11mo ago
    [deleted]
  - imiric 11mo ago
    That article is from August 2024. A lot has changed since then. Specifically, performance of SOTA models has been reaching a plateau on all popular benchmarks, and this has been especially evident in 2025. This is why every major model announcement shows comparisons relative to other models, but not a historical graph of performance over time. Regardless, benchmarks are far from being a reliable measurement of the capabilities of these tools, and they will continue to be reinvented and gamed, but the asymptote is showing even on their own benchmarks. We can certainly continue to throw more compute at the problem. But the point is that scaling the current generation of tech will continue to have fewer returns. To make up for this, "AI" companies are now focusing on engineering. 2025 has been the year of MCP, "agents", "skills", etc., which will continue in 2026. This is a good thing, as these tools need better engineering around them, so they can deliver actual value. But the hype train is running out of steam, and unless there is a significant breakthrough soon, I suspect that next year will be a turning point in this hype cycle.
    - ojbyrne 11mo ago
      I’m curious how you deduced it’s from 2024. Timestamps on the article and the embedded video are both November 2025.
      - latexr 11mo ago
        It says at the top it was published Aug 20, 2024, and the Internet Archive has it since Nov 13, 2024. https://web.archive.org/web/20241113185615/https://epoch.ai/blog/can-ai-scaling-continue-through-2030 https://web.archive.org/web/20241113185615/https://epoch.ai/...
        ojbyrne 11mo ago
        Sorry, I didn't follow the thread, thought you were referring to the top level article.
rdedev 11mo ago
The 3rd graph is interesting. Once the model performance reaches above human baseline, the growth seems to be logarithmic instead of exponential.
an0malous 11mo ago
“Time it takes for a human to complete a task that AI can complete 50% of the time” seems like a really contrived metric. Suppose it takes 30 minutes to write code to scrape a page and also 30 minutes to identify a bug in a SQL query, an AI’s ability to solve the former has virtually no bearing on its ability to solve the latter but we’re considering them all in the same set of “30 minute problems.” Where do they get the data for task durations anyway?

by Herring 11mo ago

https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/ https://metr.org/blog/2025-03-19-measuring-ai-ability-to-com... He’s wrong we still scaling, boys.

epistasis 11mo ago

That blog post is eight months old. That feels like pretty old news in the age of AI. Has it held since then?

conception 11mo ago
It looks like it’s been updated as it has codex 5.1 max on it
deleted 11mo ago
[deleted]

rockinghigh 11mo ago

You should read the transcript. He's including 2025 in the age of scaling. > Maybe here’s another way to put it. Up until 2020, from 2012 to 2020, it was the age of research. Now, from 2020 to 2025, it was the age of scaling—maybe plus or minus, let’s add error bars to those years—because people say, “This is amazing. You’ve got to scale more. Keep scaling.” The one word: scaling. > But now the scale is so big. Is the belief really, “Oh, it’s so big, but if you had 100x more, everything would be so different?” It would be different, for sure. But is the belief that if you just 100x the scale, everything would be transformed? I don’t think that’s true. So it’s back to the age of research again, just with big computers.

Herring 11mo ago
Nope, Epoch.ai thinks we have enough to scale till 2030 at least. https://epoch.ai/blog/can-ai-scaling-continue-through-2030 https://epoch.ai/blog/can-ai-scaling-continue-through-2030 ^ /_\ ***
- deleted 11mo ago
  [deleted]
- imiric 11mo ago
  That article is from August 2024. A lot has changed since then. Specifically, performance of SOTA models has been reaching a plateau on all popular benchmarks, and this has been especially evident in 2025. This is why every major model announcement shows comparisons relative to other models, but not a historical graph of performance over time. Regardless, benchmarks are far from being a reliable measurement of the capabilities of these tools, and they will continue to be reinvented and gamed, but the asymptote is showing even on their own benchmarks. We can certainly continue to throw more compute at the problem. But the point is that scaling the current generation of tech will continue to have fewer returns. To make up for this, "AI" companies are now focusing on engineering. 2025 has been the year of MCP, "agents", "skills", etc., which will continue in 2026. This is a good thing, as these tools need better engineering around them, so they can deliver actual value. But the hype train is running out of steam, and unless there is a significant breakthrough soon, I suspect that next year will be a turning point in this hype cycle.
  - ojbyrne 11mo ago
    I’m curious how you deduced it’s from 2024. Timestamps on the article and the embedded video are both November 2025.
    - latexr 11mo ago
      It says at the top it was published Aug 20, 2024, and the Internet Archive has it since Nov 13, 2024. https://web.archive.org/web/20241113185615/https://epoch.ai/blog/can-ai-scaling-continue-through-2030 https://web.archive.org/web/20241113185615/https://epoch.ai/...
      - ojbyrne 11mo ago
        Sorry, I didn't follow the thread, thought you were referring to the top level article.

rdedev 11mo ago

The 3rd graph is interesting. Once the model performance reaches above human baseline, the growth seems to be logarithmic instead of exponential.

an0malous 11mo ago

“Time it takes for a human to complete a task that AI can complete 50% of the time” seems like a really contrived metric. Suppose it takes 30 minutes to write code to scrape a page and also 30 minutes to identify a bug in a SQL query, an AI’s ability to solve the former has virtually no bearing on its ability to solve the latter but we’re considering them all in the same set of “30 minute problems.” Where do they get the data for task durations anyway?