3 ms·

I am constantly astonished that articles like this even pass the smell test. It is not rational to predict exponential growth just because you've seen exponenti

I am constantly astonished that articles like this even pass the smell test. It is not rational to predict exponential growth just because you've seen exponential growth before! Incidentally, that is not what people did during COVID, they predicted exponential growth for reasons. Specific, articulable reasons, that consisted of more than just "look, like go up. line go up more?". Incidentally, the benchmarks quoted are extremely dubious. They do not even really make sense. "The length of tasks AI can do is doubling every 7 months". Seriously, what does that mean? If the AI suddenly took double the time to answer the same question, that would not be progress. Indeed, that isn't what they did, they just... picked some times at random? You might counter that these are actually human completion times, but then why are we comparing such distinct and unrelated tasks as "count words in a passage" (trivial, any child can do) and "train adversarially robust image model" (expert-level task, could take anywhere between an hour and never-complete). Honestly, the most hilarious line in the article is probably this one: > You might object that this plot looks like it might be levelling off, but this is probably mostly an artefact of GPT-5 being very consumer-focused. This is a plot with three points in it! You might as well be looking at tea leaves!

jsnell 1y ago
> but then why are we comparing such distinct and unrelated tasks as ... Because a few years ago the LLMs could only do trivial tasks that a child could do, and now they're able to do complex research and software development tasks. If you just have the trivial tasks, the benchmark is saturated within a year. If you just have the very complex tasks, the benchmark is has no sensitivity at all for years (just everything scoring a 0) and then abruptly becomes useful for a brief moment. This seems pretty obvious, and I can't figure out what your actual concern is. You're just implying it is a flawed design without pointing out anything concrete.
- danlitt 1y ago
  The key word is "unrelated"! Being able to count the number of words in a paragraph and being able to train an image classifier are so different as to be unrelated for all practical purposes. The assumption underlying this kind of a "benchmark" is that all tasks have a certain attribute called complexity which is a numerical value we can use to discriminate tasks, presumably so that if you can complete tasks up to a certain "complexity" then you can complete all other tasks of lower complexity. No such attribute exists! I am sure there are "4 hour" tasks an LLM can do and "5 second" tasks that no LLM can do. The underlying frustration here is that there is so much latitude possible in choosing which tasks to test, which ones to present, and how to quantify "success" that the metrics given are completely meaningless, and do not help anyone to make a prediction. I would bet my entire life savings that by the time the hype bubble bursts, we will still have 10 brainless articles per day coming out saying AGI is round the corner.
  - Voultapher 1y ago
    Well put, the metric is cherry picked to further the narrative.
oofbey 1y ago
"It’s Difficult to Make Predictions, Especially About the Future" - Yogi Berra. It's funny because it's true. So if you want to try to do this difficult task, because say there's billions of dollars and millions of people's livelihoods on the line, how do you do it? Gather a bunch of data, and see if there's some trend? Then maybe it makes sense to extrapolate. Seems pretty reasonable to me. Definitely passes the sniff test. Not sure why you think "line go up more" is such a stupid concept.
- danlitt 1y ago
  It's a stupid concept because it's behind every ponzi scheme.
  - oofbey 1y ago
    In Ponzi schemes the numbers are generally faked.
    - athrowaway3z 1y ago
      Isn't one of those scalers getting money from NVIDIA to buy NVIDIA cards which they use as collateral to borrow more money to buy more NVIDIA cards which NVIDIA put up as revenue which bumps the stock price which they invest into OpenAI which invests into Oracle which buys more NVIDIA cards? Its not a Ponzi scheme, and I don't have a crystal ball to determine where supply and demand will best meet, but a lot seems to be riding on the promise of future demand selling at a premium. I'm not yet ready to believe this is the thing that permanently breaks supply and demand. More compute demand is likely, but the state of the art: resale-users, providers, and suppliers, will all get hit with more competition.
    - danlitt 1y ago
      The numbers are fake, but the returns are real, until they aren't. If all you go off is past performance then you will fall for any scam.
lou1306 1y ago
> The length of tasks AI can do is doubling every 7 months The claim is "At time t0, an AI can solve a task that would take a human 2 minutes. At time t0+dt, they can solve 4-minutes tasks. At time t0+2dt, it's 8 minutes" and so on. I still find these claims extremely dubious, just wanted to clarify.
- deleted 1y ago
  [deleted]
- danlitt 1y ago
  Yes, I get that, I did allow for it in my original comment. I remain convinced this is a gibberish metric - there is probably no such thing as "a task that would take a human 2 minutes", and certainly no such thing as "an AI that can do every task that would take a human 2 minutes".

jsnell 1y ago

> but then why are we comparing such distinct and unrelated tasks as ... Because a few years ago the LLMs could only do trivial tasks that a child could do, and now they're able to do complex research and software development tasks. If you just have the trivial tasks, the benchmark is saturated within a year. If you just have the very complex tasks, the benchmark is has no sensitivity at all for years (just everything scoring a 0) and then abruptly becomes useful for a brief moment. This seems pretty obvious, and I can't figure out what your actual concern is. You're just implying it is a flawed design without pointing out anything concrete.

danlitt 1y ago
The key word is "unrelated"! Being able to count the number of words in a paragraph and being able to train an image classifier are so different as to be unrelated for all practical purposes. The assumption underlying this kind of a "benchmark" is that all tasks have a certain attribute called complexity which is a numerical value we can use to discriminate tasks, presumably so that if you can complete tasks up to a certain "complexity" then you can complete all other tasks of lower complexity. No such attribute exists! I am sure there are "4 hour" tasks an LLM can do and "5 second" tasks that no LLM can do. The underlying frustration here is that there is so much latitude possible in choosing which tasks to test, which ones to present, and how to quantify "success" that the metrics given are completely meaningless, and do not help anyone to make a prediction. I would bet my entire life savings that by the time the hype bubble bursts, we will still have 10 brainless articles per day coming out saying AGI is round the corner.
- Voultapher 1y ago
  Well put, the metric is cherry picked to further the narrative.

oofbey 1y ago

"It’s Difficult to Make Predictions, Especially About the Future" - Yogi Berra. It's funny because it's true. So if you want to try to do this difficult task, because say there's billions of dollars and millions of people's livelihoods on the line, how do you do it? Gather a bunch of data, and see if there's some trend? Then maybe it makes sense to extrapolate. Seems pretty reasonable to me. Definitely passes the sniff test. Not sure why you think "line go up more" is such a stupid concept.

danlitt 1y ago
It's a stupid concept because it's behind every ponzi scheme.
- oofbey 1y ago
  In Ponzi schemes the numbers are generally faked.
  - athrowaway3z 1y ago
    Isn't one of those scalers getting money from NVIDIA to buy NVIDIA cards which they use as collateral to borrow more money to buy more NVIDIA cards which NVIDIA put up as revenue which bumps the stock price which they invest into OpenAI which invests into Oracle which buys more NVIDIA cards? Its not a Ponzi scheme, and I don't have a crystal ball to determine where supply and demand will best meet, but a lot seems to be riding on the promise of future demand selling at a premium. I'm not yet ready to believe this is the thing that permanently breaks supply and demand. More compute demand is likely, but the state of the art: resale-users, providers, and suppliers, will all get hit with more competition.
  - danlitt 1y ago
    The numbers are fake, but the returns are real, until they aren't. If all you go off is past performance then you will fall for any scam.

lou1306 1y ago

> The length of tasks AI can do is doubling every 7 months The claim is "At time t0, an AI can solve a task that would take a human 2 minutes. At time t0+dt, they can solve 4-minutes tasks. At time t0+2dt, it's 8 minutes" and so on. I still find these claims extremely dubious, just wanted to clarify.

deleted 1y ago
[deleted]
danlitt 1y ago
Yes, I get that, I did allow for it in my original comment. I remain convinced this is a gibberish metric - there is probably no such thing as "a task that would take a human 2 minutes", and certainly no such thing as "an AI that can do every task that would take a human 2 minutes".