7 ms·

Ask HN: Is the rate of progress in AI exponential?

I've frequently seen the claim that progress in machine learning and artificial intelligence, particularly with respect to large language models, has been exponential over the recent past. This arises often in discussions around the existential risk posed by artificial general or super intelligence, but even those more positively disposed to such technology seem to believe in rapid advancements. My problem is that I just don't see it? I see progress, yes, but not exponentially accelerating progress. Focusing on large language models, it took several years to go from the invention of the transformer architecture to a broadly useful autoregressive text completer. Between March of 2022 and March of 2023 OpenAI went from GPT-3.5 to GPT-4; a clear improvement, but not a transformationally massive one. GPT-4 remains the most capable model widely available and that shows no signs of changing -- it's not clear at this point if GPT-5 is even a glimmer in Ilya Sutskever's eye. Further progress towards more capable large language models requires either ever-increasing amounts of data and compute for marginal gains, or (IMO) the development of new approaches that better leverage available data. But the research and development I'm seeing is all about: 1. Better ways to prompt LLMs (chain of thought, tree of thoughts) 2. New LLMs that use less compute than GPT-4 (but are almost uniformly worse, e.g. Bard) 3. Better ways of running smaller (i.e. worse) models on commodity hardware 4. New software tools that employ LLMs (document search, web search, etc) That stuff might be important, but does it count as exponential progress? I think it only seems exponential because ChatGPT seemed to come out of nowhere and now LLMs have sucked all the energy out of the room, but high interest and activity aren't perfect indicators of rapid progress; precisely how many versions of "use embeddings to search through documents" do we need? If progress were truly exponential, I would expect something noticeably better than GPT-4 on the horizon, if not already released. But I'm not even sure such a thing is at all imminent. Heck, OpenAI can't even keep up with the GPT-4 demand right now, let alone a more compute-intensive successor. Maybe something like GPT-5 comes out tomorrow and makes me look like a fool, I just don't see any indication that it will. I'm not necessarily arguing that this means we shouldn't worry about the risks posed by AI. Disinformation is still an issue, and it's possible that jobs will be automated by GPT-4 (though I think such job losses will be marginal for the time being). I just don't see the rapid progress that I'm apparently supposed to be seeing, and I don't know if I'm just missing something obvious?

h_mirin 3y ago
What might be seen as exponential progress could actually be the accumulation of a few tremendous leaps, I believe. For instance, within the architecture realm, we've witnessed leaps such as: Stacked neural networks (deep learning) / CNN / Transformer From the application perspective, there are also areas that have seen significant progress: AlphaGo (An application of reinforcement learning) / AlphaFold (An application for biological sequences) / ChatGPT (An application of Large Language Models / Reinforcement Learning from Human Feedback) You may pick different ideas. However, what I want to say is that these great advancements happened approximately once every two or three years. It may take some time before we witness another major leap. In truth, the majority of research and development does not contribute to exponential progress. It's sad. However, this does not render such effort meaningless. Although it may seem as though there's too much competition within a narrow field, this may not necessarily be detrimental to humanity.
- rfergie 3y ago
  > What might be seen as exponential progress could actually be the accumulation of a few tremendous leaps This is what exponential progress is as long as the leaps continue happening
  - h_mirin 3y ago
    While I agree with your point, I think it ultimately depends on the timescale we're looking at. In other words, if we're looking at a timescale of several years, it's conceivable that we've been experiencing such leaps several times over the previous rate (or more?) for about the past decade. However, if we're looking at a matter of a few weeks, it may seem like we're stagnating. The point is that we cannot predict if these leaps will continue to occur, and when they will happen.
Madmallard 3y ago
This is impossible to know because our advancements in this field are basically at the PHD of the PHD level. Every advancement is an unknown timeline with unknown impact.
3y ago

h_mirin 3y ago

What might be seen as exponential progress could actually be the accumulation of a few tremendous leaps, I believe. For instance, within the architecture realm, we've witnessed leaps such as: Stacked neural networks (deep learning) / CNN / Transformer From the application perspective, there are also areas that have seen significant progress: AlphaGo (An application of reinforcement learning) / AlphaFold (An application for biological sequences) / ChatGPT (An application of Large Language Models / Reinforcement Learning from Human Feedback) You may pick different ideas. However, what I want to say is that these great advancements happened approximately once every two or three years. It may take some time before we witness another major leap. In truth, the majority of research and development does not contribute to exponential progress. It's sad. However, this does not render such effort meaningless. Although it may seem as though there's too much competition within a narrow field, this may not necessarily be detrimental to humanity.

rfergie 3y ago
> What might be seen as exponential progress could actually be the accumulation of a few tremendous leaps This is what exponential progress is as long as the leaps continue happening
- h_mirin 3y ago
  While I agree with your point, I think it ultimately depends on the timescale we're looking at. In other words, if we're looking at a timescale of several years, it's conceivable that we've been experiencing such leaps several times over the previous rate (or more?) for about the past decade. However, if we're looking at a matter of a few weeks, it may seem like we're stagnating. The point is that we cannot predict if these leaps will continue to occur, and when they will happen.

Madmallard 3y ago

This is impossible to know because our advancements in this field are basically at the PHD of the PHD level. Every advancement is an unknown timeline with unknown impact.

3y ago