6 ms·

Quoting Rich Sutton, who wrote the perfect response some years ago: "The biggest lesson that can be read from 70 years of AI research is that general methods t

by cs702 3y ago

Quoting Rich Sutton, who wrote the perfect response some years ago: "The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. The ultimate reason for this is Moore's law, or rather its generalization of continued exponentially falling cost per unit of computation. Most AI research has been conducted as if the computation available to the agent were constant (in which case leveraging human knowledge would be one of the only ways to improve performance) but, over a slightly longer time than a typical research project, massively more computation inevitably becomes available. Seeking an improvement that makes a difference in the shorter term, researchers seek to leverage their human knowledge of the domain, but the only thing that matters in the long run is the leveraging of computation. These two need not run counter to each other, but in practice they tend to. Time spent on one is time not spent on the other. There are psychological commitments to investment in one approach or the other. And the human-knowledge approach tends to complicate methods in ways that make them less suited to taking advantage of general methods leveraging computation."[a] The very smart folks at the Alan Turing Institute are learning firsthand how bitter the lesson can be. --- [a] http://incompleteideas.net/IncIdeas/BitterLesson.html http://incompleteideas.net/IncIdeas/BitterLesson.html

diego_moita 3y ago
Funny plot twist: the pioneer on leveraging computation on neural networks is actually a British: Geoffrey Hinton, living in Canada. Btw, Rich Sutton was born in the U.S. but renounced his American citizenship, becoming Canadian.
- NalNezumi 3y ago
  And a MORE funny story is,(according to an coworker of mine, whose PhD supervisor was a friend to Hinton) that when Hinton was looking for university to conduct his work in, he was rejected by the UK universities which was his first choices. So ended up in Canada. So the plot twist comes with a bit of irony!
  - mafribe 3y ago
    Yes, Hinton was on a temporary position at the University of Sussex (IIRC, the Centre for Cognitive Science) for a while, but was not offered a permanent academic position there when he applied.
- fulafel 3y ago
  Also, the history of the the underlying advances is a lot more international than current popular telling of the history lets on. See eg https://people.idsia.ch/~juergen/scientific-integrity-turing-award-deep-learning.html https://people.idsia.ch/~juergen/scientific-integrity-turing...
adventured 3y ago
Along that line re Moore's Law, the biggest clear advantage the US has in regards to AI, is: Nvidia, AMD, Intel (and obviously particularly Nvidia up to this point, although AMD and Intel are producing some potent GPUs). The reason the US was able to pull off a leap forward via OpenAI or LLaMa, is due to having Nvidia as basically a national treasure, but it's an integrated whole, the US has all the components necessary and the ecosystem that produces all the components (including talent, thought process, money, start-ups, pay scale (to lure talent)). The Europeans have never lacked for the brains side of being able to do it, certainly. Until they fill out the rest of the ecosystem they won't be able to really compete in AI (they'll lag far behind, with lots of blaming and empty promises big government projects). And China is its own worst enemy these days, all we need is for Xi to remain in power indefinitely and he'll throttle their potential as a global competitor. The US is close to locking up another round in the tech wars, riding on the same approach that has served it so well since WW2. Hopefully our do-something legislators are hands off long enough (ie don't snatch defeat from the jaws of victory).
- cs702 3y ago
  > including talent, thought process, money, start-ups, pay scale (to lure talent) ...and a culture of risk-taking and ambition to change the world.
  - adventured 3y ago
    Yeah I tried to cover that with the start-ups implication. The US has that in spades in regards to aggressively funding start-ups by the zillions when a new tech inflection hits (whether software, Web, mobile, cloud and now AI). The VCs always go overboard, which is ideal (billions in destroyed capital is meaningless compared to producing the next Nvidia, Google, Amazon, Microsoft, etc).
    - cs702 3y ago
      Couldn't agree more!
- reedciccio 3y ago
ragnese 3y ago
As someone with no knowledge of the fields of machine learning and artificial intelligence, I wonder how much of recent AI stuff is due to "true" Moore's Law (GPUs and such getting much faster/cheaper), and how much is due to the data version of Moore's Law (web-scale data farming/storage to "teach" LLMs and such).
- cs702 3y ago
  Making use of more data requires more compute (e.g., longer training, more powerful hardware, or both).
  - TremendousJudge 3y ago
    At this point I'm not sure progress is due to Moore's Law as stated originally (cheaper compute) than it is due to companies just spending more on compute. Effect is the same for now, but with a clear limit.
deleted 3y ago
[deleted]
jmole 3y ago
I think the problem is actually worse than the article implies, because there are two things being leveraged here: compute and data. Short-term improvements made by domain-specific AI result in better outputs than more general AIs, ceteris paribus. But these better outputs can then later be fed back into more powerful general purpose AIs, and consuming the data*compute product from the domain-specific models is a very effective way to train domain-specific behavior. Today, we see this in reverse – people are training smaller models based on outputs from GPT-4. However, I expect that we'll start to see more and more training going the opposite direction in the future: Domain-specific generative models will be used to build scenarios for large general-purpose AIs to train against. Here's a concrete example – image diffusion models are really bad at physics, so you can't tell one to draw a person upside down, because it's not well-represented in the dataset, and if you force it to with something like controlnet you typically get a disfigured and horrific image. So obviously diffusion models are not the best long-term solution for image generation. But how do you get this concept of "upside down" into an AI model? Well, maybe you add some kind of neat segmentation technique that involves using several diffusion models and rotating and stitching together their outputs. Great, you made an upside-down generator. Now, you generate 100,000 images of "upside down" people, and the next advance in image generation AI can come along and learn that concept with ease thanks to the larger data set that it has. So it's not just that "more compute wins", it's more like: not only does more compute wins, it wins even more because short-term improvements feed directly into the data pipeline that enables it to win.
luplex 3y ago
At least in the field of computer vision, there seems to be lots of algorithmic progress too. The algorithms improve every 9 months by an amount equivalent to a doubling of compute budget. https://epochai.org/blog/revisiting-algorithmic-progress https://epochai.org/blog/revisiting-algorithmic-progress
- version_five 3y ago
  Also, while it gets lost in the foundation model stuff, a major trend in computer vision is toward smaller, high quality datasets. Arguably CV had its V1 llm moment years ago with models trained on imagenet, which produced amazing general results but weren't good enough for much specific stuff. If you look at what, e.g. Andrew Ng was talking about last year, there was a big emphasis on "small data" and getting good datasets.
- famouswaffles 3y ago
  The bitter lesson isn't really "algorithms bad", "don't try different approaches", "don't innovate" or "only work on models with massive compute". The heart of the bitter lesson is "don't try to codify "insight" into the process". It's basically the age old "you don't know what you don't know". The Transformer is kind of a perfect example. It boasts algorithmic improvements over RNNs and LLMs are by far the best performing take on language modelling ever. And yet the architecture itself has basically no breakthrough from understanding language itself. It's an improvement over standard RNNs but not really because of any new found insight or implementation on language itself. Basically trying to cram human high level instincts/insights into the process of solving a problem doesn't work better than giving a general architecture tons of data and letting it figure that all out by itself.
  - FeepingCreature 3y ago
    I'd argue it's even "you don't know what you do know." We cannot codify what we don't understand, and while we understand and can verbalize some parts of our thinking, others, maybe even the great majority, are hidden from us. We just get a feeling.
  - 3y ago
xtracto 3y ago
The EU+UK need a project like the large hadron collider but for AI: Develop a really really large computational infrastructure that allows researchers to study AI experiments with technology that may be 20 or 30 years far away from being commoditized.
- dr_dshiv 3y ago
  They are doing that for quantum computing. But 20 years is about right. It seems like Netherlands completely missed the boat on LLMs, too… but I don’t blame them. I just hope they pivot quickly.
dr_dshiv 3y ago
Let’s be honest, though—very, very few people expected large language models to be so ungodly effective.
- clueless 3y ago
  is this really true?
- readthenotes1 3y ago
  Perhaps many many many more people would have made the bet on LLM versus NFTs or virtue signalling with “Data as an instrument of coloniality: A panel discussion on digital and data colonialism”, right?

by cs702 3y ago

diego_moita 3y ago

Funny plot twist: the pioneer on leveraging computation on neural networks is actually a British: Geoffrey Hinton, living in Canada. Btw, Rich Sutton was born in the U.S. but renounced his American citizenship, becoming Canadian.

NalNezumi 3y ago
And a MORE funny story is,(according to an coworker of mine, whose PhD supervisor was a friend to Hinton) that when Hinton was looking for university to conduct his work in, he was rejected by the UK universities which was his first choices. So ended up in Canada. So the plot twist comes with a bit of irony!
- mafribe 3y ago
  Yes, Hinton was on a temporary position at the University of Sussex (IIRC, the Centre for Cognitive Science) for a while, but was not offered a permanent academic position there when he applied.
fulafel 3y ago
Also, the history of the the underlying advances is a lot more international than current popular telling of the history lets on. See eg https://people.idsia.ch/~juergen/scientific-integrity-turing-award-deep-learning.html https://people.idsia.ch/~juergen/scientific-integrity-turing...

adventured 3y ago

Along that line re Moore's Law, the biggest clear advantage the US has in regards to AI, is: Nvidia, AMD, Intel (and obviously particularly Nvidia up to this point, although AMD and Intel are producing some potent GPUs). The reason the US was able to pull off a leap forward via OpenAI or LLaMa, is due to having Nvidia as basically a national treasure, but it's an integrated whole, the US has all the components necessary and the ecosystem that produces all the components (including talent, thought process, money, start-ups, pay scale (to lure talent)). The Europeans have never lacked for the brains side of being able to do it, certainly. Until they fill out the rest of the ecosystem they won't be able to really compete in AI (they'll lag far behind, with lots of blaming and empty promises big government projects). And China is its own worst enemy these days, all we need is for Xi to remain in power indefinitely and he'll throttle their potential as a global competitor. The US is close to locking up another round in the tech wars, riding on the same approach that has served it so well since WW2. Hopefully our do-something legislators are hands off long enough (ie don't snatch defeat from the jaws of victory).

cs702 3y ago
> including talent, thought process, money, start-ups, pay scale (to lure talent) ...and a culture of risk-taking and ambition to change the world.
- adventured 3y ago
  Yeah I tried to cover that with the start-ups implication. The US has that in spades in regards to aggressively funding start-ups by the zillions when a new tech inflection hits (whether software, Web, mobile, cloud and now AI). The VCs always go overboard, which is ideal (billions in destroyed capital is meaningless compared to producing the next Nvidia, Google, Amazon, Microsoft, etc).
  - cs702 3y ago
    Couldn't agree more!
reedciccio 3y ago

ragnese 3y ago

As someone with no knowledge of the fields of machine learning and artificial intelligence, I wonder how much of recent AI stuff is due to "true" Moore's Law (GPUs and such getting much faster/cheaper), and how much is due to the data version of Moore's Law (web-scale data farming/storage to "teach" LLMs and such).

cs702 3y ago
Making use of more data requires more compute (e.g., longer training, more powerful hardware, or both).
- TremendousJudge 3y ago
  At this point I'm not sure progress is due to Moore's Law as stated originally (cheaper compute) than it is due to companies just spending more on compute. Effect is the same for now, but with a clear limit.

deleted 3y ago

[deleted]

jmole 3y ago

I think the problem is actually worse than the article implies, because there are two things being leveraged here: compute and data. Short-term improvements made by domain-specific AI result in better outputs than more general AIs, ceteris paribus. But these better outputs can then later be fed back into more powerful general purpose AIs, and consuming the data*compute product from the domain-specific models is a very effective way to train domain-specific behavior. Today, we see this in reverse – people are training smaller models based on outputs from GPT-4. However, I expect that we'll start to see more and more training going the opposite direction in the future: Domain-specific generative models will be used to build scenarios for large general-purpose AIs to train against. Here's a concrete example – image diffusion models are really bad at physics, so you can't tell one to draw a person upside down, because it's not well-represented in the dataset, and if you force it to with something like controlnet you typically get a disfigured and horrific image. So obviously diffusion models are not the best long-term solution for image generation. But how do you get this concept of "upside down" into an AI model? Well, maybe you add some kind of neat segmentation technique that involves using several diffusion models and rotating and stitching together their outputs. Great, you made an upside-down generator. Now, you generate 100,000 images of "upside down" people, and the next advance in image generation AI can come along and learn that concept with ease thanks to the larger data set that it has. So it's not just that "more compute wins", it's more like: not only does more compute wins, it wins even more because short-term improvements feed directly into the data pipeline that enables it to win.

luplex 3y ago

At least in the field of computer vision, there seems to be lots of algorithmic progress too. The algorithms improve every 9 months by an amount equivalent to a doubling of compute budget. https://epochai.org/blog/revisiting-algorithmic-progress https://epochai.org/blog/revisiting-algorithmic-progress

version_five 3y ago
Also, while it gets lost in the foundation model stuff, a major trend in computer vision is toward smaller, high quality datasets. Arguably CV had its V1 llm moment years ago with models trained on imagenet, which produced amazing general results but weren't good enough for much specific stuff. If you look at what, e.g. Andrew Ng was talking about last year, there was a big emphasis on "small data" and getting good datasets.
famouswaffles 3y ago
The bitter lesson isn't really "algorithms bad", "don't try different approaches", "don't innovate" or "only work on models with massive compute". The heart of the bitter lesson is "don't try to codify "insight" into the process". It's basically the age old "you don't know what you don't know". The Transformer is kind of a perfect example. It boasts algorithmic improvements over RNNs and LLMs are by far the best performing take on language modelling ever. And yet the architecture itself has basically no breakthrough from understanding language itself. It's an improvement over standard RNNs but not really because of any new found insight or implementation on language itself. Basically trying to cram human high level instincts/insights into the process of solving a problem doesn't work better than giving a general architecture tons of data and letting it figure that all out by itself.
- FeepingCreature 3y ago
  I'd argue it's even "you don't know what you do know." We cannot codify what we don't understand, and while we understand and can verbalize some parts of our thinking, others, maybe even the great majority, are hidden from us. We just get a feeling.
- 3y ago

xtracto 3y ago

The EU+UK need a project like the large hadron collider but for AI: Develop a really really large computational infrastructure that allows researchers to study AI experiments with technology that may be 20 or 30 years far away from being commoditized.

dr_dshiv 3y ago
They are doing that for quantum computing. But 20 years is about right. It seems like Netherlands completely missed the boat on LLMs, too… but I don’t blame them. I just hope they pivot quickly.

dr_dshiv 3y ago

Let’s be honest, though—very, very few people expected large language models to be so ungodly effective.

clueless 3y ago
is this really true?
readthenotes1 3y ago
Perhaps many many many more people would have made the bet on LLM versus NFTs or virtue signalling with “Data as an instrument of coloniality: A panel discussion on digital and data colonialism”, right?