3 ms·

The opposing argument, by Rich Sutton, Distinguished Research Scientist at DeepMind: http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incom

by cs702 7y ago

The opposing argument, by Rich Sutton, Distinguished Research Scientist at DeepMind: http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html

justicezyx 7y ago
Seems talking about different issues
cjauvin 7y ago
I would say this is also an opposing argument: https://twitter.com/ylecun/status/1066568396177842176 https://twitter.com/ylecun/status/1066568396177842176 i.e. gradient-based learning is the final word on the matter.
- YeGoblynQueenne 7y ago
  Yes, gradient optimisation always gets the final word. But what does that word, well, say? _ / \ ,----------------------------. | Hey! I found your minimum! | '------------ ------------' / .----. \ / / \ / / \ \/ / \ / / \ @ / \ / / `'--' \ / / \ / / \ / / \ / / \ .-. / / \_/ `\ / \.__.' \ / \ / `;._.-'
Barrin92 7y ago
there are some surprisingly weak arguments in the text. It's correct to not treat computational resources as constant, but ot treat them as unimportant or negligible is awful as well. Already computational resources are becoming prohibitive with only a few institutions producing state of the art models at high financial cost. If the goal is AGI this might get exponentially worse. Intelligence needs to take resource consumption into account. The models we produce aren't even close to high level reasoning and we're already consuming significantly more energy than humans or animals, something is wrong. The scale argument isn't great either because deep learning is running into the inverse issue of classical AI. Now instead of having to program all logic explicitly we have to formulate every individual problem as training data. This doesn't scale either. If an AI gets attacked by a wild animal the solution can't be to first produce 10k pictures of mauled victims, intelligence includes to reason about things in the abscence of data. We can't have autonomous cars constantly running into things until we provide huge amounts of data for every problem, this does not scale either.
- roenxi 7y ago
  > Already computational resources are becoming prohibitive with only a few institutions producing state of the art models at high financial cost. That is something of an illusion. Obviously there will be some sort of uneven distribution of computing power; some institutions will have more, some less. The institutions with more power will create models at the limit of what they can do, because that is the best use of their power. So if the thesis of more power = more results holds then truly cutting results will always be by people with resources that are practically unattainable by everyone else. Google's AlphaGo wasn't a particularly clever model, for example. It just had a lot of horsepower behind it to train it and the various ranging shot attempts Deepmind would have gone through. Someone else would have figured it out albeit more slowly in a few years as computing power became available. Computational power is still getting exponentially more affordable [0]. Costs aren't really rising, so much as the people who have spent more money get a few years ahead of everyone else and can preview what is about to become cheap. [0] https://aiimpacts.org/recent-trend-in-the-cost-of-computing/ https://aiimpacts.org/recent-trend-in-the-cost-of-computing/
  - 7y ago
deleted 7y ago
[deleted]
6gvONxR4sf7o 7y ago
This argument completely ignores anything statistical, which is another limiting factor. It reminds me of the difference in bandit research and full RL research. RL researchers are fine throwing a thousand years of experience at their algorithm, because it can be simulated. Meanwhile people using bandits in the real world care about statistical efficiency (learning lots with little data), and it's reflected in the research. Most decisions aren't made with a huge abundance of data (most of us aren't google or facebook).

by cs702 7y ago

justicezyx 7y ago

Seems talking about different issues

cjauvin 7y ago

I would say this is also an opposing argument: https://twitter.com/ylecun/status/1066568396177842176 https://twitter.com/ylecun/status/1066568396177842176 i.e. gradient-based learning is the final word on the matter.

YeGoblynQueenne 7y ago
Yes, gradient optimisation always gets the final word. But what does that word, well, say? _ / \ ,----------------------------. | Hey! I found your minimum! | '------------ ------------' / .----. \ / / \ / / \ \/ / \ / / \ @ / \ / / `'--' \ / / \ / / \ / / \ / / \ .-. / / \_/ `\ / \.__.' \ / \ / `;._.-'

Barrin92 7y ago

there are some surprisingly weak arguments in the text. It's correct to not treat computational resources as constant, but ot treat them as unimportant or negligible is awful as well. Already computational resources are becoming prohibitive with only a few institutions producing state of the art models at high financial cost. If the goal is AGI this might get exponentially worse. Intelligence needs to take resource consumption into account. The models we produce aren't even close to high level reasoning and we're already consuming significantly more energy than humans or animals, something is wrong. The scale argument isn't great either because deep learning is running into the inverse issue of classical AI. Now instead of having to program all logic explicitly we have to formulate every individual problem as training data. This doesn't scale either. If an AI gets attacked by a wild animal the solution can't be to first produce 10k pictures of mauled victims, intelligence includes to reason about things in the abscence of data. We can't have autonomous cars constantly running into things until we provide huge amounts of data for every problem, this does not scale either.

roenxi 7y ago
> Already computational resources are becoming prohibitive with only a few institutions producing state of the art models at high financial cost. That is something of an illusion. Obviously there will be some sort of uneven distribution of computing power; some institutions will have more, some less. The institutions with more power will create models at the limit of what they can do, because that is the best use of their power. So if the thesis of more power = more results holds then truly cutting results will always be by people with resources that are practically unattainable by everyone else. Google's AlphaGo wasn't a particularly clever model, for example. It just had a lot of horsepower behind it to train it and the various ranging shot attempts Deepmind would have gone through. Someone else would have figured it out albeit more slowly in a few years as computing power became available. Computational power is still getting exponentially more affordable [0]. Costs aren't really rising, so much as the people who have spent more money get a few years ahead of everyone else and can preview what is about to become cheap. [0] https://aiimpacts.org/recent-trend-in-the-cost-of-computing/ https://aiimpacts.org/recent-trend-in-the-cost-of-computing/
- 7y ago

deleted 7y ago

[deleted]

6gvONxR4sf7o 7y ago

This argument completely ignores anything statistical, which is another limiting factor. It reminds me of the difference in bandit research and full RL research. RL researchers are fine throwing a thousand years of experience at their algorithm, because it can be simulated. Meanwhile people using bandits in the real world care about statistical efficiency (learning lots with little data), and it's reflected in the research. Most decisions aren't made with a huge abundance of data (most of us aren't google or facebook).