2 ms·
there are some surprisingly weak arguments in the text. It's correct to not treat computational resources as constant, but ot treat them as unimportant or negli
by Barrin92 7y ago
there are some surprisingly weak arguments in the text. It's correct to not treat computational resources as constant, but ot treat them as unimportant or negligible is awful as well.
Already computational resources are becoming prohibitive with only a few institutions producing state of the art models at high financial cost. If the goal is AGI this might get exponentially worse. Intelligence needs to take resource consumption into account. The models we produce aren't even close to high level reasoning and we're already consuming significantly more energy than humans or animals, something is wrong.
The scale argument isn't great either because deep learning is running into the inverse issue of classical AI. Now instead of having to program all logic explicitly we have to formulate every individual problem as training data. This doesn't scale either. If an AI gets attacked by a wild animal the solution can't be to first produce 10k pictures of mauled victims, intelligence includes to reason about things in the abscence of data. We can't have autonomous cars constantly running into things until we provide huge amounts of data for every problem, this does not scale either.
- roenxi 7y ago> Already computational resources are becoming prohibitive with only a few institutions producing state of the art models at high financial cost. That is something of an illusion. Obviously there will be some sort of uneven distribution of computing power; some institutions will have more, some less. The institutions with more power will create models at the limit of what they can do, because that is the best use of their power. So if the thesis of more power = more results holds then truly cutting results will always be by people with resources that are practically unattainable by everyone else. Google's AlphaGo wasn't a particularly clever model, for example. It just had a lot of horsepower behind it to train it and the various ranging shot attempts Deepmind would have gone through. Someone else would have figured it out albeit more slowly in a few years as computing power became available. Computational power is still getting exponentially more affordable [0]. Costs aren't really rising, so much as the people who have spent more money get a few years ahead of everyone else and can preview what is about to become cheap. [0] https://aiimpacts.org/recent-trend-in-the-cost-of-computing/ https://aiimpacts.org/recent-trend-in-the-cost-of-computing/
- lonelappde 7y agoThat article says that doubling flops/$ used to take 1 year and now takes 3. It's open question whether that time gap will confirm to grow. The recent phase shift was due to approaching an asymptote in single core performance, and switching to optimizing multicore overhead. The low hanging fruit there will be consumed as well.
- chillacy 7y agoIs the weak argument that computation is not guaranteed to increase in the future, or that it won't increase fast enough for AI researchers? Most of the article is describing past scenarios, only the last 3 paragraphs make the argument that the past is a good representation of the present