4 ms·
The "Bitter Lesson" post from Rich Sutton from earlier this year [1] seems a very good complement to this article: he explain how all of the big improvements in
by m3at 7y ago
The "Bitter Lesson" post from Rich Sutton from earlier this year [1] seems a very good complement to this article: he explain how all of the big improvements in the field came from new methods that leveraged the much larger compute available from Moore's law, instead of progressive buildup over existing methods.
A great quote from McCarthy also regularly referenced by Sutton is "Intelligence is the computational part of the ability to achieve goals", which (IMO) help picture the tight link between compute growth and AI.
It's only a few minutes read, I highly recommend it:
[1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
- _bxg1 7y agoIt could be that the slowing of Moore's Law results in a more diverse array of specialized computing hardware. After all, the resurgence of ML didn't happen because x86 got fast enough, but because video games - of all things - funded the maturation of a whole new category of massively-parallel chips.
- codesushi42 7y agothe resurgence of ML ... because video games - of all things - funded the maturation of a whole new category of massively-parallel chips Fake news. It has far more to do with the rise of distributed computing than the existence of GPUs.
- jchw 7y agoI don’t really see this; All of the currently popular machine learning frameworks either support GPU or are oriented around GPU based execution. For most developers working on AI it is still the defacto standard.
- sgt101 7y agoI think.. both. Distributed computing (Hadoop) was why a lot of data got collected and made available(ish). GPU's are the engines that made CNN's (in particular) tractable, and opened up a bunch of applications for many companies, and opened up a reasonable route to results for a generation of researchers.
- unkulunkulu 7y agoCan anybody elaborate on why this is downvoted? This would be my guess as well, simd parallelism of GPUs solves only part of the challenges, you still need a general purpose data crunching machine to prepare and handle learning data.
- fspeech 7y agoFor one GPU speedup over CPU isn’t that dramatic for small to medium sized problems, e.g. MNIST or CIFAR that one would try algorithm ideas on. So I think it’s a stretch to see GPU as essential to the new algorithms. On the other hand for large problems like the original Alpha Go you need to figure out the distributed computing to really scale. This isn’t to say that GPUs aren’t nice. They do save time or for the same amount of time let you produce more polished results, which means in a competitive environment everyone would use them.
- codesushi42 7y agoExactly. GPUs are necessary now, but did not originally herald the deep learning revolution.
- codesushi42 7y agoExcept that's "currently". NNs can and were trained on CPUs originally.
- adwn 7y agoDon't call a statement you disagree with "fake news". It a) doesn't even make sense in this case, and b) distracts from the discussion.
- solidasparagus 7y agoI don't think this is very true. You can trace the deep learning revolution back to VGG and a fundamental driver in the success of the first multi-level networks was the ability to train in semi-reasonable amounts of time using GPUs. Even today distributed training is relatively uncommon while pretty much everyone uses NVIDIA GPUs.
- codesushi42 7y agoVGG was a long time after the first deep learning revolution.
- solidasparagus 7y agoYou could say it started with ImageNet 2012 where AlexNet showed that deep networks were a newly promising area of study - however the actual performance of AlexNet was very very far from human performance. I tend to say the revolution started at ImageNet 2014 with VGG/GoogLeNet, the first human-caliber performances. Or you could say it was ImageNet 2015, when the first ResNet had better-than-human performance.
- Veedrac 7y agoMoore's law never slowed. At worst, one can rightly say Intel had issues with 10nm. https://docs.google.com/spreadsheets/d/1NNOqbJfcISFyMd0EsSrhppW7PT6GCfnrVGhxhLA5PVw/edit#gid=1105655921 https://docs.google.com/spreadsheets/d/1NNOqbJfcISFyMd0EsSrh...
- ChuckMcM 7y agoThank you for that, it was excellent.
- perlpimp 7y agowhat a great article. would give one a way to have introspection of how and why these methods work and going forward learning machine learning/AI.
- YeGoblynQueenne 7y agoGreat read. To follow up also read a comment on that piece titled "A better lesson" by Rodney Brooks: https://rodneybrooks.com/a-better-lesson/ https://rodneybrooks.com/a-better-lesson/
- djokkataja 7y ago> One of the most celebrated successes of Deep Learning is image labeling, using CNNs, Convolutional Neural Networks, but the very essence of CNNs is that the front end of the network is designed by humans to manage translational invariance, the idea that objects can appear anywhere in the frame. To have a Deep Learning network also have to learn that seems pedantic to the extreme, and will drive up the computational costs of the learning by many orders of magnitude. Thanks for the link, this is a pretty strong rebuttal. That said... > Massive data sets are not at all what humans need to learn things so something is missing. Today’s data sets can have billions of examples, where a human may only require a handful to learn the same thing. I've come across this idea before, the notion that we should be able to create an AI that can learn something from just a handful of examples because humans can do that. This assumes some rough level of equality in lack of related knowledge / capability between a tabula rasa AI system and a human adult. But a human adult has been awake and continuously learning & testing conscious and unconscious predictions about everything they interact with for at least 18 years (1 year = 31536000 seconds * 2/3 (accounting for sleep) = 21024000 seconds awake per year, * 18 years = 378 432 000 seconds). In other words, a human adult is already a highly trained system. And even very young human children have been awake and learning for quite a long time compared to anything we might imagine to be a tabula rasa system. So I don't think that the human capacity for learning supports the idea that a tabula rasa AI system should be able to learn any particular thing with just a handful of examples.
- Terr_ 7y agoI often imagine young babies are staring around not in "childlike wonder" as much as an LSD drug trip, with their little minds laboring to discover any kind of correlation in an assault of data.
- YeGoblynQueenne 7y ago
- KKKKkkkk1 7y agoSutton argues that algorithm development needs to be shaped by the assumption that compute power will continue growing exponentially into the future. At this point, it is commonly believed that this is not the case. One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning.
- Veedrac 7y agoI'm fairly sure the consensus from the hardware side, especially considering how companies like NVIDIA, Intel and Arm are reacting, is that we still have a lot of exponential growth to go, especially for AI. A factor 1000 increase in model sizes seems within reason within 10ish years; I go over some arguments in the link below. https://www.reddit.com/r/MachineLearning/comments/ds1xvc/d_deep_learning_has_a_size_problem_we_need_to/f6nnq8d/ https://www.reddit.com/r/MachineLearning/comments/ds1xvc/d_d...
- buboard 7y agothats great arguments except vision. Knowledge about the physiology of the visual system DID go into designing convolutional networks. That probably doesn't extend to other domains though, and things prove once again that our intuitions may be misleading.