4 ms·

I don't argue with the general claim of this article, but the following statements are just wrong: > Deep learning entered the sphere of mainstream awareness l

I don't argue with the general claim of this article, but the following statements are just wrong: > Deep learning entered the sphere of mainstream awareness less than a decade ago, but the science is ancient. This 1986 paper in Nature contains almost all of the core technical innovations that make it work well. (The term deep learning hadn’t been coined yet and neural networks was used instead). > However, the dataset sizes and compute resources that were available in the 1980s weren’t enough to demonstrate the effectiveness of deep learning, and other machine learning techniques like support vector machines took center stage. I think there are two glaring errors here. First, multilayer error backpropagation wasn't developed in 1986. It was developed in the early 1970s, but was quashed by the elimination of funding after the Perceptrons papers and book. It was rediscovered several times but didn't take hold until the PDP group (Rummelhart et al) pushed things through with enough gravitas. Second, and this is the big one: there were plenty of reasonably large datasets, and frankly plenty of sufficiently large compute resources in the late 1980s and certainly the 1990s to do this research. That wasn't the problem. The problem was that neural networks were largely restricted to two layers due to the vanishing gradient problem, so they had to be fat rather than deep, which wasn't good. That's the whole point of the name deep learning -- by overcoming issues which prevented deep networks, researchers were able to do the stuff they can now. That didn't come until later.

jeremysalwen 4y ago
The story your telling was the mainstream story from when "deep learning" first arose, using Boltzmann machines for pretraining, etc. But then the field exploded, more compute power was directed at the problem, and people discovered that yes, in fact basic MLPs with SGD work well with enough compute.
- sdenton4 4y ago
  But the point is that there were super important algorithm advanced that allowed training bigger models. It wasn't all hardware... A big part of it was getting more eyes on the problems (via hype over imagenet results circa 2012) to figure out how to solve them.
rolisz 4y ago
How big was the biggest dataset in the 1980s? And I'm pretty sure that a modern computer with 4 GPUs beats the best clusters from the 80s
- SeanLuke 4y ago
  You don't need to develop GPT-3 to make major progress in deep neural networks: but in the 1990s the field had stalled due to other reasons that had nothing to do with either compute power or data set size.
  - JohnHaugeland 4y ago
    most practicioners claim those were the primary causes of the stall i'm curious what you believe it was instead
sinenomine 4y ago
>Second, and this is the big one: there were plenty of reasonably large datasets, and frankly plenty of sufficiently large compute resources in the 1980s and certainly the 1990s to do research. That wasn't the problem. The problem was that neural networks were largely restricted to two layers due to the vanishing gradient problem, so they had to be fat rather than deep, which wasn't good. Even that was not the decisive problem - given enough compute multilayer NNs could be trained just fine. The real obstruction was a political failure of deep learners (connectionists) in securing academic prestige and associated access to large-scale compute; By now it's all but forgotten, but then this was a prominent academical political debate of the decade: https://en.wikipedia.org/wiki/Neats_and_scruffies https://en.wikipedia.org/wiki/Neats_and_scruffies and even though "neats" eventually prevailed, the connectionists were still sidelined by statistical ML practicioners (whose efforts didn't happen to scale, although more than a few weighty theses and books happened to be published as a result). Perhaps in the near future, when this wave of AI will give us impossible to deny life-changing outcomes (for example: new cures for chronic diseases) we will look back and think about how many years or even decades of progress were postponed, essentially lost for nothing but academic hubris of a camp that won the funding battle then, only to be swept away in a few decades. Until we haven't really understood the Bitter Lesson http://incompleteideas.net/IncIdeas/BitterLesson.html http://incompleteideas.net/IncIdeas/BitterLesson.html every bit of progress that was not realized is our fault.
- snek_case 4y ago
  > Even that was not the decisive problem - given enough compute multilayer NNs could be trained just fine. As someone who has worked in deep learning, this isn't true. Figuring out the right recipe is extremely important. The regime in which you can effectively train a neural network with good performance is just very small. If you structure things just slightly wrong, training just won't converge. IMO we totally could have trained something like a smaller ImageNet with a decently large dataset on a supercomputer in the late 90s. We just didn't know how. In the late 90s and early 2000s, NN researchers didn't know about (or didn't appreciate the importance of) batchnorm, or ReLUs, or which hyper-parameters to use. Convnets had been invented but hadn't yet been popularized. In the late 90s, you might have tried training an 8-layer deep MLP on a dataset of 50K images on your Cray T3E supercomputer. But you would most likely have failed, because you weren't using convnets, your sigmoid activation function lead to a vanishing gradient, and your hyper-parameters were off. You might let this run overnight, but 12 hours later, your loss barely went down, and you were out of compute budget on a supercomputer you were sharing with other researchers. Given the right code, the right recipe, you could have made it work on that 90s supercomputer and with the compute budget you had, but the right recipe just wasn't known at the time. We needed time to figure out and to publish about the really basic ingredients needed to train bigger neural networks successfully. Having access to more compute can help figure out what the right recipe is through trial and error, but ReLUs are a super basic innovation. They are a very simple function, and essentially a theoretical innovation. Someone needed to sit down and figure out that vanishing gradients were actually a problem we needed to think about.
api 4y ago
There's a third too: Moore's law. Few researchers in the world had access to the compute resources to do this stuff in the 1980s and 1990s. Today a high-end desktop can train models large enough to do compelling things in a few hours and a phone or a small laptop can execute a model to render a prediction or do pattern classification in seconds. For the really huge models we now have commodity cloud compute available when in the past you would have had to be among the elect with access to a supercomputer. You couldn't just go whip out a credit card and rent time on a Cray II or a Connection Machine and commodity hardware back then would have taken years to train something like GPT-3 if it even had enough storage.

jeremysalwen 4y ago

The story your telling was the mainstream story from when "deep learning" first arose, using Boltzmann machines for pretraining, etc. But then the field exploded, more compute power was directed at the problem, and people discovered that yes, in fact basic MLPs with SGD work well with enough compute.

sdenton4 4y ago
But the point is that there were super important algorithm advanced that allowed training bigger models. It wasn't all hardware... A big part of it was getting more eyes on the problems (via hype over imagenet results circa 2012) to figure out how to solve them.

rolisz 4y ago

How big was the biggest dataset in the 1980s? And I'm pretty sure that a modern computer with 4 GPUs beats the best clusters from the 80s

SeanLuke 4y ago
You don't need to develop GPT-3 to make major progress in deep neural networks: but in the 1990s the field had stalled due to other reasons that had nothing to do with either compute power or data set size.
- JohnHaugeland 4y ago
  most practicioners claim those were the primary causes of the stall i'm curious what you believe it was instead

sinenomine 4y ago

>Second, and this is the big one: there were plenty of reasonably large datasets, and frankly plenty of sufficiently large compute resources in the 1980s and certainly the 1990s to do research. That wasn't the problem. The problem was that neural networks were largely restricted to two layers due to the vanishing gradient problem, so they had to be fat rather than deep, which wasn't good. Even that was not the decisive problem - given enough compute multilayer NNs could be trained just fine. The real obstruction was a political failure of deep learners (connectionists) in securing academic prestige and associated access to large-scale compute; By now it's all but forgotten, but then this was a prominent academical political debate of the decade: https://en.wikipedia.org/wiki/Neats_and_scruffies https://en.wikipedia.org/wiki/Neats_and_scruffies and even though "neats" eventually prevailed, the connectionists were still sidelined by statistical ML practicioners (whose efforts didn't happen to scale, although more than a few weighty theses and books happened to be published as a result). Perhaps in the near future, when this wave of AI will give us impossible to deny life-changing outcomes (for example: new cures for chronic diseases) we will look back and think about how many years or even decades of progress were postponed, essentially lost for nothing but academic hubris of a camp that won the funding battle then, only to be swept away in a few decades. Until we haven't really understood the Bitter Lesson http://incompleteideas.net/IncIdeas/BitterLesson.html http://incompleteideas.net/IncIdeas/BitterLesson.html every bit of progress that was not realized is our fault.

snek_case 4y ago
> Even that was not the decisive problem - given enough compute multilayer NNs could be trained just fine. As someone who has worked in deep learning, this isn't true. Figuring out the right recipe is extremely important. The regime in which you can effectively train a neural network with good performance is just very small. If you structure things just slightly wrong, training just won't converge. IMO we totally could have trained something like a smaller ImageNet with a decently large dataset on a supercomputer in the late 90s. We just didn't know how. In the late 90s and early 2000s, NN researchers didn't know about (or didn't appreciate the importance of) batchnorm, or ReLUs, or which hyper-parameters to use. Convnets had been invented but hadn't yet been popularized. In the late 90s, you might have tried training an 8-layer deep MLP on a dataset of 50K images on your Cray T3E supercomputer. But you would most likely have failed, because you weren't using convnets, your sigmoid activation function lead to a vanishing gradient, and your hyper-parameters were off. You might let this run overnight, but 12 hours later, your loss barely went down, and you were out of compute budget on a supercomputer you were sharing with other researchers. Given the right code, the right recipe, you could have made it work on that 90s supercomputer and with the compute budget you had, but the right recipe just wasn't known at the time. We needed time to figure out and to publish about the really basic ingredients needed to train bigger neural networks successfully. Having access to more compute can help figure out what the right recipe is through trial and error, but ReLUs are a super basic innovation. They are a very simple function, and essentially a theoretical innovation. Someone needed to sit down and figure out that vanishing gradients were actually a problem we needed to think about.

api 4y ago

There's a third too: Moore's law. Few researchers in the world had access to the compute resources to do this stuff in the 1980s and 1990s. Today a high-end desktop can train models large enough to do compelling things in a few hours and a phone or a small laptop can execute a model to render a prediction or do pattern classification in seconds. For the really huge models we now have commodity cloud compute available when in the past you would have had to be among the elect with access to a supercomputer. You couldn't just go whip out a credit card and rent time on a Cray II or a Connection Machine and commodity hardware back then would have taken years to train something like GPT-3 if it even had enough storage.