3 ms·
It's easy to say that, in the same way that people now wouldn't be surprised if we could factor large numbers in linear time if we had a functional quantum comp
by nl 11y ago
It's easy to say that, in the same way that people now wouldn't be surprised if we could factor large numbers in linear time if we had a functional quantum computer!!
10 years ago no one believed it was possible to train deep nets[1].
It wasn't until the current "revolution" that people learned how important parameter initialization was. Sure, it's not a new algorithm, but it made the problem tractable.
So far as algorithmic innovations go, there's always ReLU (2011) and leaky ReLU (2014). The one-weird-trick paper was pretty important too.
[1] Training deep multi-layered neural networks is known to be hard. The standard learning strategy—
consisting of randomly initializing the weights of the network and applying gradient descent using
backpropagation—is known empirically to find poor solutions for networks with 3 or more hidden
layers. As this is a negative result, it has not been much reported in the machine learning literature.
For that reason, artificial neural networks have been limited to one or two hidden layers
http://deeplearning.cs.cmu.edu/pdfs/1111/jmlr10_larochelle.pdf http://deeplearning.cs.cmu.edu/pdfs/1111/jmlr10_larochelle.p...
- eru 11y agoDropout (and maxout) might also count.
- fma 11y agoIt's only difficult because no one threw money at it. It's like saying going to Mars is difficult. It is - but most of the technology is there already, just need money to improve what was used to go to the moon. If you asked people 10 years ago before the moon landing if it was possible, I too would agree it's impossible. But after that breakthrough it opened up the realm is possibilities. I see AlphaGo more of an incremental improvement than a breakthrough.
- oldmanjay 11y agoIt's a basic human bias to believe that anything that you don't have to do (or know how to do) "just needs money" to get done.
- nl 11y agoSo are you arguing that superhuman-level performance in just a matter of engineering effort? Or am I missing something? I'm generally considered to be way over optimistic in my assessment of AI progress. But wow.. that's pretty optimistic!
- argonaut 11y agoI interpreted him as saying superhuman-performance at Go was just a matter of engineering effort, which I wholeheartedly agree with.