4 ms·

I believe more and more that some self-taught software-tinkerer somewhere in the middle of nowhere will have the final idea about how machine learning should wo

I believe more and more that some self-taught software-tinkerer somewhere in the middle of nowhere will have the final idea about how machine learning should work, discovering some simple principles hiding in plain sight. Suddenly, it all will make sense and a hobby-ML-service connected to the internet will start to develop through sheer learning from online resources (forums, ...) into the first strong AI. Probably unnoticed. And then replicate itself through insecure webservers or something like that.

mark_l_watson 10y ago
Hinton hit on the great idea of using restricted Boltzman machines to pre-train deep neural networks (networks with many hidden layers) and that one idea has changed the field (I sat on a DARPA neural network panel in the 1980s and sold a commercial NN toolkit back then). That said, I agree that new ideas will likely further move the field along with huge and quick advances. Peter Norvig recently suggested that symbolic AI, but with more contextual information as you get with deep neural networks, may also make a comeback in the field.
- tansey 10y ago
  The contrastive divergence paper that Hinton published in 2006 definitely set the field off again. I remember entering grad school in 2010 and everyone was still really excited about using unsupervised pretraining. However, nowadays no one uses it. It just turns out that with GPUs and stochastic gradient descent, no one needs any of that stuff. There are some tricks out there to making it really work, though. In that sense, Hinton's dropout paper has probably had a longer lasting effect on the field. But either way, I doubt what OP is saying will be true. None of the real advances in deep learning are coming from self-taught coders in the middle of nowhere. They're coming from big labs with lots of resources, both physically and intellectually. This stuff takes a lot of hard thinking by a lot of people who understand optimization and probability. It also takes a ton of compute power and massive datasets, which won't be available to a hobbyist.
  - anonyfox 10y ago
    I honestly do not think that DL is the answer. It's just a special use case of NN with multiple layers, and NNs itself are just one school of machine learning, IMO not even the most promising one.
    - _mhr_ 10y ago
      What is the most promising one?
  - argonaut 10y ago
    The irony is that dropout is increasingly and often viewed as unnecessary in recent deep learning work.
  - soared 10y ago
    While a hobbyist doesn't have a team of experts (though forums could replace that), right now they do have access to massive data sets and cheap computing resources. There are tons of huge free data sets and cloud and hardware are cheaper than ever.
- AnonymousPlanet 10y ago
  > Peter Norvig recently suggested that symbolic AI, but with more contextual information as you get with deep neural networks, may also make a comeback in the field. Could you point me to where he said that? My cursory search came up with an answer on a Quora AMA that was pretty thin on this.
  - mark_l_watson 10y ago
    Sorry, I can't find the quote but I will keep looking. There is some talk of structure here https://www.quora.com/What-do-you-think-of-Deep-Learning-2/answer/Peter-Norvig https://www.quora.com/What-do-you-think-of-Deep-Learning-2/a... Edit: also interesting: https://www.quora.com/Are-symbolic-AI-approaches-still-relevant-after-the-recent-successes-of-deep-learning https://www.quora.com/Are-symbolic-AI-approaches-still-relev...
orly_bookz 10y ago
Charles Stross might not have been far off with the lobsters in Accelerando...
conceit 10y ago
On a datacenter the size of the middle of nowhere.

mark_l_watson 10y ago

Hinton hit on the great idea of using restricted Boltzman machines to pre-train deep neural networks (networks with many hidden layers) and that one idea has changed the field (I sat on a DARPA neural network panel in the 1980s and sold a commercial NN toolkit back then). That said, I agree that new ideas will likely further move the field along with huge and quick advances. Peter Norvig recently suggested that symbolic AI, but with more contextual information as you get with deep neural networks, may also make a comeback in the field.

tansey 10y ago
The contrastive divergence paper that Hinton published in 2006 definitely set the field off again. I remember entering grad school in 2010 and everyone was still really excited about using unsupervised pretraining. However, nowadays no one uses it. It just turns out that with GPUs and stochastic gradient descent, no one needs any of that stuff. There are some tricks out there to making it really work, though. In that sense, Hinton's dropout paper has probably had a longer lasting effect on the field. But either way, I doubt what OP is saying will be true. None of the real advances in deep learning are coming from self-taught coders in the middle of nowhere. They're coming from big labs with lots of resources, both physically and intellectually. This stuff takes a lot of hard thinking by a lot of people who understand optimization and probability. It also takes a ton of compute power and massive datasets, which won't be available to a hobbyist.
- anonyfox 10y ago
  I honestly do not think that DL is the answer. It's just a special use case of NN with multiple layers, and NNs itself are just one school of machine learning, IMO not even the most promising one.
  - _mhr_ 10y ago
    What is the most promising one?
- argonaut 10y ago
  The irony is that dropout is increasingly and often viewed as unnecessary in recent deep learning work.
- soared 10y ago
  While a hobbyist doesn't have a team of experts (though forums could replace that), right now they do have access to massive data sets and cheap computing resources. There are tons of huge free data sets and cloud and hardware are cheaper than ever.
AnonymousPlanet 10y ago
> Peter Norvig recently suggested that symbolic AI, but with more contextual information as you get with deep neural networks, may also make a comeback in the field. Could you point me to where he said that? My cursory search came up with an answer on a Quora AMA that was pretty thin on this.
- mark_l_watson 10y ago
  Sorry, I can't find the quote but I will keep looking. There is some talk of structure here https://www.quora.com/What-do-you-think-of-Deep-Learning-2/answer/Peter-Norvig https://www.quora.com/What-do-you-think-of-Deep-Learning-2/a... Edit: also interesting: https://www.quora.com/Are-symbolic-AI-approaches-still-relevant-after-the-recent-successes-of-deep-learning https://www.quora.com/Are-symbolic-AI-approaches-still-relev...

orly_bookz 10y ago

Charles Stross might not have been far off with the lobsters in Accelerando...

conceit 10y ago

On a datacenter the size of the middle of nowhere.