5 ms·

Neural networks is advanced curve fitting -- That's why. It isn't really all that magical or mysterious.

nl 11y ago
It really kinda is. In the Ilya Sutskever Talking Machines podcast[1] he describes exactly how it is "magic". He talks about how there is no theoretical basis to think that a deep neural network should converge, and prior to around 2006 the accepted wisdom was that networks deep enough to outperform other methods of machine learning were useless because they couldn't be trained. And then they discovered how to initialize the random starting values in such a way that they do converge - for reasons no one really understands. I've heard a lot of people claim there is no magic, but I tend to think Sutskever[2] knows a little bit more than most people about deep neural networks. [1] http://www.thetalkingmachines.com/blog/2015/1/15/machine-learning-and-magical-thinking http://www.thetalkingmachines.com/blog/2015/1/15/machine-lea... [2] http://www.cs.toronto.edu/~ilya/ http://www.cs.toronto.edu/~ilya/
- dasboth 11y ago
  I was just listening to that podcast yesterday and loved the way he presented this information. I know next to nothing about NNs but I could pretty much follow everything that he said (from an intuitive sense, anyway, if not a technical one). This "magic" of converging networks reminds me of how ensemble methods, such as random forests, are effective but people aren't sure why. There's certainly (AFAIK) no theoretical grounds to say that a bunch of random decision trees should yield universally good results.
  - mziel 11y ago
    There are. It's all about variance reduction. See the Breiman's paper on Bagging. There's nothing special about random forest though (apart from the fact that a decision tree is a good learner, because of the nonlinearities for example), you can you ensemble learning with any "basic" learner.
    - dasboth 11y ago
      I read Breiman's random forest paper, but I'll give his others a read too. Does the variance reduction effect apply to any ensemble method then?
- emcq 11y ago
  Reasons like these are why I get frustrated when people claim the only reason why NNs are popular now is because of computational advances. This isn't true - there is lots of little learnings that have been developed in recent time to make training these networks reliable in production. Large companies (and universities) have had access to high performance computers and clusters to train interesting networks (say char-rnn) for some time.
shas3 11y ago
That isn't saying much. Every supervised learning algorithm can be cast as a problem of advanced curve fitting, because at the end of the day, your 'curve' is the mapping that takes you from your 'x-axis' of data to the 'y-axis' of labels.

nl 11y ago

It really kinda is. In the Ilya Sutskever Talking Machines podcast[1] he describes exactly how it is "magic". He talks about how there is no theoretical basis to think that a deep neural network should converge, and prior to around 2006 the accepted wisdom was that networks deep enough to outperform other methods of machine learning were useless because they couldn't be trained. And then they discovered how to initialize the random starting values in such a way that they do converge - for reasons no one really understands. I've heard a lot of people claim there is no magic, but I tend to think Sutskever[2] knows a little bit more than most people about deep neural networks. [1] http://www.thetalkingmachines.com/blog/2015/1/15/machine-learning-and-magical-thinking http://www.thetalkingmachines.com/blog/2015/1/15/machine-lea... [2] http://www.cs.toronto.edu/~ilya/ http://www.cs.toronto.edu/~ilya/

dasboth 11y ago
I was just listening to that podcast yesterday and loved the way he presented this information. I know next to nothing about NNs but I could pretty much follow everything that he said (from an intuitive sense, anyway, if not a technical one). This "magic" of converging networks reminds me of how ensemble methods, such as random forests, are effective but people aren't sure why. There's certainly (AFAIK) no theoretical grounds to say that a bunch of random decision trees should yield universally good results.
- mziel 11y ago
  There are. It's all about variance reduction. See the Breiman's paper on Bagging. There's nothing special about random forest though (apart from the fact that a decision tree is a good learner, because of the nonlinearities for example), you can you ensemble learning with any "basic" learner.
  - dasboth 11y ago
    I read Breiman's random forest paper, but I'll give his others a read too. Does the variance reduction effect apply to any ensemble method then?
emcq 11y ago
Reasons like these are why I get frustrated when people claim the only reason why NNs are popular now is because of computational advances. This isn't true - there is lots of little learnings that have been developed in recent time to make training these networks reliable in production. Large companies (and universities) have had access to high performance computers and clusters to train interesting networks (say char-rnn) for some time.

shas3 11y ago

That isn't saying much. Every supervised learning algorithm can be cast as a problem of advanced curve fitting, because at the end of the day, your 'curve' is the mapping that takes you from your 'x-axis' of data to the 'y-axis' of labels.