4 ms·
This feels overly sensationalist. It's no surprise that in domains where low-parameter models work at all, they often work better than NNs. In econometric fore
by goodside 7y ago
This feels overly sensationalist.
It's no surprise that in domains where low-parameter models work at all, they often work better than NNs. In econometric forecasting, where the traditional methods encapsulate more prior knowledge than could ever be inferred from the data, neural networks aren't anywhere near competitive. Even in domains where NNs are state-of-the-art, they might not be worth the training costs over GLMs or gradient-boosted forests or whatever.
But there are many other domains, like raw waveform modeling, large natural images (not MNIST), or NLP, where anything that isn't neural fails miserably. These are the domains where people are most excited about neural networks — problems that otherwise have no solution at all. Neural networks won't help you get a better interpretation of your n=25 daily-resolution cohort study, or predict user retention from your 300-column, 20K-row Excel spreadsheet.
In their paper, the authors dismiss most of NN's greatest recent successes as being "specialized" applications that only work because they incorporate tricks like recurrence and convolution, and not because they are neural. This is a much clearer, bolder claim, but they hardly support it as far as I can tell. The things that they call "specialized" (RNNs and CNNs!) are what most would recognize as the bread-and-butter of neural network design.
- ummonk 7y agoRight, the whole point is that neural networks let you apply tricks like convolution or recurrence to capture a lot of the structure in the model domain and exponentially reduce the search space for your model when dealing with large inputs like detailed images or waveforms.
- enriquto 7y agoThis has to do with the symmetry of the input to translations, that allows a convolutional neural network. But neural networks need not be convolutional.
- posterboy 7y agoCould one say, in a sense, that recurrence is essentially differential equations and convolution essentially more complicated operations than those of arithmetic polynomials? This might sound like nonsense. On the one hand, most trivial convolutions use trivial operators; "polynomials" might include higher operations anyhow, or approximate some of the more important ones, none of which is appealing if simplicity equals efficiency. On the other hand, I never really understood diff-eqs; ODEs seem like polynomials over self similar polynomials, to me, hence "recurrent"; All the other diff-eqs I can't begin to fathom.
- apl 7y agoThere's many perspectives on everything. Deep ConvNets, for instance, can be expressed as a continuously evolving ODE. Here's a fantastic paper on this view: https://papers.nips.cc/paper/7892-neural-ordinary-differential-equations https://papers.nips.cc/paper/7892-neural-ordinary-differenti...
- ummonk 7y agoYeah, that's a reasonable way of looking at it. Convolution would be a partial differential equation in this view.
- deleted 7y ago[deleted]