3 ms·

I have an orthogonal perspective - I really think that current Deep Learning works only because somehow those non-linear optimization tricks like Adam fit the m

by bitL 8y ago

I have an orthogonal perspective - I really think that current Deep Learning works only because somehow those non-linear optimization tricks like Adam fit the model structure - you as a model-preparer restrict the search space with some structural rules (CNN, LSTM, 1x1 conv etc.) and unleashing fairly simple non-linear optimization procedures suddenly works as they don't have to wander unguided around much more complex space (like why these simple optimizers don't work well with large fully-connected networks, which in theory should be better than any specializations). By combining both structural rules (reducing dimensionality, providing some guidance in form of restricted connectivity) and making optimization procedures that can take advantage of it IMO gives us current great DL performance; if we can squeeze even more from optimizers that could help, and I am glad fast.ai people are doing it. I simply view it in a dual(-metaoptimization) way and both structural and optimization "match" should be researched.

mlthoughts2018 8y ago
What you call “structural rules” is just the machine learning term regularization. Dropout, early stopping, batch norm, advanced initialization, etc. are all equally useful ways of regularizing models as are explicit penalty terms or implicit restrictions to subspaces of the possible weight space. Whichever of these engineering tricks works best for your use case either (a) doesn’t matter much because they all work about the same or (b) is a matter of specialized experimentation and case-by-case analysis anyway. Also, the theory of using this type of approach for model fitting is not new in modern deep learning. Older models like RANSAC did similar things even when the model itself is far simpler and there is no network topology effect. Arguably, advanced MCMC methods like NUTS, or approximation methods like ADVI, are even more formalized approaches to the deep learning hacks, where the network structure and regularization constraints defines a prior over model parameter space, and when combining with the data to get a likelihood model, you really just want something that draws from the posterior over model parameter space. The reason it’s a “hack” in deep learning is that nobody is trying to formally define the posterior based on the model and the complicated metaparameters, rather they are just adding tweaks on top of tweaks to use deeply inefficient momentum SGD methods to sample from the posterior and optimize a la simulated annealing, instead of making the models truly amenable to something like NUTS. Which IMO is yet another reason to view incremental optimizer tweaks as a negative thing, slowing down and distracting the community.
- HelloNurse 8y ago
  There doesn't seem to be much interest, among NN researchers, for finding the right tool for the job, in terms of understanding the task at hand and engineering appropriate inputs and features, appropriate neural network types and appropriate training and optimization techniques. Finding the right job for the tool, to the point of repurposing and adapting "proven" network structures or even already trained networks, is an easier and more popular approach.
- cs702 8y ago
  Yes, deep learning today is a trade. If we want to be more charitable, we can call it an experimental science -- some work truly deserves the moniker. The state of deep learning today is analogous to the state of bridge-engineering before the advent of physics: https://www.technologyreview.com/s/608911/is-ai-riding-a-one-trick-pony/ https://www.technologyreview.com/s/608911/is-ai-riding-a-one... -- everyone in the trade is aware of this, I think. But that doesn't mean we should stop building better deep-learning models, finding more efficient ways to train them, and improving state-of-the-art performance in increasingly challenging tasks.
  - amelius 8y ago
    Perhaps we can reach the singularity with slowly-trained models, and from there get fast training methods for free ;)
a-dub 8y ago
If your perspective were truly orthogonal, wouldn't that mean, by definition, that you have nothing to say at all on the topic of the long term intellectual value to humanity of fiddling with loss functions and making optimizer tweaks?

by bitL 8y ago

mlthoughts2018 8y ago

What you call “structural rules” is just the machine learning term regularization. Dropout, early stopping, batch norm, advanced initialization, etc. are all equally useful ways of regularizing models as are explicit penalty terms or implicit restrictions to subspaces of the possible weight space. Whichever of these engineering tricks works best for your use case either (a) doesn’t matter much because they all work about the same or (b) is a matter of specialized experimentation and case-by-case analysis anyway. Also, the theory of using this type of approach for model fitting is not new in modern deep learning. Older models like RANSAC did similar things even when the model itself is far simpler and there is no network topology effect. Arguably, advanced MCMC methods like NUTS, or approximation methods like ADVI, are even more formalized approaches to the deep learning hacks, where the network structure and regularization constraints defines a prior over model parameter space, and when combining with the data to get a likelihood model, you really just want something that draws from the posterior over model parameter space. The reason it’s a “hack” in deep learning is that nobody is trying to formally define the posterior based on the model and the complicated metaparameters, rather they are just adding tweaks on top of tweaks to use deeply inefficient momentum SGD methods to sample from the posterior and optimize a la simulated annealing, instead of making the models truly amenable to something like NUTS. Which IMO is yet another reason to view incremental optimizer tweaks as a negative thing, slowing down and distracting the community.

HelloNurse 8y ago
There doesn't seem to be much interest, among NN researchers, for finding the right tool for the job, in terms of understanding the task at hand and engineering appropriate inputs and features, appropriate neural network types and appropriate training and optimization techniques. Finding the right job for the tool, to the point of repurposing and adapting "proven" network structures or even already trained networks, is an easier and more popular approach.
cs702 8y ago
Yes, deep learning today is a trade. If we want to be more charitable, we can call it an experimental science -- some work truly deserves the moniker. The state of deep learning today is analogous to the state of bridge-engineering before the advent of physics: https://www.technologyreview.com/s/608911/is-ai-riding-a-one-trick-pony/ https://www.technologyreview.com/s/608911/is-ai-riding-a-one... -- everyone in the trade is aware of this, I think. But that doesn't mean we should stop building better deep-learning models, finding more efficient ways to train them, and improving state-of-the-art performance in increasingly challenging tasks.
- amelius 8y ago
  Perhaps we can reach the singularity with slowly-trained models, and from there get fast training methods for free ;)

a-dub 8y ago

If your perspective were truly orthogonal, wouldn't that mean, by definition, that you have nothing to say at all on the topic of the long term intellectual value to humanity of fiddling with loss functions and making optimizer tweaks?