5 ms·

This reads a bit too much like a plug for fastai’s framework, which, frankly, is not very good and training performance results are not much of a reason to care

by mlthoughts2018 8y ago

This reads a bit too much like a plug for fastai’s framework, which, frankly, is not very good and training performance results are not much of a reason to care about it (for instance, adding AdamW to Keras is basically trivial). In most use cases, the difference between these optimizer choices, and how they interact with initialization schemes and metaparameters, is extremely overkill. If you’re training a pretty standard variant of e.g. ResNet50, for some fairly common classification or localization task, splitting hairs between these things just doesn’t matter. I actually really think these optimizer hacks are not a productive line of research for deep learning. Too many projects try to do just the slightest amount of incremental tweaks to an optimization scheme to eke out something they can call “state of the art” and milk it for conference presentations and the release of some sexy new deep learning package. But these changes are often entirely immaterial except for the very largest network training schemes, which are usually already doing their own highly customized and distributed optimization schemes anyway. Especially when it’s tied to making a plug for a framework, like this is for fastai, it’s just too focused on hype and creates this peacock feather effect where everyone had to spread themselves so thin just to keep up with all this middling, incremental tweaks, that it actually very likely inhibits research that might actually produce difference of kind results.

typon 8y ago
Besides like 10 papers a year almost everything else is incremental tweaks that are largely immaterial
- mlthoughts2018 8y ago
  I agree, but it seems like we went past a bend in the hype curve with how this plays out in deep learning vs. other fields or historically in machine learning. A paper like this used to be a minor thing. Neat result, A for effort, great to see if the authors take the time to generate PRs adding the updated method into mainstream packages and libraries. But no more than that. Now, with deep learning, every little result has to be trumpeted out with blog posts, claims about what is “state of the art” on some cherry picked example cases, and you’ll be treated like a dunce if you don’t get push notifications direct from arxiv and drop what you’re doing to read everything immediately. It’s a form of credit inflation. Researchers want sweet, sweet deep learning hype credit for fame & fortune. Instead of just admitting that a lot of results are inapplicable to most practitioners and that’s totally fine and common in published research ... it has to be always a hype arms race about who knows more trivia.
  - deleted 8y ago
    [deleted]
  - Macuyiko 8y ago
    Very well said. Edit: feel like I should clarify a bit to avoid the "me too +1" style comment. I've been torn in recent months between fast.ai's great hands on and to the point style of teaching (which I think is great and have strived to use as an example in my own courses) and their somewhat overhyped presentation of the fast.ai library. It's not bad, but ultimately a collection of rather hacky function calls that are build on top of other frameworks. This being said, from an academicians point of view (to which I so far can still consider myself to belong to, albeit perhaps not for much longer), I do appreciate most things Jeremy and his team has brought into the spotlight, e.g. insights around learning rate and embeddings for sparse categoricals. What amuses me most is that these things are by no means very new, but I enjoy the spirit of fast.ai pointing out that these things are not new and should be considered (again) today in the midst of a research community that is going way too fast. Then again, I also think this post is a bit too much hype centric and hence falling into the same traps. I am also somewhat disappointed with fast.ai's claims to state of art lying mainly in "known" image detection and text regions, whilst not talking much or at all about e.g. RL, in courses or papers. I think the work being done here by OpenAI for instance during the past few months is far more exciting.
  - skybrian 8y ago
    Not my field, but from a humble programmer's perspective, a 5x to 10x speedup sure sounds like a significant result. It certainly would be in most areas where we already have efficient algorithms. Why is this one "inapplicable to most practitioners"?
    - mlthoughts2018 8y ago
      Because the published result is almost surely not repeatable in each nuanced case for day to day practitioners that have custom layers, extra regularization, heterogeneous hardware for training, etc. When you’ve seen enough of these same papers come and go, your eyes glaze over. It’s like p-hacking, but for finding tasks that show X% improvement. Basically it needs a huge Your Mileage May Vary sticker attached, because for practitioner’s specific situations, you still have to do the experiments to tune these parameters. Use AdamW or rmsprop or adagrad or ... and use dropout (what fraction?) and use penalty terms, and use early stopping, etc. etc., ... ? You can’t rely on papers like this for much insight beyond just saying AdamW is yet another tool in the toolbox... and that’s all. It would be comparable to someone writing a paper about some slight variation of Timsort that is ~10% faster on some empirical distribution of data, like data already 80% sorted. Is it good for you? You have to do legwork to know. Such a result is perfectly good research, but would be silly for a bunch of blog posts about “state of the art” sorting, and plugs for some software package that has it.
minimaxir 8y ago
Relatedly, optimizer performance can behave very wildly on different types of models. Although many new DL projects use Adam as a baseline, in one of my projects simple RMSprop performed significantly better after a bit of testing. The optimizer is just another hyperparameter to tune.
bitL 8y ago
I have an orthogonal perspective - I really think that current Deep Learning works only because somehow those non-linear optimization tricks like Adam fit the model structure - you as a model-preparer restrict the search space with some structural rules (CNN, LSTM, 1x1 conv etc.) and unleashing fairly simple non-linear optimization procedures suddenly works as they don't have to wander unguided around much more complex space (like why these simple optimizers don't work well with large fully-connected networks, which in theory should be better than any specializations). By combining both structural rules (reducing dimensionality, providing some guidance in form of restricted connectivity) and making optimization procedures that can take advantage of it IMO gives us current great DL performance; if we can squeeze even more from optimizers that could help, and I am glad fast.ai people are doing it. I simply view it in a dual(-metaoptimization) way and both structural and optimization "match" should be researched.
- mlthoughts2018 8y ago
  What you call “structural rules” is just the machine learning term regularization. Dropout, early stopping, batch norm, advanced initialization, etc. are all equally useful ways of regularizing models as are explicit penalty terms or implicit restrictions to subspaces of the possible weight space. Whichever of these engineering tricks works best for your use case either (a) doesn’t matter much because they all work about the same or (b) is a matter of specialized experimentation and case-by-case analysis anyway. Also, the theory of using this type of approach for model fitting is not new in modern deep learning. Older models like RANSAC did similar things even when the model itself is far simpler and there is no network topology effect. Arguably, advanced MCMC methods like NUTS, or approximation methods like ADVI, are even more formalized approaches to the deep learning hacks, where the network structure and regularization constraints defines a prior over model parameter space, and when combining with the data to get a likelihood model, you really just want something that draws from the posterior over model parameter space. The reason it’s a “hack” in deep learning is that nobody is trying to formally define the posterior based on the model and the complicated metaparameters, rather they are just adding tweaks on top of tweaks to use deeply inefficient momentum SGD methods to sample from the posterior and optimize a la simulated annealing, instead of making the models truly amenable to something like NUTS. Which IMO is yet another reason to view incremental optimizer tweaks as a negative thing, slowing down and distracting the community.
master_yoda_1 8y ago
I totally agree with you. Here is one more example https://openreview.net/forum?id=SyrGJYlRZ https://openreview.net/forum?id=SyrGJYlRZ

typon 8y ago

Besides like 10 papers a year almost everything else is incremental tweaks that are largely immaterial

mlthoughts2018 8y ago
I agree, but it seems like we went past a bend in the hype curve with how this plays out in deep learning vs. other fields or historically in machine learning. A paper like this used to be a minor thing. Neat result, A for effort, great to see if the authors take the time to generate PRs adding the updated method into mainstream packages and libraries. But no more than that. Now, with deep learning, every little result has to be trumpeted out with blog posts, claims about what is “state of the art” on some cherry picked example cases, and you’ll be treated like a dunce if you don’t get push notifications direct from arxiv and drop what you’re doing to read everything immediately. It’s a form of credit inflation. Researchers want sweet, sweet deep learning hype credit for fame & fortune. Instead of just admitting that a lot of results are inapplicable to most practitioners and that’s totally fine and common in published research ... it has to be always a hype arms race about who knows more trivia.
- deleted 8y ago
  [deleted]
- Macuyiko 8y ago
  Very well said. Edit: feel like I should clarify a bit to avoid the "me too +1" style comment. I've been torn in recent months between fast.ai's great hands on and to the point style of teaching (which I think is great and have strived to use as an example in my own courses) and their somewhat overhyped presentation of the fast.ai library. It's not bad, but ultimately a collection of rather hacky function calls that are build on top of other frameworks. This being said, from an academicians point of view (to which I so far can still consider myself to belong to, albeit perhaps not for much longer), I do appreciate most things Jeremy and his team has brought into the spotlight, e.g. insights around learning rate and embeddings for sparse categoricals. What amuses me most is that these things are by no means very new, but I enjoy the spirit of fast.ai pointing out that these things are not new and should be considered (again) today in the midst of a research community that is going way too fast. Then again, I also think this post is a bit too much hype centric and hence falling into the same traps. I am also somewhat disappointed with fast.ai's claims to state of art lying mainly in "known" image detection and text regions, whilst not talking much or at all about e.g. RL, in courses or papers. I think the work being done here by OpenAI for instance during the past few months is far more exciting.
- skybrian 8y ago
  Not my field, but from a humble programmer's perspective, a 5x to 10x speedup sure sounds like a significant result. It certainly would be in most areas where we already have efficient algorithms. Why is this one "inapplicable to most practitioners"?
  - mlthoughts2018 8y ago
    Because the published result is almost surely not repeatable in each nuanced case for day to day practitioners that have custom layers, extra regularization, heterogeneous hardware for training, etc. When you’ve seen enough of these same papers come and go, your eyes glaze over. It’s like p-hacking, but for finding tasks that show X% improvement. Basically it needs a huge Your Mileage May Vary sticker attached, because for practitioner’s specific situations, you still have to do the experiments to tune these parameters. Use AdamW or rmsprop or adagrad or ... and use dropout (what fraction?) and use penalty terms, and use early stopping, etc. etc., ... ? You can’t rely on papers like this for much insight beyond just saying AdamW is yet another tool in the toolbox... and that’s all. It would be comparable to someone writing a paper about some slight variation of Timsort that is ~10% faster on some empirical distribution of data, like data already 80% sorted. Is it good for you? You have to do legwork to know. Such a result is perfectly good research, but would be silly for a bunch of blog posts about “state of the art” sorting, and plugs for some software package that has it.

minimaxir 8y ago

Relatedly, optimizer performance can behave very wildly on different types of models. Although many new DL projects use Adam as a baseline, in one of my projects simple RMSprop performed significantly better after a bit of testing. The optimizer is just another hyperparameter to tune.

bitL 8y ago

I have an orthogonal perspective - I really think that current Deep Learning works only because somehow those non-linear optimization tricks like Adam fit the model structure - you as a model-preparer restrict the search space with some structural rules (CNN, LSTM, 1x1 conv etc.) and unleashing fairly simple non-linear optimization procedures suddenly works as they don't have to wander unguided around much more complex space (like why these simple optimizers don't work well with large fully-connected networks, which in theory should be better than any specializations). By combining both structural rules (reducing dimensionality, providing some guidance in form of restricted connectivity) and making optimization procedures that can take advantage of it IMO gives us current great DL performance; if we can squeeze even more from optimizers that could help, and I am glad fast.ai people are doing it. I simply view it in a dual(-metaoptimization) way and both structural and optimization "match" should be researched.

mlthoughts2018 8y ago
What you call “structural rules” is just the machine learning term regularization. Dropout, early stopping, batch norm, advanced initialization, etc. are all equally useful ways of regularizing models as are explicit penalty terms or implicit restrictions to subspaces of the possible weight space. Whichever of these engineering tricks works best for your use case either (a) doesn’t matter much because they all work about the same or (b) is a matter of specialized experimentation and case-by-case analysis anyway. Also, the theory of using this type of approach for model fitting is not new in modern deep learning. Older models like RANSAC did similar things even when the model itself is far simpler and there is no network topology effect. Arguably, advanced MCMC methods like NUTS, or approximation methods like ADVI, are even more formalized approaches to the deep learning hacks, where the network structure and regularization constraints defines a prior over model parameter space, and when combining with the data to get a likelihood model, you really just want something that draws from the posterior over model parameter space. The reason it’s a “hack” in deep learning is that nobody is trying to formally define the posterior based on the model and the complicated metaparameters, rather they are just adding tweaks on top of tweaks to use deeply inefficient momentum SGD methods to sample from the posterior and optimize a la simulated annealing, instead of making the models truly amenable to something like NUTS. Which IMO is yet another reason to view incremental optimizer tweaks as a negative thing, slowing down and distracting the community.

master_yoda_1 8y ago

I totally agree with you. Here is one more example https://openreview.net/forum?id=SyrGJYlRZ https://openreview.net/forum?id=SyrGJYlRZ