3 ms·

Besides like 10 papers a year almost everything else is incremental tweaks that are largely immaterial

by typon 8y ago

mlthoughts2018 8y ago
I agree, but it seems like we went past a bend in the hype curve with how this plays out in deep learning vs. other fields or historically in machine learning. A paper like this used to be a minor thing. Neat result, A for effort, great to see if the authors take the time to generate PRs adding the updated method into mainstream packages and libraries. But no more than that. Now, with deep learning, every little result has to be trumpeted out with blog posts, claims about what is “state of the art” on some cherry picked example cases, and you’ll be treated like a dunce if you don’t get push notifications direct from arxiv and drop what you’re doing to read everything immediately. It’s a form of credit inflation. Researchers want sweet, sweet deep learning hype credit for fame & fortune. Instead of just admitting that a lot of results are inapplicable to most practitioners and that’s totally fine and common in published research ... it has to be always a hype arms race about who knows more trivia.
- deleted 8y ago
  [deleted]
- Macuyiko 8y ago
  Very well said. Edit: feel like I should clarify a bit to avoid the "me too +1" style comment. I've been torn in recent months between fast.ai's great hands on and to the point style of teaching (which I think is great and have strived to use as an example in my own courses) and their somewhat overhyped presentation of the fast.ai library. It's not bad, but ultimately a collection of rather hacky function calls that are build on top of other frameworks. This being said, from an academicians point of view (to which I so far can still consider myself to belong to, albeit perhaps not for much longer), I do appreciate most things Jeremy and his team has brought into the spotlight, e.g. insights around learning rate and embeddings for sparse categoricals. What amuses me most is that these things are by no means very new, but I enjoy the spirit of fast.ai pointing out that these things are not new and should be considered (again) today in the midst of a research community that is going way too fast. Then again, I also think this post is a bit too much hype centric and hence falling into the same traps. I am also somewhat disappointed with fast.ai's claims to state of art lying mainly in "known" image detection and text regions, whilst not talking much or at all about e.g. RL, in courses or papers. I think the work being done here by OpenAI for instance during the past few months is far more exciting.
- skybrian 8y ago
  Not my field, but from a humble programmer's perspective, a 5x to 10x speedup sure sounds like a significant result. It certainly would be in most areas where we already have efficient algorithms. Why is this one "inapplicable to most practitioners"?
  - mlthoughts2018 8y ago
    Because the published result is almost surely not repeatable in each nuanced case for day to day practitioners that have custom layers, extra regularization, heterogeneous hardware for training, etc. When you’ve seen enough of these same papers come and go, your eyes glaze over. It’s like p-hacking, but for finding tasks that show X% improvement. Basically it needs a huge Your Mileage May Vary sticker attached, because for practitioner’s specific situations, you still have to do the experiments to tune these parameters. Use AdamW or rmsprop or adagrad or ... and use dropout (what fraction?) and use penalty terms, and use early stopping, etc. etc., ... ? You can’t rely on papers like this for much insight beyond just saying AdamW is yet another tool in the toolbox... and that’s all. It would be comparable to someone writing a paper about some slight variation of Timsort that is ~10% faster on some empirical distribution of data, like data already 80% sorted. Is it good for you? You have to do legwork to know. Such a result is perfectly good research, but would be silly for a bunch of blog posts about “state of the art” sorting, and plugs for some software package that has it.
    - jph00 8y ago
      > the published result is almost surely not repeatable in each nuanced case for day to day practitioners that have custom layers, extra regularization, heterogeneous hardware for training As shown in the post and linked code, it works very well with custom layers and extra regularization. Hardware does not impact the results at all. The results hold across CNNs, RNNs, and even QRNNs, across a wide variety of datasets. We literally tried to find a real-world NLP or vision dataset it didn't work with, and failed to do so.
      - mlthoughts2018 8y ago
- jph00 8y ago
  > It’s a form of credit inflation. Researchers want sweet, sweet deep learning hype credit for fame & fortune. The authors of the AdamW or super-convergence papers did not write the blog post - we did. There is no connection between the authors of the papers mentioned and the authors of the blog post. It's simply a genuine and honest attempt by us to study these papers and share what we learned, in order to help others get similar results.
  - mlthoughts2018 8y ago
    I was referring only to the fastai post plugging the fastai toolkit. The research papers themselves seem fine enough, as they aren’t making a bombastic claim like saying AdamW + 1cycle should be considered the general default starting point for training any NN.

Besides like 10 papers a year almost everything else is incremental tweaks that are largely immaterial

by typon 8y ago

Besides like 10 papers a year almost everything else is incremental tweaks that are largely immaterial

mlthoughts2018 8y ago

I agree, but it seems like we went past a bend in the hype curve with how this plays out in deep learning vs. other fields or historically in machine learning. A paper like this used to be a minor thing. Neat result, A for effort, great to see if the authors take the time to generate PRs adding the updated method into mainstream packages and libraries. But no more than that. Now, with deep learning, every little result has to be trumpeted out with blog posts, claims about what is “state of the art” on some cherry picked example cases, and you’ll be treated like a dunce if you don’t get push notifications direct from arxiv and drop what you’re doing to read everything immediately. It’s a form of credit inflation. Researchers want sweet, sweet deep learning hype credit for fame & fortune. Instead of just admitting that a lot of results are inapplicable to most practitioners and that’s totally fine and common in published research ... it has to be always a hype arms race about who knows more trivia.

deleted 8y ago
[deleted]
Macuyiko 8y ago
Very well said. Edit: feel like I should clarify a bit to avoid the "me too +1" style comment. I've been torn in recent months between fast.ai's great hands on and to the point style of teaching (which I think is great and have strived to use as an example in my own courses) and their somewhat overhyped presentation of the fast.ai library. It's not bad, but ultimately a collection of rather hacky function calls that are build on top of other frameworks. This being said, from an academicians point of view (to which I so far can still consider myself to belong to, albeit perhaps not for much longer), I do appreciate most things Jeremy and his team has brought into the spotlight, e.g. insights around learning rate and embeddings for sparse categoricals. What amuses me most is that these things are by no means very new, but I enjoy the spirit of fast.ai pointing out that these things are not new and should be considered (again) today in the midst of a research community that is going way too fast. Then again, I also think this post is a bit too much hype centric and hence falling into the same traps. I am also somewhat disappointed with fast.ai's claims to state of art lying mainly in "known" image detection and text regions, whilst not talking much or at all about e.g. RL, in courses or papers. I think the work being done here by OpenAI for instance during the past few months is far more exciting.
skybrian 8y ago
Not my field, but from a humble programmer's perspective, a 5x to 10x speedup sure sounds like a significant result. It certainly would be in most areas where we already have efficient algorithms. Why is this one "inapplicable to most practitioners"?
- mlthoughts2018 8y ago
  Because the published result is almost surely not repeatable in each nuanced case for day to day practitioners that have custom layers, extra regularization, heterogeneous hardware for training, etc. When you’ve seen enough of these same papers come and go, your eyes glaze over. It’s like p-hacking, but for finding tasks that show X% improvement. Basically it needs a huge Your Mileage May Vary sticker attached, because for practitioner’s specific situations, you still have to do the experiments to tune these parameters. Use AdamW or rmsprop or adagrad or ... and use dropout (what fraction?) and use penalty terms, and use early stopping, etc. etc., ... ? You can’t rely on papers like this for much insight beyond just saying AdamW is yet another tool in the toolbox... and that’s all. It would be comparable to someone writing a paper about some slight variation of Timsort that is ~10% faster on some empirical distribution of data, like data already 80% sorted. Is it good for you? You have to do legwork to know. Such a result is perfectly good research, but would be silly for a bunch of blog posts about “state of the art” sorting, and plugs for some software package that has it.
  - jph00 8y ago
    > the published result is almost surely not repeatable in each nuanced case for day to day practitioners that have custom layers, extra regularization, heterogeneous hardware for training As shown in the post and linked code, it works very well with custom layers and extra regularization. Hardware does not impact the results at all. The results hold across CNNs, RNNs, and even QRNNs, across a wide variety of datasets. We literally tried to find a real-world NLP or vision dataset it didn't work with, and failed to do so.
    - mlthoughts2018 8y ago
jph00 8y ago
> It’s a form of credit inflation. Researchers want sweet, sweet deep learning hype credit for fame & fortune. The authors of the AdamW or super-convergence papers did not write the blog post - we did. There is no connection between the authors of the papers mentioned and the authors of the blog post. It's simply a genuine and honest attempt by us to study these papers and share what we learned, in order to help others get similar results.
- mlthoughts2018 8y ago
  I was referring only to the fastai post plugging the fastai toolkit. The research papers themselves seem fine enough, as they aren’t making a bombastic claim like saying AdamW + 1cycle should be considered the general default starting point for training any NN.