3 ms·

Google published "Attention is all you need" in 2017, which introduced the Transformer Deep Learning Architecture. This is the general framework by which GPT-3

Google published "Attention is all you need" in 2017, which introduced the Transformer Deep Learning Architecture. This is the general framework by which GPT-3 was trained, and is increasingly replacing most other architectures in NLP tasks. So.. I guess I disagree strongly with your premise? Look, it takes some time for these ideas to propagate to non-experts. In 20 years a hacker-news poster will likely post the same comment, referencing the 2017 Transformer paper to enforce the idea that all the _good_ science in DL was done years ago.

deleted 5y ago
[deleted]
YeGoblynQueenne 5y ago
You're mistaking the hot new fad of Transformers for "new science" and the field of NLP for a scientific field. Transformers have only taken over NLP because that entire field of research has run out of new ideas, and has been desperately grasping for straws in the last twenty years. Consequently, NLP researchers have been jumping on this bandwagon or that ever since I can remember. I enquired about a PhD in NLP as soon as I finished my degree, in 2011, but by 2014, when I finished my Master's, I knew that was the last thing I wanted to study: a subject riddled with sloppy practices, flooded with shaky empirical results, without any real theory to guide it, without even any useful benchmarks or credible metrics that could justify the empirical direction. That is why the GPT-3s and the BERTs and all those data- and power-hungry monstrosities reign supreme: because NLP is built like a house of cards, with no scientific foundation whatsoever, so that anybody can claim anything they like and there is almost nobody left to challenge even the most outlandish claims. >> In 20 years a hacker-news poster will likely post the same comment, referencing the 2017 Transformer paper to enforce the idea that all the _good_ science in DL was done years ago. On the contrary, my expectation is that in 20 years from now people will point to Transformers and laugh and joke about how misguided people were "back then" to think that just building larger neural nets would lead to AGI, as they do now for earlier AI approaches. What I'm not convinced is that you are any different than the "non-expert" in your comment, to whom "it takes some time" for ideas to propagate. You are holding up Transformers as some kind of scientific breakthrough but that is only the most obvious conclusion to draw from the overhyping of the approach on social media and the lay press.
- dekhn 5y ago
  I work in the field and transformers have literally transformed science.
  - YeGoblynQueenne 5y ago
    Can you please clarify which "field" is that and how do you "work" in it?
    - dekhn 5y ago
      the field is the application of machine learning to biology (specifically drug design, but also the wider area of computational biology in all its forms). Here's some of my prior work: https://arxiv.org/abs/1502.02072 https://arxiv.org/abs/1502.02072 however that doesn't use transformers. The DeepMind work on protein structure prediction does, and it represents one of the greatest technical achievements of the modern era. Transformers basically took anything that was "sequence learning" (large fractions of computational biology are) and made it work 100X better over night.
      - YeGoblynQueenne 5y ago
        Thank you for clarifying. I see that the paper has six authors two of whom declare equal contribution. Could you please further clarify what was your personal contribution to the paper (to the extent that this doesn't unecessarily expose your personal information)? Regarding Transformers, the results on protein structure prediction are certainly impressive, but they are not enough to justify your comment that "transformers have literally transformed science". At most we can tell that they have considerably improved protein structure prediction. Perhaps your comment was an exaggeration? Edit: I can't find the paper on the ICML 2015 website, although I can see that it has a few hundred citations, as a preprint, which is not uncommon these days. But, could you point to some published work you contributed to?

deleted 5y ago

[deleted]

YeGoblynQueenne 5y ago

You're mistaking the hot new fad of Transformers for "new science" and the field of NLP for a scientific field. Transformers have only taken over NLP because that entire field of research has run out of new ideas, and has been desperately grasping for straws in the last twenty years. Consequently, NLP researchers have been jumping on this bandwagon or that ever since I can remember. I enquired about a PhD in NLP as soon as I finished my degree, in 2011, but by 2014, when I finished my Master's, I knew that was the last thing I wanted to study: a subject riddled with sloppy practices, flooded with shaky empirical results, without any real theory to guide it, without even any useful benchmarks or credible metrics that could justify the empirical direction. That is why the GPT-3s and the BERTs and all those data- and power-hungry monstrosities reign supreme: because NLP is built like a house of cards, with no scientific foundation whatsoever, so that anybody can claim anything they like and there is almost nobody left to challenge even the most outlandish claims. >> In 20 years a hacker-news poster will likely post the same comment, referencing the 2017 Transformer paper to enforce the idea that all the _good_ science in DL was done years ago. On the contrary, my expectation is that in 20 years from now people will point to Transformers and laugh and joke about how misguided people were "back then" to think that just building larger neural nets would lead to AGI, as they do now for earlier AI approaches. What I'm not convinced is that you are any different than the "non-expert" in your comment, to whom "it takes some time" for ideas to propagate. You are holding up Transformers as some kind of scientific breakthrough but that is only the most obvious conclusion to draw from the overhyping of the approach on social media and the lay press.

dekhn 5y ago
I work in the field and transformers have literally transformed science.
- YeGoblynQueenne 5y ago
  Can you please clarify which "field" is that and how do you "work" in it?
  - dekhn 5y ago
    the field is the application of machine learning to biology (specifically drug design, but also the wider area of computational biology in all its forms). Here's some of my prior work: https://arxiv.org/abs/1502.02072 https://arxiv.org/abs/1502.02072 however that doesn't use transformers. The DeepMind work on protein structure prediction does, and it represents one of the greatest technical achievements of the modern era. Transformers basically took anything that was "sequence learning" (large fractions of computational biology are) and made it work 100X better over night.
    - YeGoblynQueenne 5y ago
      Thank you for clarifying. I see that the paper has six authors two of whom declare equal contribution. Could you please further clarify what was your personal contribution to the paper (to the extent that this doesn't unecessarily expose your personal information)? Regarding Transformers, the results on protein structure prediction are certainly impressive, but they are not enough to justify your comment that "transformers have literally transformed science". At most we can tell that they have considerably improved protein structure prediction. Perhaps your comment was an exaggeration? Edit: I can't find the paper on the ICML 2015 website, although I can see that it has a few hundred citations, as a preprint, which is not uncommon these days. But, could you point to some published work you contributed to?