10 ms·

Respectfully I don't think you're characterization of early 20th century and today's research is accurate at all. There is a stupendous amount of new science b

Respectfully I don't think you're characterization of early 20th century and today's research is accurate at all. There is a stupendous amount of new science being carried out. Science that we'll look back on 80 years later (just as we look back on the science of the 1920s) and recognize how impactful that science has been. In fact, I would imagine that the science that is being worked on today will prove far more impactful than the research of the 1920s. Think about the seminal technologies that are still in their infancy! Deep learning in it's current form (AKA the form that actually works) is ~10 years old. That's it. And in those ~10 years we've gone from cute theory to being able to convincingly emulate video, translate languages, solve dictation, and generate near-human quality text. Then there's CRISPR, which could potentially cure 1000s of diseases and dramatically improve diagnostic tools, and is similarly ~20 years old. There are many thousands of other groundbreaking technologies that are being worked on as we speak. Contemporary scientists are conducting this research.

YeGoblynQueenne 5y ago
Deep learning has nothing to do with "science" and the last algorithmic advances that enabled it happened rather more than 10 years ago: the neocognitron, in 1979, the Long-Short Term Memory recurrent neural nets in 1995, backpropagation (for neural network training) in 1986, etc. In general, all the interesting work that enabled today's deep learning boom happened towards the end of the 20th century and recent advances are primarily owed to increases in computational power and availability of data sets. Says not me, but Geoff Hinton: Geoffrey Hinton: I think it’s mainly because of the amount of computation and the amount of data now around but it’s also partly because there have been some technical improvements in the algorithms. Particularly in the algorithms for doing unsupervised learning where you’re not told what the right answer is but the main thing is the computation and the amount of data. The algorithms we had in the old days would have worked perfectly well if computers had been a million times faster and datasets had been a million times bigger but if we’d said that thirty years ago people would have just laughed. http://techjaw.com/2015/06/07/geoffrey-hinton-deep-learning-in-baby-steps-and-the-future-of-google-and-ai/ http://techjaw.com/2015/06/07/geoffrey-hinton-deep-learning-...
- danny_codes 5y ago
  Google published "Attention is all you need" in 2017, which introduced the Transformer Deep Learning Architecture. This is the general framework by which GPT-3 was trained, and is increasingly replacing most other architectures in NLP tasks. So.. I guess I disagree strongly with your premise? Look, it takes some time for these ideas to propagate to non-experts. In 20 years a hacker-news poster will likely post the same comment, referencing the 2017 Transformer paper to enforce the idea that all the _good_ science in DL was done years ago.
  - deleted 5y ago
    [deleted]
  - YeGoblynQueenne 5y ago
    You're mistaking the hot new fad of Transformers for "new science" and the field of NLP for a scientific field. Transformers have only taken over NLP because that entire field of research has run out of new ideas, and has been desperately grasping for straws in the last twenty years. Consequently, NLP researchers have been jumping on this bandwagon or that ever since I can remember. I enquired about a PhD in NLP as soon as I finished my degree, in 2011, but by 2014, when I finished my Master's, I knew that was the last thing I wanted to study: a subject riddled with sloppy practices, flooded with shaky empirical results, without any real theory to guide it, without even any useful benchmarks or credible metrics that could justify the empirical direction. That is why the GPT-3s and the BERTs and all those data- and power-hungry monstrosities reign supreme: because NLP is built like a house of cards, with no scientific foundation whatsoever, so that anybody can claim anything they like and there is almost nobody left to challenge even the most outlandish claims. >> In 20 years a hacker-news poster will likely post the same comment, referencing the 2017 Transformer paper to enforce the idea that all the _good_ science in DL was done years ago. On the contrary, my expectation is that in 20 years from now people will point to Transformers and laugh and joke about how misguided people were "back then" to think that just building larger neural nets would lead to AGI, as they do now for earlier AI approaches. What I'm not convinced is that you are any different than the "non-expert" in your comment, to whom "it takes some time" for ideas to propagate. You are holding up Transformers as some kind of scientific breakthrough but that is only the most obvious conclusion to draw from the overhyping of the approach on social media and the lay press.

YeGoblynQueenne 5y ago

Deep learning has nothing to do with "science" and the last algorithmic advances that enabled it happened rather more than 10 years ago: the neocognitron, in 1979, the Long-Short Term Memory recurrent neural nets in 1995, backpropagation (for neural network training) in 1986, etc. In general, all the interesting work that enabled today's deep learning boom happened towards the end of the 20th century and recent advances are primarily owed to increases in computational power and availability of data sets. Says not me, but Geoff Hinton: Geoffrey Hinton: I think it’s mainly because of the amount of computation and the amount of data now around but it’s also partly because there have been some technical improvements in the algorithms. Particularly in the algorithms for doing unsupervised learning where you’re not told what the right answer is but the main thing is the computation and the amount of data. The algorithms we had in the old days would have worked perfectly well if computers had been a million times faster and datasets had been a million times bigger but if we’d said that thirty years ago people would have just laughed. http://techjaw.com/2015/06/07/geoffrey-hinton-deep-learning-in-baby-steps-and-the-future-of-google-and-ai/ http://techjaw.com/2015/06/07/geoffrey-hinton-deep-learning-...

danny_codes 5y ago
Google published "Attention is all you need" in 2017, which introduced the Transformer Deep Learning Architecture. This is the general framework by which GPT-3 was trained, and is increasingly replacing most other architectures in NLP tasks. So.. I guess I disagree strongly with your premise? Look, it takes some time for these ideas to propagate to non-experts. In 20 years a hacker-news poster will likely post the same comment, referencing the 2017 Transformer paper to enforce the idea that all the _good_ science in DL was done years ago.
- deleted 5y ago
  [deleted]
- YeGoblynQueenne 5y ago
  You're mistaking the hot new fad of Transformers for "new science" and the field of NLP for a scientific field. Transformers have only taken over NLP because that entire field of research has run out of new ideas, and has been desperately grasping for straws in the last twenty years. Consequently, NLP researchers have been jumping on this bandwagon or that ever since I can remember. I enquired about a PhD in NLP as soon as I finished my degree, in 2011, but by 2014, when I finished my Master's, I knew that was the last thing I wanted to study: a subject riddled with sloppy practices, flooded with shaky empirical results, without any real theory to guide it, without even any useful benchmarks or credible metrics that could justify the empirical direction. That is why the GPT-3s and the BERTs and all those data- and power-hungry monstrosities reign supreme: because NLP is built like a house of cards, with no scientific foundation whatsoever, so that anybody can claim anything they like and there is almost nobody left to challenge even the most outlandish claims. >> In 20 years a hacker-news poster will likely post the same comment, referencing the 2017 Transformer paper to enforce the idea that all the _good_ science in DL was done years ago. On the contrary, my expectation is that in 20 years from now people will point to Transformers and laugh and joke about how misguided people were "back then" to think that just building larger neural nets would lead to AGI, as they do now for earlier AI approaches. What I'm not convinced is that you are any different than the "non-expert" in your comment, to whom "it takes some time" for ideas to propagate. You are holding up Transformers as some kind of scientific breakthrough but that is only the most obvious conclusion to draw from the overhyping of the approach on social media and the lay press.