3 ms·

How has the progress stalled? Just last year there was a breakthrough in decades old problem of protein-folding. The year before that we figured out that large

How has the progress stalled? Just last year there was a breakthrough in decades old problem of protein-folding. The year before that we figured out that large language models have astonishing few shot learning abilities. CLIP introduced strong 0-shot capabilities to image recognition. Progress in image generation from descriptions has been nothing but astonishing with Dall-E and GLIDE from OpenAI. Not to mention breakthroughs in game theoretical problems like Counterfactual Regret Minimization and more recent application of Deep Learning to make CFR practical in actual games. Including continuous progress in the AlphaGo linage to AlphaZero, MuZero, and last year EfficientZero and Player of Games.

YeGoblynQueenne 5y ago
With the exception of counterfactual regret minimisation, which has nothing to do with deep learning, all those are applications of existing approaches that have become possible because of the increased expenditure of resources- processing power and data. There have been no theoretical advances, no algorithmic advances to speak of, no new knowledge learned in the last 20 years of reserach in deep learning. All that has been achieved is new SOTA on old benchmarks [1]. To make an analogy it's as if we could sort ever larger lists because we keep sorting them with ever bigger computers... but the only sorting algorithm we know is bubblesort. You bring up large language models. Language models are about as old as computational linguistics [6]. Large language models are... large. There's nothing that's new about them. "Attention" is an anthropomorphically misnamed engineering tweak of the kind that was rightly lambasted by Drew McDermot fourty years ago [7]. Self-play and reinforcement Learning are now more than 60 years old [8]. As to "few shot" learning, that is said for a language model previously trained on a copy of the entire web! In what sense is that "few shot"? It is truly shocking to hear researchers make such obviously false claims with a straight face. And that they are rewarded with publication of their work is nothing less than a scandal. Deep learning research is going nowhere. It is stuck in a rut. It is dead. It has given up the spirit. It is an ex-research field. This research field is a goner. It is dead, jim. And I'm sure it will be decades before people start to say it out loud (the current generation of reserachers will leave the dirty job to their students), but that doesn't change a thing. ______________ [1] Except in identifying new weaknesses of deep learning systems, as for example Overinterpretation in the paper I cite above, adversarial examples [2], shortcut learning [3], "the elephant in the room" [4], being "right for the wrong reasons" [5] etc etc etc etc. [2] https://arxiv.org/abs/1312.6199 https://arxiv.org/abs/1312.6199 [3] https://www.nature.com/articles/s42256-020-00257-z https://www.nature.com/articles/s42256-020-00257-z [4] https://arxiv.org/abs/1808.03305 https://arxiv.org/abs/1808.03305 [5] https://aclanthology.org/P19-1334.pdf https://aclanthology.org/P19-1334.pdf Which of course are very welcome. The only way to address the limitations of current approaches is to become aware of them and try to understand them. Unfortunately, it is only a tiny, tiny minority in the field who does that, while the majority is happy to spout nonsense like "[a billion plus] few shot" learning etc. [6] https://www.semanticscholar.org/paper/Speech-and-language-processing-an-introduction-to-Jurafsky-Martin/b54bcfca3fddc26b8889739a247a25e445818149 https://www.semanticscholar.org/paper/Speech-and-language-pr... [7] https://www.semanticscholar.org/paper/Artificial-intelligence-meets-natural-stupidity-McDermott/d9566ac89cd9b7fccc080b764aab5107430da28c https://www.semanticscholar.org/paper/Artificial-intelligenc... [8] Arhtur Samuel's checkers player beat a human champion in 1961. Donald Michie build MENACE, a reinforcement learning algorithm implemented in matchboxes in 1960: https://rodneybrooks.com/forai-machine-learning-explained/ https://rodneybrooks.com/forai-machine-learning-explained/ Ctrl+F for "Machine learning started with games" to find the relevant section.
- Isinlor 5y ago
  So much that you write is blatantly false, or stretch to the point of being as good as false. But you seem well read, so I don't think there is a point in trying to convince you. It's also interesting coming from someone researching Inductive Logic Programming that provides no results at all. From my perspective, I don't care for theoretical advances. I care about results and if scale provides results, I'm happy. For people who are just reading this exchange take a look here: https://pbs.twimg.com/media/FHHPjU0VIAUrSMA?format=jpg&name=900x900 https://pbs.twimg.com/media/FHHPjU0VIAUrSMA?format=jpg&name=... It's 16 computer generated images from the description below each image. Having capability like that is just very useful and fun. I wish you also luck in finding a "cat paying checkers in style of Salvador Dali" or a "psychedelic painting of a hamster dragon" on the internet. There is a lot of stuff on the web, but not everything.
  - deleted 5y ago
    [deleted]
  - deleted 5y ago
    [deleted]
  - YeGoblynQueenne 5y ago
    >> From my perspective, I don't care for theoretical advances. I care about results and if scale provides results, I'm happy. Like I say you can sort ever larger lists with ever bigger computers and bubblesort. But you can go a lot further if you use the brain and come up with a better sorting algorithm. But I think you would be really surprised to find out how much humans have achieved by using their brains rather than ever bigger computers. Here's a small example that happened to be on my mind recently: https://en.wikipedia.org/wiki/Copernican_Revolution https://en.wikipedia.org/wiki/Copernican_Revolution That is an example of the difference between expensive toys like Dall-E and world-changing scientific work. Though I appreciate that the difference can sometimes appear somewhat mudddled and a machine that can generate a "hamster dragon" may seem as an important scientific breakthrough. Depends on what you're more interested in: hamster dragons, or understanding how the world works. Oh well. As to this: >> It's also interesting coming from someone researching Inductive Logic Programming that provides no results at all. That's an obvious -and odious- attempt at trolling me, but unlike yours, my HN profile is linked to my real-world identity and so I am forced to respond with politenes and to represent my field with dignity. Therefore, I will point out a famous achievement of ILP with which you are sadly unfamiliar and that I expect you to immediately discount on the grounds that you didn't know about it, so it can't have been all that important. I'm linking to an article in the popular press that will be more accessible. It doesn't go into depth over the machine learning approach used but it's ILP; I link to the scholarly article immediately after: Robot Makes Scientific Discovery All by Itself For the first time, a robotic system has made a novel scientific discovery with virtually no human intellectual input. Scientists designed “Adam” to carry out the entire scientific process on its own: formulating hypotheses, designing and running experiments, analyzing data, and deciding which experiments to run next. https://www.wired.com/2009/04/robotscientist/ https://www.wired.com/2009/04/robotscientist/ And the scholarly article: Functional genomic hypothesis generation and experimentation by a robot scientist https://www.nature.com/articles/nature02236 https://www.nature.com/articles/nature02236

YeGoblynQueenne 5y ago

With the exception of counterfactual regret minimisation, which has nothing to do with deep learning, all those are applications of existing approaches that have become possible because of the increased expenditure of resources- processing power and data. There have been no theoretical advances, no algorithmic advances to speak of, no new knowledge learned in the last 20 years of reserach in deep learning. All that has been achieved is new SOTA on old benchmarks [1]. To make an analogy it's as if we could sort ever larger lists because we keep sorting them with ever bigger computers... but the only sorting algorithm we know is bubblesort. You bring up large language models. Language models are about as old as computational linguistics [6]. Large language models are... large. There's nothing that's new about them. "Attention" is an anthropomorphically misnamed engineering tweak of the kind that was rightly lambasted by Drew McDermot fourty years ago [7]. Self-play and reinforcement Learning are now more than 60 years old [8]. As to "few shot" learning, that is said for a language model previously trained on a copy of the entire web! In what sense is that "few shot"? It is truly shocking to hear researchers make such obviously false claims with a straight face. And that they are rewarded with publication of their work is nothing less than a scandal. Deep learning research is going nowhere. It is stuck in a rut. It is dead. It has given up the spirit. It is an ex-research field. This research field is a goner. It is dead, jim. And I'm sure it will be decades before people start to say it out loud (the current generation of reserachers will leave the dirty job to their students), but that doesn't change a thing. ______________ [1] Except in identifying new weaknesses of deep learning systems, as for example Overinterpretation in the paper I cite above, adversarial examples [2], shortcut learning [3], "the elephant in the room" [4], being "right for the wrong reasons" [5] etc etc etc etc. [2] https://arxiv.org/abs/1312.6199 https://arxiv.org/abs/1312.6199 [3] https://www.nature.com/articles/s42256-020-00257-z https://www.nature.com/articles/s42256-020-00257-z [4] https://arxiv.org/abs/1808.03305 https://arxiv.org/abs/1808.03305 [5] https://aclanthology.org/P19-1334.pdf https://aclanthology.org/P19-1334.pdf Which of course are very welcome. The only way to address the limitations of current approaches is to become aware of them and try to understand them. Unfortunately, it is only a tiny, tiny minority in the field who does that, while the majority is happy to spout nonsense like "[a billion plus] few shot" learning etc. [6] https://www.semanticscholar.org/paper/Speech-and-language-processing-an-introduction-to-Jurafsky-Martin/b54bcfca3fddc26b8889739a247a25e445818149 https://www.semanticscholar.org/paper/Speech-and-language-pr... [7] https://www.semanticscholar.org/paper/Artificial-intelligence-meets-natural-stupidity-McDermott/d9566ac89cd9b7fccc080b764aab5107430da28c https://www.semanticscholar.org/paper/Artificial-intelligenc... [8] Arhtur Samuel's checkers player beat a human champion in 1961. Donald Michie build MENACE, a reinforcement learning algorithm implemented in matchboxes in 1960: https://rodneybrooks.com/forai-machine-learning-explained/ https://rodneybrooks.com/forai-machine-learning-explained/ Ctrl+F for "Machine learning started with games" to find the relevant section.

Isinlor 5y ago
So much that you write is blatantly false, or stretch to the point of being as good as false. But you seem well read, so I don't think there is a point in trying to convince you. It's also interesting coming from someone researching Inductive Logic Programming that provides no results at all. From my perspective, I don't care for theoretical advances. I care about results and if scale provides results, I'm happy. For people who are just reading this exchange take a look here: https://pbs.twimg.com/media/FHHPjU0VIAUrSMA?format=jpg&name=900x900 https://pbs.twimg.com/media/FHHPjU0VIAUrSMA?format=jpg&name=... It's 16 computer generated images from the description below each image. Having capability like that is just very useful and fun. I wish you also luck in finding a "cat paying checkers in style of Salvador Dali" or a "psychedelic painting of a hamster dragon" on the internet. There is a lot of stuff on the web, but not everything.
- deleted 5y ago
  [deleted]
- deleted 5y ago
  [deleted]
- YeGoblynQueenne 5y ago
  >> From my perspective, I don't care for theoretical advances. I care about results and if scale provides results, I'm happy. Like I say you can sort ever larger lists with ever bigger computers and bubblesort. But you can go a lot further if you use the brain and come up with a better sorting algorithm. But I think you would be really surprised to find out how much humans have achieved by using their brains rather than ever bigger computers. Here's a small example that happened to be on my mind recently: https://en.wikipedia.org/wiki/Copernican_Revolution https://en.wikipedia.org/wiki/Copernican_Revolution That is an example of the difference between expensive toys like Dall-E and world-changing scientific work. Though I appreciate that the difference can sometimes appear somewhat mudddled and a machine that can generate a "hamster dragon" may seem as an important scientific breakthrough. Depends on what you're more interested in: hamster dragons, or understanding how the world works. Oh well. As to this: >> It's also interesting coming from someone researching Inductive Logic Programming that provides no results at all. That's an obvious -and odious- attempt at trolling me, but unlike yours, my HN profile is linked to my real-world identity and so I am forced to respond with politenes and to represent my field with dignity. Therefore, I will point out a famous achievement of ILP with which you are sadly unfamiliar and that I expect you to immediately discount on the grounds that you didn't know about it, so it can't have been all that important. I'm linking to an article in the popular press that will be more accessible. It doesn't go into depth over the machine learning approach used but it's ILP; I link to the scholarly article immediately after: Robot Makes Scientific Discovery All by Itself For the first time, a robotic system has made a novel scientific discovery with virtually no human intellectual input. Scientists designed “Adam” to carry out the entire scientific process on its own: formulating hypotheses, designing and running experiments, analyzing data, and deciding which experiments to run next. https://www.wired.com/2009/04/robotscientist/ https://www.wired.com/2009/04/robotscientist/ And the scholarly article: Functional genomic hypothesis generation and experimentation by a robot scientist https://www.nature.com/articles/nature02236 https://www.nature.com/articles/nature02236