5 ms·

"Numbers talk, bullshit walks." (L. Torvalds) Let's see the performance of ART on some benchmark tasks, then we can talk.

by kleiba 5y ago

YeGoblynQueenne 5y ago
I appreciate that you're saying something that you think is only common sense, but I suspect that is most likely because 99% of the machine learning papers you've seen do nothing else but claim a new SOTA result on some set of benchmarks. Yet just because that's what (almost) everyone is doing doesn't mean it's good research. Quite the contrary, that's the reason why machine learning research has become a quagmire and progress has stalled. Here's Geoff Hinton on benchmarks (again; I cite that all the time): GH: One big challenge the community faces is that if you want to get a paper published in machine learning now it's got to have a table in it, with all these different data sets across the top, and all these different methods along the side, and your method has to look like the best one. If it doesn’t look like that, it’s hard to get published. I don't think that's encouraging people to think about radically new ideas. Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad. What we should be going for, particularly in the basic science conferences, is radically new ideas. Because we know a radically new idea in the long run is going to be much more influential than a tiny improvement. That's I think the main downside of the fact that we've got this inversion now, where you've got a few senior guys and a gazillion young guys. https://www.wired.com/story/googles-ai-guru-computers-think-more-like-brains/ https://www.wired.com/story/googles-ai-guru-computers-think-... And here's a recent paper noting yet another way machine learning benchmarks are borked: https://arxiv.org/abs/2003.08907 https://arxiv.org/abs/2003.08907 P.S. And, please leave that aggressive attitude out of research discussions ("bullshit walks"?). Science is not the constant comparison of dick measurements. That's perhaps the point of sports, but anything that stifles debate and inhibits the development of new ideas is anti-science.
- Isinlor 5y ago
  How has the progress stalled? Just last year there was a breakthrough in decades old problem of protein-folding. The year before that we figured out that large language models have astonishing few shot learning abilities. CLIP introduced strong 0-shot capabilities to image recognition. Progress in image generation from descriptions has been nothing but astonishing with Dall-E and GLIDE from OpenAI. Not to mention breakthroughs in game theoretical problems like Counterfactual Regret Minimization and more recent application of Deep Learning to make CFR practical in actual games. Including continuous progress in the AlphaGo linage to AlphaZero, MuZero, and last year EfficientZero and Player of Games.
  - YeGoblynQueenne 5y ago
    With the exception of counterfactual regret minimisation, which has nothing to do with deep learning, all those are applications of existing approaches that have become possible because of the increased expenditure of resources- processing power and data. There have been no theoretical advances, no algorithmic advances to speak of, no new knowledge learned in the last 20 years of reserach in deep learning. All that has been achieved is new SOTA on old benchmarks [1]. To make an analogy it's as if we could sort ever larger lists because we keep sorting them with ever bigger computers... but the only sorting algorithm we know is bubblesort. You bring up large language models. Language models are about as old as computational linguistics [6]. Large language models are... large. There's nothing that's new about them. "Attention" is an anthropomorphically misnamed engineering tweak of the kind that was rightly lambasted by Drew McDermot fourty years ago [7]. Self-play and reinforcement Learning are now more than 60 years old [8]. As to "few shot" learning, that is said for a language model previously trained on a copy of the entire web! In what sense is that "few shot"? It is truly shocking to hear researchers make such obviously false claims with a straight face. And that they are rewarded with publication of their work is nothing less than a scandal. Deep learning research is going nowhere. It is stuck in a rut. It is dead. It has given up the spirit. It is an ex-research field. This research field is a goner. It is dead, jim. And I'm sure it will be decades before people start to say it out loud (the current generation of reserachers will leave the dirty job to their students), but that doesn't change a thing. ______________ [1] Except in identifying new weaknesses of deep learning systems, as for example Overinterpretation in the paper I cite above, adversarial examples [2], shortcut learning [3], "the elephant in the room" [4], being "right for the wrong reasons" [5] etc etc etc etc. [2] https://arxiv.org/abs/1312.6199 https://arxiv.org/abs/1312.6199 [3] https://www.nature.com/articles/s42256-020-00257-z https://www.nature.com/articles/s42256-020-00257-z [4] https://arxiv.org/abs/1808.03305 https://arxiv.org/abs/1808.03305 [5] https://aclanthology.org/P19-1334.pdf https://aclanthology.org/P19-1334.pdf Which of course are very welcome. The only way to address the limitations of current approaches is to become aware of them and try to understand them. Unfortunately, it is only a tiny, tiny minority in the field who does that, while the majority is happy to spout nonsense like "[a billion plus] few shot" learning etc. [6] https://www.semanticscholar.org/paper/Speech-and-language-processing-an-introduction-to-Jurafsky-Martin/b54bcfca3fddc26b8889739a247a25e445818149 https://www.semanticscholar.org/paper/Speech-and-language-pr... [7] https://www.semanticscholar.org/paper/Artificial-intelligence-meets-natural-stupidity-McDermott/d9566ac89cd9b7fccc080b764aab5107430da28c https://www.semanticscholar.org/paper/Artificial-intelligenc... [8] Arhtur Samuel's checkers player beat a human champion in 1961. Donald Michie build MENACE, a reinforcement learning algorithm implemented in matchboxes in 1960: https://rodneybrooks.com/forai-machine-learning-explained/ https://rodneybrooks.com/forai-machine-learning-explained/ Ctrl+F for "Machine learning started with games" to find the relevant section.
throway453sde 5y ago
That statement was in the context of implementing well known algorithms rather than researching unknown ideas. ART is totally a new approach in comparison to backward prop, hope it inspires some to continue research in this area. Deep Learning has so many use cases yet to be implemented that has business value but from the point of research its close to saturation. Time to explore new ideas.

"Numbers talk, bullshit walks." (L. Torvalds) Let's see the performance of ART on some benchmark tasks, then we can talk.

by kleiba 5y ago

"Numbers talk, bullshit walks." (L. Torvalds) Let's see the performance of ART on some benchmark tasks, then we can talk.

YeGoblynQueenne 5y ago

I appreciate that you're saying something that you think is only common sense, but I suspect that is most likely because 99% of the machine learning papers you've seen do nothing else but claim a new SOTA result on some set of benchmarks. Yet just because that's what (almost) everyone is doing doesn't mean it's good research. Quite the contrary, that's the reason why machine learning research has become a quagmire and progress has stalled. Here's Geoff Hinton on benchmarks (again; I cite that all the time): GH: One big challenge the community faces is that if you want to get a paper published in machine learning now it's got to have a table in it, with all these different data sets across the top, and all these different methods along the side, and your method has to look like the best one. If it doesn’t look like that, it’s hard to get published. I don't think that's encouraging people to think about radically new ideas. Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad. What we should be going for, particularly in the basic science conferences, is radically new ideas. Because we know a radically new idea in the long run is going to be much more influential than a tiny improvement. That's I think the main downside of the fact that we've got this inversion now, where you've got a few senior guys and a gazillion young guys. https://www.wired.com/story/googles-ai-guru-computers-think-more-like-brains/ https://www.wired.com/story/googles-ai-guru-computers-think-... And here's a recent paper noting yet another way machine learning benchmarks are borked: https://arxiv.org/abs/2003.08907 https://arxiv.org/abs/2003.08907 P.S. And, please leave that aggressive attitude out of research discussions ("bullshit walks"?). Science is not the constant comparison of dick measurements. That's perhaps the point of sports, but anything that stifles debate and inhibits the development of new ideas is anti-science.

Isinlor 5y ago
How has the progress stalled? Just last year there was a breakthrough in decades old problem of protein-folding. The year before that we figured out that large language models have astonishing few shot learning abilities. CLIP introduced strong 0-shot capabilities to image recognition. Progress in image generation from descriptions has been nothing but astonishing with Dall-E and GLIDE from OpenAI. Not to mention breakthroughs in game theoretical problems like Counterfactual Regret Minimization and more recent application of Deep Learning to make CFR practical in actual games. Including continuous progress in the AlphaGo linage to AlphaZero, MuZero, and last year EfficientZero and Player of Games.
- YeGoblynQueenne 5y ago
  With the exception of counterfactual regret minimisation, which has nothing to do with deep learning, all those are applications of existing approaches that have become possible because of the increased expenditure of resources- processing power and data. There have been no theoretical advances, no algorithmic advances to speak of, no new knowledge learned in the last 20 years of reserach in deep learning. All that has been achieved is new SOTA on old benchmarks [1]. To make an analogy it's as if we could sort ever larger lists because we keep sorting them with ever bigger computers... but the only sorting algorithm we know is bubblesort. You bring up large language models. Language models are about as old as computational linguistics [6]. Large language models are... large. There's nothing that's new about them. "Attention" is an anthropomorphically misnamed engineering tweak of the kind that was rightly lambasted by Drew McDermot fourty years ago [7]. Self-play and reinforcement Learning are now more than 60 years old [8]. As to "few shot" learning, that is said for a language model previously trained on a copy of the entire web! In what sense is that "few shot"? It is truly shocking to hear researchers make such obviously false claims with a straight face. And that they are rewarded with publication of their work is nothing less than a scandal. Deep learning research is going nowhere. It is stuck in a rut. It is dead. It has given up the spirit. It is an ex-research field. This research field is a goner. It is dead, jim. And I'm sure it will be decades before people start to say it out loud (the current generation of reserachers will leave the dirty job to their students), but that doesn't change a thing. ______________ [1] Except in identifying new weaknesses of deep learning systems, as for example Overinterpretation in the paper I cite above, adversarial examples [2], shortcut learning [3], "the elephant in the room" [4], being "right for the wrong reasons" [5] etc etc etc etc. [2] https://arxiv.org/abs/1312.6199 https://arxiv.org/abs/1312.6199 [3] https://www.nature.com/articles/s42256-020-00257-z https://www.nature.com/articles/s42256-020-00257-z [4] https://arxiv.org/abs/1808.03305 https://arxiv.org/abs/1808.03305 [5] https://aclanthology.org/P19-1334.pdf https://aclanthology.org/P19-1334.pdf Which of course are very welcome. The only way to address the limitations of current approaches is to become aware of them and try to understand them. Unfortunately, it is only a tiny, tiny minority in the field who does that, while the majority is happy to spout nonsense like "[a billion plus] few shot" learning etc. [6] https://www.semanticscholar.org/paper/Speech-and-language-processing-an-introduction-to-Jurafsky-Martin/b54bcfca3fddc26b8889739a247a25e445818149 https://www.semanticscholar.org/paper/Speech-and-language-pr... [7] https://www.semanticscholar.org/paper/Artificial-intelligence-meets-natural-stupidity-McDermott/d9566ac89cd9b7fccc080b764aab5107430da28c https://www.semanticscholar.org/paper/Artificial-intelligenc... [8] Arhtur Samuel's checkers player beat a human champion in 1961. Donald Michie build MENACE, a reinforcement learning algorithm implemented in matchboxes in 1960: https://rodneybrooks.com/forai-machine-learning-explained/ https://rodneybrooks.com/forai-machine-learning-explained/ Ctrl+F for "Machine learning started with games" to find the relevant section.

throway453sde 5y ago

That statement was in the context of implementing well known algorithms rather than researching unknown ideas. ART is totally a new approach in comparison to backward prop, hope it inspires some to continue research in this area. Deep Learning has so many use cases yet to be implemented that has business value but from the point of research its close to saturation. Time to explore new ideas.