3 ms·

>I am surprised by how dismissive the whole post sounds. I wouldn't be surprised, it's Gary Marcus. He's an academic with a lot of prestige to lose if the LLM

>I am surprised by how dismissive the whole post sounds. I wouldn't be surprised, it's Gary Marcus. He's an academic with a lot of prestige to lose if the LLM approach is actually good/useful/insightful, who's only widely publicly known now because AI has had a backlash and media needed an expert to quote for "the other side". Same as the computational linguistics researchers who always get quoted for the same reason. In general, academics in competing fields whose funding threatens to get tanked or eclipsed by research approaches that work on principles they have fundamental academic disagreements with are going to talk negatively about the tech, no matter what it is achieving. Where I think it can be valuable to listen to them is when they're giving the technology credit - generally they'll only do that when it's something really undeniable or potentially concerning.

Ukv 3y ago
> it's Gary Marcus. He's an academic with a lot of prestige to lose if the LLM approach is actually good/useful/insightful I wouldn't bet on loss of perceived expert on the other side status or prestige no matter how poorly his gloomy LLM forecasts fare. It hasn't happened from any of his previous consistently pessimistic predictions around deep learning since the beginning of its resurgence (which has clearly far surpassed its state in 2012, even if currently overhyped). To quote Gary Marcus quoting Gary Marcus: > Yet deep learning may well be approaching a wall, much as I anticipated earlier, at beginning of the resurgence (Marcus, 2012)
tomhoward 3y ago
This is such a case of a bad-comment that seems clever and insightful. It boils down to saying we don't need to debate or even consider the content of his arguments because we can assume he's only motivated by prestige and money (but without considering the second-order effects on his credibility and funding if he actually turns out to be proven substantially wrong in the future). I don't know how right or wrong he is - none of us do. That's why it's all still being debated. The one thing I know is that we can only truly understand a topic by fully understanding arguments for and against all the claims. I also know the pro-LLM set have way more money (double-digit billions as we saw just this week) and credibility to lose over this topic than Gary Marcus does.
- lucubratory 3y ago
  I completely understand your comment, it's reasonable to want sources. I didn't have enough time when I made my original comment to do some good sourcing, but I had that time now. In the interest of not making this comment absurdly long and not repeating work that's already been done, I'll try to be brief and link to secondary sources that contain accurate analysis where possible, rather than linking to primary sources with mostly my own analysis. >but without considering the second-order effects on his credibility and funding if he actually turns out to be proven substantially wrong in the future He has continuously made predictions about how bad deep learning is and how it would never achieve much, unlike his "much better" neurosymbolic theories that have never shown results anywhere near as significant (there might be good ideas there, doesn't mean that his continuous goalpost shifting is good science). Those predictions have continuously been proven wrong for the last decade, including a number of predictions specifically about LLMs. This is relatively normal in the academy, people have hobby horses, but he's now engaging in public discourse and advocacy about an actually contentious topic and there are different standards here, so I'm going to respond in that context. If you're not familiar with him at all, that's fine, here's some of his track record and explanations of why he's wrong: For deep learning in general, in case you want an overview of where he's coming from (he stands by everything in this article today, don't let the age inspire sympathy, he's a deep learning skeptic to the core): https://www.newyorker.com/news/news-desk/is-deep-learning-a-revolution-in-artificial-intelligence https://www.newyorker.com/news/news-desk/is-deep-learning-a-... For GPT-2 (a moderately important retrospective is in the comments): https://www.lesswrong.com/posts/ZFtesgbY9XwtqqyZ5/human-psycholinguists-a-critical-appraisal https://www.lesswrong.com/posts/ZFtesgbY9XwtqqyZ5/human-psyc... For GPT-3: https://www.tumblr.com/nostalgebraist/628024664310136832/gary-marcus-has-co-authored-a-brief-critique-of https://www.tumblr.com/nostalgebraist/628024664310136832/gar... CTRL-F "Gary Marcus" on this page to find a bunch of sourced, specific claims from Marcus alongside evidence they are wrong: https://gwern.net/gpt-3-nonfiction https://gwern.net/gpt-3-nonfiction He has never stopped, and probably never will. He's the student of the original guy who first buried neural nets back in the sixties or seventies; he's royalty in symbolic AI. The only concession I've ever seen him make to the continued progress of deep learning and LLMs in spite of his continuous early proclamations of death is, bizzarely, that he is concerned about existential risk from current AI research. I don't personally see how he squares "Deep learning and LLMs are all nonsense, have hit a wall, off-ramp on the road to AGI" with "Current AI research [which is overwhelmingly deep learning, even if he doesn't say that part out loud] presents an existential risk", but he does it somehow. Basically, no matter what new AI research or artifacts come out, if they are made with neural nets he will explain: 1. They are not that important 2. It's a dead end 3. They're harmful because people might wrongly get the impression that they work 4. They're nowhere near as good as his favoured AI architectures are, if only his favoured theories had as much funding as that broken deep learning crap

Ukv 3y ago

> it's Gary Marcus. He's an academic with a lot of prestige to lose if the LLM approach is actually good/useful/insightful I wouldn't bet on loss of perceived expert on the other side status or prestige no matter how poorly his gloomy LLM forecasts fare. It hasn't happened from any of his previous consistently pessimistic predictions around deep learning since the beginning of its resurgence (which has clearly far surpassed its state in 2012, even if currently overhyped). To quote Gary Marcus quoting Gary Marcus: > Yet deep learning may well be approaching a wall, much as I anticipated earlier, at beginning of the resurgence (Marcus, 2012)

tomhoward 3y ago

This is such a case of a bad-comment that seems clever and insightful. It boils down to saying we don't need to debate or even consider the content of his arguments because we can assume he's only motivated by prestige and money (but without considering the second-order effects on his credibility and funding if he actually turns out to be proven substantially wrong in the future). I don't know how right or wrong he is - none of us do. That's why it's all still being debated. The one thing I know is that we can only truly understand a topic by fully understanding arguments for and against all the claims. I also know the pro-LLM set have way more money (double-digit billions as we saw just this week) and credibility to lose over this topic than Gary Marcus does.

lucubratory 3y ago
I completely understand your comment, it's reasonable to want sources. I didn't have enough time when I made my original comment to do some good sourcing, but I had that time now. In the interest of not making this comment absurdly long and not repeating work that's already been done, I'll try to be brief and link to secondary sources that contain accurate analysis where possible, rather than linking to primary sources with mostly my own analysis. >but without considering the second-order effects on his credibility and funding if he actually turns out to be proven substantially wrong in the future He has continuously made predictions about how bad deep learning is and how it would never achieve much, unlike his "much better" neurosymbolic theories that have never shown results anywhere near as significant (there might be good ideas there, doesn't mean that his continuous goalpost shifting is good science). Those predictions have continuously been proven wrong for the last decade, including a number of predictions specifically about LLMs. This is relatively normal in the academy, people have hobby horses, but he's now engaging in public discourse and advocacy about an actually contentious topic and there are different standards here, so I'm going to respond in that context. If you're not familiar with him at all, that's fine, here's some of his track record and explanations of why he's wrong: For deep learning in general, in case you want an overview of where he's coming from (he stands by everything in this article today, don't let the age inspire sympathy, he's a deep learning skeptic to the core): https://www.newyorker.com/news/news-desk/is-deep-learning-a-revolution-in-artificial-intelligence https://www.newyorker.com/news/news-desk/is-deep-learning-a-... For GPT-2 (a moderately important retrospective is in the comments): https://www.lesswrong.com/posts/ZFtesgbY9XwtqqyZ5/human-psycholinguists-a-critical-appraisal https://www.lesswrong.com/posts/ZFtesgbY9XwtqqyZ5/human-psyc... For GPT-3: https://www.tumblr.com/nostalgebraist/628024664310136832/gary-marcus-has-co-authored-a-brief-critique-of https://www.tumblr.com/nostalgebraist/628024664310136832/gar... CTRL-F "Gary Marcus" on this page to find a bunch of sourced, specific claims from Marcus alongside evidence they are wrong: https://gwern.net/gpt-3-nonfiction https://gwern.net/gpt-3-nonfiction He has never stopped, and probably never will. He's the student of the original guy who first buried neural nets back in the sixties or seventies; he's royalty in symbolic AI. The only concession I've ever seen him make to the continued progress of deep learning and LLMs in spite of his continuous early proclamations of death is, bizzarely, that he is concerned about existential risk from current AI research. I don't personally see how he squares "Deep learning and LLMs are all nonsense, have hit a wall, off-ramp on the road to AGI" with "Current AI research [which is overwhelmingly deep learning, even if he doesn't say that part out loud] presents an existential risk", but he does it somehow. Basically, no matter what new AI research or artifacts come out, if they are made with neural nets he will explain: 1. They are not that important 2. It's a dead end 3. They're harmful because people might wrongly get the impression that they work 4. They're nowhere near as good as his favoured AI architectures are, if only his favoured theories had as much funding as that broken deep learning crap