3 ms·

Your first paragraph is an ad-hominem that you should be ashamed of. "Not really taken seriously by the community"? Which community is that? Last time I saw Mar

Your first paragraph is an ad-hominem that you should be ashamed of. "Not really taken seriously by the community"? Which community is that? Last time I saw Marcus he was a co-organizer of a panel on Dug Lenat and Knowledge Representation in AAAI 2024 [1]. Does that sound like the AI community is not taking him seriously? Is anyone asking you to orgranise panels in AAAI? The question is rhetorical: I can see you're a PhD student. My advice is to learn to have some respect for the person of others, as you will want them to have respect of your person when, one day, you find yourself saying something that "the community" disagrees with. And if you're not planning to, one day, find yourself in that situation, consider the possibility that you're in the wrong job. And what is the above article saying that you think should not be taken seriously? Is it not a widely recognised fact that neural nets performance only improves with more data, more compute and more parameters? A five year old could tell you that. Is it controversial that this is a limitation? _____________________ [1] https://aaai.org/aaai-conference/aaai-24-panels/ https://aaai.org/aaai-conference/aaai-24-panels/

albertzeyer 3y ago
I did not intend to attack Gary or so in any way. But I realize that my statement is probably too strong. Of course it's not the whole AI community. My intention with my post was just to give some perspective, some background for people who have not heard about Gary Marcus before. Maybe I'm also in a bubble, but I was speaking mostly about the people I frequently read from, i.e. lots of people from Google Brain, DeepMind, other people who frequently publish on NeurIPS, ICLR, ICML, etc. Among those people, Gary is usually not taken seriously. At least that was my impression. But let's not make this so much about Gary: Most of these people disagree with the opinion that Gary shares, i.e. they don't really see such a big need for symbolic AI, or they see much more potential in pure neural approaches (after all, the human brain is fully neural).
- YeGoblynQueenne 3y ago
  >> Of course it's not the whole AI community. My intention with my post was just to give some perspective, some background for people who have not heard about Gary Marcus before. Yes, I get it. And the perspective you wanted to give was to not take Marcus seriously because the people you follow on social media say he's not to be taken seriously. That's nothing but a form of collective online bullying that attacks the person and not the opinion, and like I say in my other comment above, shameful. Consider for a moment the impression that you make when you say that some people you know, when they're not publishing on NeurIPS, are on social media dogpiling on someone who criticises their work. That's not researchers any more, but common social media trolls. >> But let's not make this so much about Gary: Most of these people disagree with the opinion that Gary shares, i.e. they don't really see such a big need for symbolic AI, or they see much more potential in pure neural approaches (after all, the human brain is fully neural). To my experience, the majority of neural net researchers don't know anything concrete about symbolic AI, just what they have heard second-hand, usually on social media again, usually by people who disagree with Marcus, who's the most famous proponent of neuro-symbolic AI (NeSy). So whatever opinion they have on NeSy is not an informed opinion. There's plenty of literature on NeSy which is a bona fide field of research with a conference etc. This year Leslie Valiant was the keynote speaker and Yan LeCun the honoured guest: https://sites.google.com/view/nesy2023 https://sites.google.com/view/nesy2023 You really don't have to listen to what Marcus says to form an opinion on NeSy. Btw, I am not with them and I think they're going the wrong way, but at least I know what they're doing. That is much less than can be said about most neural net researchers, who rarely know anything outside their own work besides whatever preprint is trending on X. That's to the detriment of nobody but themselves.
  - oefrha 3y ago
    > Consider for a moment the impression that you make when you say that some people you know, when they're not publishing on NeurIPS, are on social media dogpiling on someone who criticises their work. That's not researchers any more, but common social media trolls. Twitter for the past decade plus has really publicized and amplified these petty, close-minded academic cliques. It's pretty disgusting to watch for someone who was also a PhD student eyeing an academic career once.
    - YeGoblynQueenne 3y ago
      Yes, this kind of behaviour is keeping people away from research, except for the ones who are fine with it which of course ends up encouraging the bad behaviour. I'm sorry to see the disappointment evident in your comment.
HarHarVeryFunny 3y ago
> Is it not a widely recognised fact that neural nets performance only improves with more data, more compute and more parameters? I'm not sure how you meant that to be parsed. 1) performance only improves by scaling up those factors, and can't be improved in any other way OR 2) performance can only (can't help but) get better as you scale up I'm guessing you meant 1), which is wrong, but just in case you meant 2), that is wrong too. Increased scaling - in the absence of other changes - will continue to improve performance until it doesn't, and no-one knows what the limit is. As far as 1), nobody thinks that scaling up is the only way to improve the performance of these systems. Architectural advances, such as the one that created them in the first place, is the other way to improve. There have already been many architectural changes since the original transformer of 2017, and I'm sure we'll see more in the models released later this year. You ask if it's controversial that there is a limit to how much training data is available, or how much compute can be used to train them. For training data, the informed consensus appears to be that this will not be a limiting factor; in the words of Anthropic's CEO Dario Amodei "It'd be nice [from safety perspective] if it [data availability] was [a limit], but it won't be". Synthetic data is all the rage, and these companies can generate as much as they need. There's also masses of human-generated audio/video data that has hardly been touched. Sure, compute would eventually become a limiting factor if scaling were the only way these models were being improved (which it isn't), but there is still plenty of headroom at the moment. As long as each generation of models make meaningful advances towards AGI, then I expect the money will be found. It'd be very surprising if the technology was advancing rapidly but development curtailed by lack of money - this is ultimately a national security issue and the government could choose to fund it if they had to.
HarHarVeryFunny 3y ago
> Last time I saw Marcus he was a co-organizer of a panel on Doug Lenat and Knowledge Representation in AAAI 2024 Next year he'll be teaming up with Rudy Giuliani to tout the success of SHRLDU at Four Seasons Landscaping. The AI community asked GPT-4 to send him an invite, and he accepted.

albertzeyer 3y ago

I did not intend to attack Gary or so in any way. But I realize that my statement is probably too strong. Of course it's not the whole AI community. My intention with my post was just to give some perspective, some background for people who have not heard about Gary Marcus before. Maybe I'm also in a bubble, but I was speaking mostly about the people I frequently read from, i.e. lots of people from Google Brain, DeepMind, other people who frequently publish on NeurIPS, ICLR, ICML, etc. Among those people, Gary is usually not taken seriously. At least that was my impression. But let's not make this so much about Gary: Most of these people disagree with the opinion that Gary shares, i.e. they don't really see such a big need for symbolic AI, or they see much more potential in pure neural approaches (after all, the human brain is fully neural).

YeGoblynQueenne 3y ago
>> Of course it's not the whole AI community. My intention with my post was just to give some perspective, some background for people who have not heard about Gary Marcus before. Yes, I get it. And the perspective you wanted to give was to not take Marcus seriously because the people you follow on social media say he's not to be taken seriously. That's nothing but a form of collective online bullying that attacks the person and not the opinion, and like I say in my other comment above, shameful. Consider for a moment the impression that you make when you say that some people you know, when they're not publishing on NeurIPS, are on social media dogpiling on someone who criticises their work. That's not researchers any more, but common social media trolls. >> But let's not make this so much about Gary: Most of these people disagree with the opinion that Gary shares, i.e. they don't really see such a big need for symbolic AI, or they see much more potential in pure neural approaches (after all, the human brain is fully neural). To my experience, the majority of neural net researchers don't know anything concrete about symbolic AI, just what they have heard second-hand, usually on social media again, usually by people who disagree with Marcus, who's the most famous proponent of neuro-symbolic AI (NeSy). So whatever opinion they have on NeSy is not an informed opinion. There's plenty of literature on NeSy which is a bona fide field of research with a conference etc. This year Leslie Valiant was the keynote speaker and Yan LeCun the honoured guest: https://sites.google.com/view/nesy2023 https://sites.google.com/view/nesy2023 You really don't have to listen to what Marcus says to form an opinion on NeSy. Btw, I am not with them and I think they're going the wrong way, but at least I know what they're doing. That is much less than can be said about most neural net researchers, who rarely know anything outside their own work besides whatever preprint is trending on X. That's to the detriment of nobody but themselves.
- oefrha 3y ago
  > Consider for a moment the impression that you make when you say that some people you know, when they're not publishing on NeurIPS, are on social media dogpiling on someone who criticises their work. That's not researchers any more, but common social media trolls. Twitter for the past decade plus has really publicized and amplified these petty, close-minded academic cliques. It's pretty disgusting to watch for someone who was also a PhD student eyeing an academic career once.
  - YeGoblynQueenne 3y ago
    Yes, this kind of behaviour is keeping people away from research, except for the ones who are fine with it which of course ends up encouraging the bad behaviour. I'm sorry to see the disappointment evident in your comment.

HarHarVeryFunny 3y ago

> Is it not a widely recognised fact that neural nets performance only improves with more data, more compute and more parameters? I'm not sure how you meant that to be parsed. 1) performance only improves by scaling up those factors, and can't be improved in any other way OR 2) performance can only (can't help but) get better as you scale up I'm guessing you meant 1), which is wrong, but just in case you meant 2), that is wrong too. Increased scaling - in the absence of other changes - will continue to improve performance until it doesn't, and no-one knows what the limit is. As far as 1), nobody thinks that scaling up is the only way to improve the performance of these systems. Architectural advances, such as the one that created them in the first place, is the other way to improve. There have already been many architectural changes since the original transformer of 2017, and I'm sure we'll see more in the models released later this year. You ask if it's controversial that there is a limit to how much training data is available, or how much compute can be used to train them. For training data, the informed consensus appears to be that this will not be a limiting factor; in the words of Anthropic's CEO Dario Amodei "It'd be nice [from safety perspective] if it [data availability] was [a limit], but it won't be". Synthetic data is all the rage, and these companies can generate as much as they need. There's also masses of human-generated audio/video data that has hardly been touched. Sure, compute would eventually become a limiting factor if scaling were the only way these models were being improved (which it isn't), but there is still plenty of headroom at the moment. As long as each generation of models make meaningful advances towards AGI, then I expect the money will be found. It'd be very surprising if the technology was advancing rapidly but development curtailed by lack of money - this is ultimately a national security issue and the government could choose to fund it if they had to.

HarHarVeryFunny 3y ago

> Last time I saw Marcus he was a co-organizer of a panel on Doug Lenat and Knowledge Representation in AAAI 2024 Next year he'll be teaming up with Rudy Giuliani to tout the success of SHRLDU at Four Seasons Landscaping. The AI community asked GPT-4 to send him an invite, and he accepted.