4 ms·

> How can you not understand the difference between "humans are not absolutely perfect or reliable at this task" and "LLMs by their very nature cannot perform t

> How can you not understand the difference between "humans are not absolutely perfect or reliable at this task" and "LLMs by their very nature cannot perform this task"? I understand the difference, and sometimes that second statement really is true. But a rigorous proof that problem X can't be reduced to architecture Y is generally very hard to construct, and most people making these claims don't have one. I've talked to more than a few people who insist that an LLM can't have a world model, or a concept of truth, or any other abstract reasoning capability that isn't a native component of its architecture.

danaris 2y ago
And I'm much less frustrated by people who are, in fact, claiming that LLMs can do these things, whether or not I agree with them. Frankly, while I have a basic understanding of the underlying technology, I'm not in the ML field myself, and can't claim to be enough of an expert to say with any real authority what an LLM could ever be able to do, just what the particular LLMs I've used or seen the detailed explanations of can do. No; this is specifically about people who stipulate that the LLMs can't do these things, but still want to claim that they are or will become AGI, so they just basically say "well, humans can't really do it, can they? so LLMs don't need to do it either!"
- godelski 2y ago
  I am an ML researcher, I don't think LLMs can reason, but similar to you I'm annoyed by people who say ML systems "will never" reason. This is a strong claim that needs be substantiated too! Just as the strong claim of LLMs reasoning needs strong evidence (which I've yet to see). It's subtle, but that matters and subtle things is why expertise is often required for many things. We don't have a proof of universal approximation in a meaningful sense with transformers (yes, I'm aware of that paper). Fwiw, I'm never frustrated by people having opinions. We're human, we all do. But I'm deeply frustrated with how common it is to watch people with no expertise argue with those that do. It's one thing for LeCun to argue with Hinton, but it's another when Musk or some random anime profile picture person does. And it's weird that people take strong sides on discussions happening in the open. Opinions, totally fine. So are discussions. But it's when people assert correctness that it turns to look religious. And there's many that over inflate the knowledge that they have. So what I'm saying is please keep this attitude. Skepticism and pushback are not problematic, they are tools that can be valuable to learn. The things you're skeptical about are good to be skeptical about. As much as I hate the AGI hype I'm also upset by the over correction many of my peers take. Neither is scientific.
godelski 2y ago
> But a rigorous proof that problem X can't be reduced to architecture Y is generally very hard to construct, and most people making these claims don't have one. Requirement for proof is backwards. It's the ones that claim that thing reasons that needs proof. They've provided evidence (albeit shakey), but evidence isn't proof. So your reasoning is a bit off base (albeit understandable and logical) since evidence contrary to the claim isn't proof either. But the burden of proof isn't on the one countering the claim, it's on the one making the claim. I need to make this extra clear because framing can make the direction of burden confusing. So using an obvious example: if I claim there's ghosts in my house (something millions of people believe and similarly claim) we generally do not dismiss someone who is skeptical of these claims and offers an alternative explanation (even when it isn't perfectly precise). Because the burden of proof is on the person making the stronger claim. Sure, there are people that will dismiss that too, but they want to believe in ghosts. So the question is if we want to believe in ghosts in the (shell) machine. It's very easy to be fooled, so we must keep our guard up. And we also shouldn't feel embarrassed when we've been tricked. It happens to everyone. Anyone that claims they've never been fooled is only telling you that they are skillful at fooling themselves. I for one did buy into AGI being close when GPT 3 came out. Most researchers I knew did too! But as we learned more about what was actually going on under the hood I think many of us changed our minds (just as we changed our minds after seeing GPT). Being able to change your mind is a good thing.

danaris 2y ago

And I'm much less frustrated by people who are, in fact, claiming that LLMs can do these things, whether or not I agree with them. Frankly, while I have a basic understanding of the underlying technology, I'm not in the ML field myself, and can't claim to be enough of an expert to say with any real authority what an LLM could ever be able to do, just what the particular LLMs I've used or seen the detailed explanations of can do. No; this is specifically about people who stipulate that the LLMs can't do these things, but still want to claim that they are or will become AGI, so they just basically say "well, humans can't really do it, can they? so LLMs don't need to do it either!"

godelski 2y ago
I am an ML researcher, I don't think LLMs can reason, but similar to you I'm annoyed by people who say ML systems "will never" reason. This is a strong claim that needs be substantiated too! Just as the strong claim of LLMs reasoning needs strong evidence (which I've yet to see). It's subtle, but that matters and subtle things is why expertise is often required for many things. We don't have a proof of universal approximation in a meaningful sense with transformers (yes, I'm aware of that paper). Fwiw, I'm never frustrated by people having opinions. We're human, we all do. But I'm deeply frustrated with how common it is to watch people with no expertise argue with those that do. It's one thing for LeCun to argue with Hinton, but it's another when Musk or some random anime profile picture person does. And it's weird that people take strong sides on discussions happening in the open. Opinions, totally fine. So are discussions. But it's when people assert correctness that it turns to look religious. And there's many that over inflate the knowledge that they have. So what I'm saying is please keep this attitude. Skepticism and pushback are not problematic, they are tools that can be valuable to learn. The things you're skeptical about are good to be skeptical about. As much as I hate the AGI hype I'm also upset by the over correction many of my peers take. Neither is scientific.

godelski 2y ago

> But a rigorous proof that problem X can't be reduced to architecture Y is generally very hard to construct, and most people making these claims don't have one. Requirement for proof is backwards. It's the ones that claim that thing reasons that needs proof. They've provided evidence (albeit shakey), but evidence isn't proof. So your reasoning is a bit off base (albeit understandable and logical) since evidence contrary to the claim isn't proof either. But the burden of proof isn't on the one countering the claim, it's on the one making the claim. I need to make this extra clear because framing can make the direction of burden confusing. So using an obvious example: if I claim there's ghosts in my house (something millions of people believe and similarly claim) we generally do not dismiss someone who is skeptical of these claims and offers an alternative explanation (even when it isn't perfectly precise). Because the burden of proof is on the person making the stronger claim. Sure, there are people that will dismiss that too, but they want to believe in ghosts. So the question is if we want to believe in ghosts in the (shell) machine. It's very easy to be fooled, so we must keep our guard up. And we also shouldn't feel embarrassed when we've been tricked. It happens to everyone. Anyone that claims they've never been fooled is only telling you that they are skillful at fooling themselves. I for one did buy into AGI being close when GPT 3 came out. Most researchers I knew did too! But as we learned more about what was actually going on under the hood I think many of us changed our minds (just as we changed our minds after seeing GPT). Being able to change your mind is a good thing.