5 ms·

I have seen far, far too many people say things along the lines of "Sure, LLMs currently don't seem to be good at [thing LLMs are, at least as of now, fundament

I have seen far, far too many people say things along the lines of "Sure, LLMs currently don't seem to be good at [thing LLMs are, at least as of now, fundamentally incapable of], but hey, some people are pretty bad at that sometimes too!" It demonstrates such a complete misunderstanding of the basic nature of the problem that I am left baffled that some of these people claim to actually be in the machine-learning field themselves. How can you not understand the difference between "humans are not absolutely perfect or reliable at this task" and "LLMs by their very nature cannot perform this task"? I do not know if AGI is possible. Honestly, I'd love to believe that it is. However, it has not remotely been demonstrated that it is possible, and as such, it follows that it cannot have been demonstrated that it is inevitable. If you want to believe that it is inevitable, then I have no quarrel with you; if you want to preach that it is inevitable, and draw specious inferences to "prove" it, then I have a big quarrel with you.

vundercind 2y ago
I think the fact that this particular fuzzy statistical analysis tool takes human language as input, and outputs more human language, is really dazzling some folks I’d not have expected to be dazzled by it. That is quickly becoming the most surprising part of this entire development, to me.
- jofla_net 2y ago
  At the very least, the last few years have laid bare some of the notions of what it takes, technically, to reconstruct certain chains of dialog, and how those chains are regarded completely differently as evidence for or against any and all intelligence it does or may take to conjure them.
- godelski 2y ago
  I'm astounded by them, still! But what is more astounding to me is all the reactions (even many in the "don't reason" camp, which I am part of). I'm an ML researcher and everyone was shocked when GPT3 came out. It is still impressive, and anyone saying it isn't is not being honest (likely to themselves). But it is amazing to me that "we compressed the entire internet and built a human language interface to access that information" is anything short of mindbogglingly impressive (and RAGs demonstrate how to decrease the lossyness of this compression). It would be complete Sci-Fi not even 10 years ago. I thought it was bad that we make them out to me much more than they are because when you bootstrap like that, you have to make that thing, and fast (e.g. iPhone). But "reasoning" is too big of a promise and we're too far from success. So I'm concerned as a researcher myself, because I like living in the summer. Because I want to work towards AGI. But if a promise is too big and the public realizes it, usually you don't just end up where you were. So it is the duty of any scientist and researcher to prevent their fields from being captured by people who overpromise. Not to "ruin the fun" but to instead make sure the party keeps going (sure, inviting a gorilla to the party may make it more exciting and "epic", but there's a good chance it also goes on a rampage and the party ends a lot sooner).
fidotron 2y ago
> How can you not understand the difference between "humans are not absolutely perfect or reliable at this task" and "LLMs by their very nature cannot perform this task"? This is a very good distillation of one side of it. What LLMs have taught us is a superficial grasp of language is good enough to reproduce a shocking proportion of what society has come to view as intelligent behaviors. i.e. it seems quite plausible a whole load of those people failing to grasp the point you are making are doing so because their internal models of the universe are closer to those of LLMs than you might want to think.
- godelski 2y ago
  I think we already knew this though. Because the Turing test was passed by Eliza in the 1960's. PARRY was even better and not even a decade later. For some reason people still talk about Chess performance as if Deep Blue didn't demonstrate this. Hell, here's even Feynman talking about many of the same things we're discussing today, but this was in the 80's https://www.youtube.com/watch?v=EKWGGDXe5MA https://www.youtube.com/watch?v=EKWGGDXe5MA
  - fidotron 2y ago
    Ten years ago I was explaining to halls of appalled academic administrators that AI would be replacing them before a robot succeeds in sorting out their socks.
    - eli_gottlieb 2y ago
      The field of AI needs to be constantly retaught the lesson that being able to replace important and powerful people doesn't mean your AI is actually intelligent. It means that important and powerful people were doing bullshit jobs.
  - matthewdgreen 2y ago
    ELIZA passed the Turing test in the same way a spooky halloween decoration can convince people that ghosts are real.
  - 2y ago
godelski 2y ago
> I have seen far, far too many people say It is perplexing. I've jokingly called it "proof of intelligence by (self) incompetence". I suspect that much of this is related to an overfitting of metrics within our own society. Such as leetcode or standardized exams. They're useful tools but only if you know what they actually measure and don't confuse the fact that they're a proxy. I also have a hard time convincing people about the duck argument in [0]. Oddly enough, I have far more difficulties having these discussions with computer scientists. It's what I'm doing my PhD in (ABD) but my undergrad was physics. After teaching a bit I think in part it is because in the hard sciences these differences get drilled into you when you do labs. Not always, but much more often. I see less of this type of conversation in CS and data science programs, where there is often a belief that there is a well defined and precise answer (always seemed odd to me since there's many ways you can write the same algorithm).
SpicyLemonZest 2y ago
> How can you not understand the difference between "humans are not absolutely perfect or reliable at this task" and "LLMs by their very nature cannot perform this task"? I understand the difference, and sometimes that second statement really is true. But a rigorous proof that problem X can't be reduced to architecture Y is generally very hard to construct, and most people making these claims don't have one. I've talked to more than a few people who insist that an LLM can't have a world model, or a concept of truth, or any other abstract reasoning capability that isn't a native component of its architecture.
- danaris 2y ago
  And I'm much less frustrated by people who are, in fact, claiming that LLMs can do these things, whether or not I agree with them. Frankly, while I have a basic understanding of the underlying technology, I'm not in the ML field myself, and can't claim to be enough of an expert to say with any real authority what an LLM could ever be able to do, just what the particular LLMs I've used or seen the detailed explanations of can do. No; this is specifically about people who stipulate that the LLMs can't do these things, but still want to claim that they are or will become AGI, so they just basically say "well, humans can't really do it, can they? so LLMs don't need to do it either!"
  - godelski 2y ago
    I am an ML researcher, I don't think LLMs can reason, but similar to you I'm annoyed by people who say ML systems "will never" reason. This is a strong claim that needs be substantiated too! Just as the strong claim of LLMs reasoning needs strong evidence (which I've yet to see). It's subtle, but that matters and subtle things is why expertise is often required for many things. We don't have a proof of universal approximation in a meaningful sense with transformers (yes, I'm aware of that paper). Fwiw, I'm never frustrated by people having opinions. We're human, we all do. But I'm deeply frustrated with how common it is to watch people with no expertise argue with those that do. It's one thing for LeCun to argue with Hinton, but it's another when Musk or some random anime profile picture person does. And it's weird that people take strong sides on discussions happening in the open. Opinions, totally fine. So are discussions. But it's when people assert correctness that it turns to look religious. And there's many that over inflate the knowledge that they have. So what I'm saying is please keep this attitude. Skepticism and pushback are not problematic, they are tools that can be valuable to learn. The things you're skeptical about are good to be skeptical about. As much as I hate the AGI hype I'm also upset by the over correction many of my peers take. Neither is scientific.
fragmede 2y ago
> "LLMs by their very nature cannot perform this task" The issue is that it's not LLMs that can't perform a given task, but that computers already can. Counting the number of Rs in strawberry or comparing 9.11 to 9.7 is trivial for a regular computer program, but hard for an LLM due to the tokenization process. Where LLMs are a pile of matrixes and some math and some look up tables, it's easy to see that as the essential nature of LLMs, which is to say theres no thinking or reasoning happening because it's just a pile of math happening and it's just glorified auto-complete. Artificial things look a lot like the thing they resemble, but they also are artificial, and as such, are markedly different from the thing they resemble. is the very nature of an LLMs being a pile of math mean that it can not perform said task if given more math and more compute and more data? given enough compute, can we change that nature? I make no prognostication as to whether or not AGI will come from transformers, and this is getting very philosophical, but I see it as irrelevant because I don't believe that AGI is the right measure.
famouswaffles 2y ago
>How can you not understand the difference between "humans are not absolutely perfect or reliable at this task" and "LLMs by their very nature cannot perform this task"? Because anyone who has said nonsense like "LLMs by their very nature cannot do x" and waited a few years has been wrong. That's why GPT-3 and 4 shocked the research world in the first place. People just have their pre-conceptions about how they think LLMs should work and what their "very nature" should preclude and are so very confident about it. People like that will say things like "LLM are always hallucinating. It doesn't know the difference between truth and fiction!" and feel like they've just said something profound about the "nature" of LLMs, all while being entirely wrong (no need to wait, plenty of different research to trash this particular take). It's just very funny seeing people who were/would be gob smacked a few years ago talking about the "very nature" of LLMs. If you understood this nature so well, why didn't you all tell us about what it would be able to do years ago ? ML is an alchemical science. The builders themselves don't understand the "very nature" of anything they're building, nevermind anyone else.
- riku_iki 2y ago
  > Because anyone who has said nonsense like "LLMs by their very nature cannot do x" and waited a few years has been wrong. That's why GPT-3 and 4 shocked the research world in the first place. there are some benchmarks which show fundamental inability of LLM perform certain tasks which human can, for example add 100 digits numbers.
  - deleted 2y ago
    [deleted]
  - famouswaffles 2y ago
    > there are some benchmarks which show fundamental inability of LLM perform certain tasks which human can, for example add 100 digits numbers. fundamental inability ? No. Current Sota LLM (4o, claude, gemini) woes with arithmetic is not a transformer weakness never mind a large language modelling one. Those benchmarks show that those particular models have problems with accuracy on that many digits, not that LLMs fundamentally cannot be accurate on that many digits. https://arxiv.org/abs/2405.17399 https://arxiv.org/abs/2405.17399 https://arxiv.org/abs/2307.03381 https://arxiv.org/abs/2307.03381 https://arxiv.org/abs/2310.02989 https://arxiv.org/abs/2310.02989 Ultimately, numbers are represented in GPT like systems in a pretty weird way. That way affects how well they can learn to do things like arithmetic and counting and the like but that way isn't a necessary way. You don't have to represent numbers like that.

vundercind 2y ago

I think the fact that this particular fuzzy statistical analysis tool takes human language as input, and outputs more human language, is really dazzling some folks I’d not have expected to be dazzled by it. That is quickly becoming the most surprising part of this entire development, to me.

jofla_net 2y ago
At the very least, the last few years have laid bare some of the notions of what it takes, technically, to reconstruct certain chains of dialog, and how those chains are regarded completely differently as evidence for or against any and all intelligence it does or may take to conjure them.
godelski 2y ago
I'm astounded by them, still! But what is more astounding to me is all the reactions (even many in the "don't reason" camp, which I am part of). I'm an ML researcher and everyone was shocked when GPT3 came out. It is still impressive, and anyone saying it isn't is not being honest (likely to themselves). But it is amazing to me that "we compressed the entire internet and built a human language interface to access that information" is anything short of mindbogglingly impressive (and RAGs demonstrate how to decrease the lossyness of this compression). It would be complete Sci-Fi not even 10 years ago. I thought it was bad that we make them out to me much more than they are because when you bootstrap like that, you have to make that thing, and fast (e.g. iPhone). But "reasoning" is too big of a promise and we're too far from success. So I'm concerned as a researcher myself, because I like living in the summer. Because I want to work towards AGI. But if a promise is too big and the public realizes it, usually you don't just end up where you were. So it is the duty of any scientist and researcher to prevent their fields from being captured by people who overpromise. Not to "ruin the fun" but to instead make sure the party keeps going (sure, inviting a gorilla to the party may make it more exciting and "epic", but there's a good chance it also goes on a rampage and the party ends a lot sooner).

fidotron 2y ago

> How can you not understand the difference between "humans are not absolutely perfect or reliable at this task" and "LLMs by their very nature cannot perform this task"? This is a very good distillation of one side of it. What LLMs have taught us is a superficial grasp of language is good enough to reproduce a shocking proportion of what society has come to view as intelligent behaviors. i.e. it seems quite plausible a whole load of those people failing to grasp the point you are making are doing so because their internal models of the universe are closer to those of LLMs than you might want to think.

godelski 2y ago
I think we already knew this though. Because the Turing test was passed by Eliza in the 1960's. PARRY was even better and not even a decade later. For some reason people still talk about Chess performance as if Deep Blue didn't demonstrate this. Hell, here's even Feynman talking about many of the same things we're discussing today, but this was in the 80's https://www.youtube.com/watch?v=EKWGGDXe5MA https://www.youtube.com/watch?v=EKWGGDXe5MA
- fidotron 2y ago
  Ten years ago I was explaining to halls of appalled academic administrators that AI would be replacing them before a robot succeeds in sorting out their socks.
  - eli_gottlieb 2y ago
    The field of AI needs to be constantly retaught the lesson that being able to replace important and powerful people doesn't mean your AI is actually intelligent. It means that important and powerful people were doing bullshit jobs.
- matthewdgreen 2y ago
  ELIZA passed the Turing test in the same way a spooky halloween decoration can convince people that ghosts are real.
- 2y ago

godelski 2y ago

> I have seen far, far too many people say It is perplexing. I've jokingly called it "proof of intelligence by (self) incompetence". I suspect that much of this is related to an overfitting of metrics within our own society. Such as leetcode or standardized exams. They're useful tools but only if you know what they actually measure and don't confuse the fact that they're a proxy. I also have a hard time convincing people about the duck argument in [0]. Oddly enough, I have far more difficulties having these discussions with computer scientists. It's what I'm doing my PhD in (ABD) but my undergrad was physics. After teaching a bit I think in part it is because in the hard sciences these differences get drilled into you when you do labs. Not always, but much more often. I see less of this type of conversation in CS and data science programs, where there is often a belief that there is a well defined and precise answer (always seemed odd to me since there's many ways you can write the same algorithm).

SpicyLemonZest 2y ago

> How can you not understand the difference between "humans are not absolutely perfect or reliable at this task" and "LLMs by their very nature cannot perform this task"? I understand the difference, and sometimes that second statement really is true. But a rigorous proof that problem X can't be reduced to architecture Y is generally very hard to construct, and most people making these claims don't have one. I've talked to more than a few people who insist that an LLM can't have a world model, or a concept of truth, or any other abstract reasoning capability that isn't a native component of its architecture.

danaris 2y ago
And I'm much less frustrated by people who are, in fact, claiming that LLMs can do these things, whether or not I agree with them. Frankly, while I have a basic understanding of the underlying technology, I'm not in the ML field myself, and can't claim to be enough of an expert to say with any real authority what an LLM could ever be able to do, just what the particular LLMs I've used or seen the detailed explanations of can do. No; this is specifically about people who stipulate that the LLMs can't do these things, but still want to claim that they are or will become AGI, so they just basically say "well, humans can't really do it, can they? so LLMs don't need to do it either!"
- godelski 2y ago
  I am an ML researcher, I don't think LLMs can reason, but similar to you I'm annoyed by people who say ML systems "will never" reason. This is a strong claim that needs be substantiated too! Just as the strong claim of LLMs reasoning needs strong evidence (which I've yet to see). It's subtle, but that matters and subtle things is why expertise is often required for many things. We don't have a proof of universal approximation in a meaningful sense with transformers (yes, I'm aware of that paper). Fwiw, I'm never frustrated by people having opinions. We're human, we all do. But I'm deeply frustrated with how common it is to watch people with no expertise argue with those that do. It's one thing for LeCun to argue with Hinton, but it's another when Musk or some random anime profile picture person does. And it's weird that people take strong sides on discussions happening in the open. Opinions, totally fine. So are discussions. But it's when people assert correctness that it turns to look religious. And there's many that over inflate the knowledge that they have. So what I'm saying is please keep this attitude. Skepticism and pushback are not problematic, they are tools that can be valuable to learn. The things you're skeptical about are good to be skeptical about. As much as I hate the AGI hype I'm also upset by the over correction many of my peers take. Neither is scientific.

fragmede 2y ago

> "LLMs by their very nature cannot perform this task" The issue is that it's not LLMs that can't perform a given task, but that computers already can. Counting the number of Rs in strawberry or comparing 9.11 to 9.7 is trivial for a regular computer program, but hard for an LLM due to the tokenization process. Where LLMs are a pile of matrixes and some math and some look up tables, it's easy to see that as the essential nature of LLMs, which is to say theres no thinking or reasoning happening because it's just a pile of math happening and it's just glorified auto-complete. Artificial things look a lot like the thing they resemble, but they also are artificial, and as such, are markedly different from the thing they resemble. is the very nature of an LLMs being a pile of math mean that it can not perform said task if given more math and more compute and more data? given enough compute, can we change that nature? I make no prognostication as to whether or not AGI will come from transformers, and this is getting very philosophical, but I see it as irrelevant because I don't believe that AGI is the right measure.

famouswaffles 2y ago

>How can you not understand the difference between "humans are not absolutely perfect or reliable at this task" and "LLMs by their very nature cannot perform this task"? Because anyone who has said nonsense like "LLMs by their very nature cannot do x" and waited a few years has been wrong. That's why GPT-3 and 4 shocked the research world in the first place. People just have their pre-conceptions about how they think LLMs should work and what their "very nature" should preclude and are so very confident about it. People like that will say things like "LLM are always hallucinating. It doesn't know the difference between truth and fiction!" and feel like they've just said something profound about the "nature" of LLMs, all while being entirely wrong (no need to wait, plenty of different research to trash this particular take). It's just very funny seeing people who were/would be gob smacked a few years ago talking about the "very nature" of LLMs. If you understood this nature so well, why didn't you all tell us about what it would be able to do years ago ? ML is an alchemical science. The builders themselves don't understand the "very nature" of anything they're building, nevermind anyone else.

riku_iki 2y ago
> Because anyone who has said nonsense like "LLMs by their very nature cannot do x" and waited a few years has been wrong. That's why GPT-3 and 4 shocked the research world in the first place. there are some benchmarks which show fundamental inability of LLM perform certain tasks which human can, for example add 100 digits numbers.
- deleted 2y ago
  [deleted]
- famouswaffles 2y ago
  > there are some benchmarks which show fundamental inability of LLM perform certain tasks which human can, for example add 100 digits numbers. fundamental inability ? No. Current Sota LLM (4o, claude, gemini) woes with arithmetic is not a transformer weakness never mind a large language modelling one. Those benchmarks show that those particular models have problems with accuracy on that many digits, not that LLMs fundamentally cannot be accurate on that many digits. https://arxiv.org/abs/2405.17399 https://arxiv.org/abs/2405.17399 https://arxiv.org/abs/2307.03381 https://arxiv.org/abs/2307.03381 https://arxiv.org/abs/2310.02989 https://arxiv.org/abs/2310.02989 Ultimately, numbers are represented in GPT like systems in a pretty weird way. That way affects how well they can learn to do things like arithmetic and counting and the like but that way isn't a necessary way. You don't have to represent numbers like that.