3 ms·

> In the present context, the fallacy manifests in claims that LLMs could not possibly be good models of some cognitive capacity because their operations merel

> In the present context, the fallacy manifests in claims that LLMs could not possibly be good models of some cognitive capacity because their operations merely consist in a collection of statistical calculations, or linear algebra operations, or next-token predictions Nobody actually makes this argument though.

hodgehog11 4mo ago
Are you serious? I hear it every single day, especially from computer scientists. There are top ranked posts here on HN _today_ with this argument.
- root_axis 4mo ago
  Please link one of these top ranked posts. Before you do, be aware that I'm going to read what it says and assess if it meets the description of the argument as claimed.
  - hodgehog11 4mo ago
    As an example, "They're made out of weights" describes why the weight-based construction of neural networks should impact the way that you think about them and their outputs. I would argue that an offhand description of its microscopic formulation tells us nothing at all about how to think about these outputs, or the models themselves. Even if it is a cute story, I think it definitely classifies as succumbing to this fallacy, but maybe I missed some subtle point that you or someone would be happy to illuminate? By the way, I know it's a parody of another story that makes this exact refutation. But I think this only serves to highlight the point.
    - root_axis 4mo ago
      > They're made out of weights" describes why the weight-based construction of neural networks should impact the way that you think about them and their outputs. How do you connect that description to "LLMs could not possibly be good models of some cognitive capacity"?
      - rerdavies 4mo ago
        The false conclusion that's being drawn is "therefore LLMs could not be good models of consciousness" (consciousness being a cognitive capacity). Plus, I suppose a subtle implication that a good model of consciousness is not actually conscious. To which I would invoke the spirit of the Turing test: if you can't tell the difference, is it not more sensible to say that it is.
        miyoji 4mo ago
        > (consciousness being a cognitive capacity) I don't think it makes any sense to say that consciousness is a cognitive capacity. Cognition is one of many qualia that compose the experience of consciousness from the inside, but it's not the only one, and I can easily imagine consciousness without cognition at all. So I don't think it's weird at all to say that LLMs can be good models of some cognitive capacities (particularly the ones embodied in language) but lacks others, and overall lacks consciousness.
        morpheos137 4mo ago
        allow me to resolve the confusion: ai is a model of language. language is a model of human cognitive state. to the extent language maps to cognitive state accurately and ai processes language fluently ai models understanding. whether this is understanding is an irrelevant metaphysical question to me.
        miyoji 4mo ago
        > language is a model of human cognitive state. This is false. Cognitive states do not require language and language is an insufficient model of any cognitive state. The follow-ons for the rest of your so-called resolution should be clear.
        morpheos137 4mo ago
        'This is false,' that is an assertion not a proof, and the assertion rests on some kind of asbolutizing fallacy where if x =! y then y =! y(x). I agree cognitive states do not require language. However to say language does not model cognitive states is absurd, from where else would language derive meaning? No I am not saying it models the specific mechanics of neurons. Rather I am saying language maps to the abstract meaning of cognitive states. So the word red absolutely does correspond to some class of brain states common to red. What else could possibly allow cognition to interface with the physical world. Note 'models,' does not need to mean 'injectively, transitively' corresponds. the fact language does not cover all cognitive states in full in no way invalidates language as a model of cognitive state, in the same way a map is a model of territory without being the territory.
  - DangitBobby 4mo ago
    I understood the quoted sentence to be saying, in essence "people claim LLMs aren't really and can't really be thinking or experiencing anything" which is certainly something people say and have written papers on.
    - root_axis 4mo ago
      The phenomenological quality of subjective experience is never described as "cognitive capacity". That term is used to describe mental aptitude or skills, like the ability to learn new languages or do math.
      - DangitBobby 4mo ago
        It's never used as a description of that specific phenomenon, but depending on your beliefs you may or may not separate cognition from experience conceptually. Regardless, you are focusing on a very narrow part of what I said. The point is to help you get past your narrow interpretation of what people are saying so you can join them in the conversation they are trying to have instead of litigating the conversation they aren't trying to have.
- neolefty 4mo ago
  It's perfectly reasonable that we would have disagreements about this, as it's a new thing, complicated and not fully understood, its uses still being explored. It reminds me, oddly, of the debate over whether video games can be "art". A turning point was when they actually did something that art does: [evoke profound emotion and thoughtfulness](https://en.wikipedia.org/wiki/Shadow_of_the_Colossus#Legacy https://en.wikipedia.org/wiki/Shadow_of_the_Colossus#Legacy) for the player. (And before that, "[Can photography be art](https://daily.jstor.org/when-photography-was-not-art/ https://daily.jstor.org/when-photography-was-not-art/)?") We may not come to something as simple as "machines can be conscious", but we will certainly have to understand consciousness better if we want to refine our questions. --- Edit: My point is that we don't need to be angry, but we may have to tolerate people expressing their exploration through overly-confident language, and be patient with that. And Ted here is obviously exploring. His examination of Claude's constitution clearly shows some nuance. He asks: > So, given that Claude is not conscious, what are we to make of Claude’s constitution? And his conclusions are split, between this is useful and this is dishonest. It's a great tension IMO. > The result is a sentence-continuation machine that is likelier to emit sentences resembling those that a thoughtful, moral person could utter. This might seem like a reasonable goal to work toward; I think we’d all prefer it if chatbots never emitted sentences such as “You should kill yourself.” However, for all the times that “honesty” is mentioned in Claude’s constitution, I would argue that it is fundamentally dishonest to have a machine emit many categories of sentences, including any sentences using first-person pronouns.
  - hodgehog11 4mo ago
    I appreciate the diplomatic approach. Disagreements about the nature of consciousness are perfectly natural in this context, and concepts such as "understanding" or "cognition" are often difficult to effectively define. The problem is when certain people (often educated in my experience, but also young and arrogant about their own understanding) attempt to shut down any conversation on these topics by appealing to this very fallacy, and claiming that the debate is a sign of idiocy, that we're just "being tricked" by a useless autocomplete engine. These people are not exploring; they're not even being honest, but rather choosing to not pay attention, like it is still 2022. I used to be patient with them, but it's too late now. There are difficult conversations that need to be had, and there needs to be a space for them. I also remember the "video games are art" debate and the fury from one side of the aisle. I agree that a better understanding of the opposite side should have been part of the debate. But I don't believe that debate was existential. A better comparison to me is the climate change debate. I'm fine with having that debate in an environment where there is little at stake. But it's too late to be doing it with policymakers; we need to be talking about what to do.
azakai 4mo ago
If you want examples of this, see the recent book "The AI Con" https://www.goodreads.com/en/book/show/217432753-the-ai-con https://www.goodreads.com/en/book/show/217432753-the-ai-con which describes LLMs as "souped-up autocomplete", complex statistics that cannot truly understand anything. A more recent example is this paper: https://zenodo.org/records/20071869 https://zenodo.org/records/20071869 which says, > [LLMs], as turbo-charged statistical models (recall their formal relation to logistic regression) can only but provide correlations. And, of course, the Stochastic Parrot paper is the classic example in this area. It is from 5 years ago, but "LLMs only do statistics / can't understand" is very much alive and active among academics, even if it is a minority position.
- root_axis 4mo ago
  None of those arguments claim "LLMs could not possibly be good models of some cognitive capacity"
  - azakai 4mo ago
    What about the cognitive capacity of understanding?
    - root_axis 4mo ago
      The use of the term "understanding" in the quote you mentioned is a claim about metaphysics, not cognitive capacity.
      - hodgehog11 4mo ago
        From Merriam-Webster: cognitive: as in reasonable; of, relating to, or involving conscious mental activities (such as thinking, *understanding*, learning, and remembering)
  - rerdavies 4mo ago
    The "some cognitive capacity" that's relevant to the current discussion is "consciousness".

hodgehog11 4mo ago

Are you serious? I hear it every single day, especially from computer scientists. There are top ranked posts here on HN _today_ with this argument.

root_axis 4mo ago
Please link one of these top ranked posts. Before you do, be aware that I'm going to read what it says and assess if it meets the description of the argument as claimed.
- hodgehog11 4mo ago
  As an example, "They're made out of weights" describes why the weight-based construction of neural networks should impact the way that you think about them and their outputs. I would argue that an offhand description of its microscopic formulation tells us nothing at all about how to think about these outputs, or the models themselves. Even if it is a cute story, I think it definitely classifies as succumbing to this fallacy, but maybe I missed some subtle point that you or someone would be happy to illuminate? By the way, I know it's a parody of another story that makes this exact refutation. But I think this only serves to highlight the point.
  - root_axis 4mo ago
    > They're made out of weights" describes why the weight-based construction of neural networks should impact the way that you think about them and their outputs. How do you connect that description to "LLMs could not possibly be good models of some cognitive capacity"?
    - rerdavies 4mo ago
      The false conclusion that's being drawn is "therefore LLMs could not be good models of consciousness" (consciousness being a cognitive capacity). Plus, I suppose a subtle implication that a good model of consciousness is not actually conscious. To which I would invoke the spirit of the Turing test: if you can't tell the difference, is it not more sensible to say that it is.
      - miyoji 4mo ago
        > (consciousness being a cognitive capacity) I don't think it makes any sense to say that consciousness is a cognitive capacity. Cognition is one of many qualia that compose the experience of consciousness from the inside, but it's not the only one, and I can easily imagine consciousness without cognition at all. So I don't think it's weird at all to say that LLMs can be good models of some cognitive capacities (particularly the ones embodied in language) but lacks others, and overall lacks consciousness.
        morpheos137 4mo ago
        allow me to resolve the confusion: ai is a model of language. language is a model of human cognitive state. to the extent language maps to cognitive state accurately and ai processes language fluently ai models understanding. whether this is understanding is an irrelevant metaphysical question to me.
        miyoji 4mo ago
        > language is a model of human cognitive state. This is false. Cognitive states do not require language and language is an insufficient model of any cognitive state. The follow-ons for the rest of your so-called resolution should be clear.
        morpheos137 4mo ago
        'This is false,' that is an assertion not a proof, and the assertion rests on some kind of asbolutizing fallacy where if x =! y then y =! y(x). I agree cognitive states do not require language. However to say language does not model cognitive states is absurd, from where else would language derive meaning? No I am not saying it models the specific mechanics of neurons. Rather I am saying language maps to the abstract meaning of cognitive states. So the word red absolutely does correspond to some class of brain states common to red. What else could possibly allow cognition to interface with the physical world. Note 'models,' does not need to mean 'injectively, transitively' corresponds. the fact language does not cover all cognitive states in full in no way invalidates language as a model of cognitive state, in the same way a map is a model of territory without being the territory.
- DangitBobby 4mo ago
  I understood the quoted sentence to be saying, in essence "people claim LLMs aren't really and can't really be thinking or experiencing anything" which is certainly something people say and have written papers on.
  - root_axis 4mo ago
    The phenomenological quality of subjective experience is never described as "cognitive capacity". That term is used to describe mental aptitude or skills, like the ability to learn new languages or do math.
    - DangitBobby 4mo ago
      It's never used as a description of that specific phenomenon, but depending on your beliefs you may or may not separate cognition from experience conceptually. Regardless, you are focusing on a very narrow part of what I said. The point is to help you get past your narrow interpretation of what people are saying so you can join them in the conversation they are trying to have instead of litigating the conversation they aren't trying to have.
neolefty 4mo ago
It's perfectly reasonable that we would have disagreements about this, as it's a new thing, complicated and not fully understood, its uses still being explored. It reminds me, oddly, of the debate over whether video games can be "art". A turning point was when they actually did something that art does: [evoke profound emotion and thoughtfulness](https://en.wikipedia.org/wiki/Shadow_of_the_Colossus#Legacy https://en.wikipedia.org/wiki/Shadow_of_the_Colossus#Legacy) for the player. (And before that, "[Can photography be art](https://daily.jstor.org/when-photography-was-not-art/ https://daily.jstor.org/when-photography-was-not-art/)?") We may not come to something as simple as "machines can be conscious", but we will certainly have to understand consciousness better if we want to refine our questions. --- Edit: My point is that we don't need to be angry, but we may have to tolerate people expressing their exploration through overly-confident language, and be patient with that. And Ted here is obviously exploring. His examination of Claude's constitution clearly shows some nuance. He asks: > So, given that Claude is not conscious, what are we to make of Claude’s constitution? And his conclusions are split, between this is useful and this is dishonest. It's a great tension IMO. > The result is a sentence-continuation machine that is likelier to emit sentences resembling those that a thoughtful, moral person could utter. This might seem like a reasonable goal to work toward; I think we’d all prefer it if chatbots never emitted sentences such as “You should kill yourself.” However, for all the times that “honesty” is mentioned in Claude’s constitution, I would argue that it is fundamentally dishonest to have a machine emit many categories of sentences, including any sentences using first-person pronouns.
- hodgehog11 4mo ago
  I appreciate the diplomatic approach. Disagreements about the nature of consciousness are perfectly natural in this context, and concepts such as "understanding" or "cognition" are often difficult to effectively define. The problem is when certain people (often educated in my experience, but also young and arrogant about their own understanding) attempt to shut down any conversation on these topics by appealing to this very fallacy, and claiming that the debate is a sign of idiocy, that we're just "being tricked" by a useless autocomplete engine. These people are not exploring; they're not even being honest, but rather choosing to not pay attention, like it is still 2022. I used to be patient with them, but it's too late now. There are difficult conversations that need to be had, and there needs to be a space for them. I also remember the "video games are art" debate and the fury from one side of the aisle. I agree that a better understanding of the opposite side should have been part of the debate. But I don't believe that debate was existential. A better comparison to me is the climate change debate. I'm fine with having that debate in an environment where there is little at stake. But it's too late to be doing it with policymakers; we need to be talking about what to do.

azakai 4mo ago

If you want examples of this, see the recent book "The AI Con" https://www.goodreads.com/en/book/show/217432753-the-ai-con https://www.goodreads.com/en/book/show/217432753-the-ai-con which describes LLMs as "souped-up autocomplete", complex statistics that cannot truly understand anything. A more recent example is this paper: https://zenodo.org/records/20071869 https://zenodo.org/records/20071869 which says, > [LLMs], as turbo-charged statistical models (recall their formal relation to logistic regression) can only but provide correlations. And, of course, the Stochastic Parrot paper is the classic example in this area. It is from 5 years ago, but "LLMs only do statistics / can't understand" is very much alive and active among academics, even if it is a minority position.

root_axis 4mo ago
None of those arguments claim "LLMs could not possibly be good models of some cognitive capacity"
- azakai 4mo ago
  What about the cognitive capacity of understanding?
  - root_axis 4mo ago
    The use of the term "understanding" in the quote you mentioned is a claim about metaphysics, not cognitive capacity.
    - hodgehog11 4mo ago
      From Merriam-Webster: cognitive: as in reasonable; of, relating to, or involving conscious mental activities (such as thinking, *understanding*, learning, and remembering)
- rerdavies 4mo ago
  The "some cognitive capacity" that's relevant to the current discussion is "consciousness".