6 ms·

The Abstraction Fallacy: Why AI Can Simulate but Not Instantiate Consciousness

32 points by LopRabbit 6mo ago

mstank 6mo ago
Glad to see Searle's Chinese Room mentioned early on in the paper. "Syntax is not sufficient for semantics," no matter how much compute we throw at the problem. My very amateur view is that until the underlying compute architecture and substrate resembles artificial biology more than silicon, we wont get there. The latest advances in AI have given me even more appreciation of biology and evolution. It's incredible what the human brain can do with about 20 watts of power, barely enough to power a lightbulb, in comparison to what it takes to run even our most basic LLM models.
- Kim_Bruning 6mo ago
  Hofstadter and Dennett have taken great pains to try to debunk Searle. No love lost in that corner of the philosophical world.
yogthos 6mo ago
The paper makes a huge assumption that only thermodynamic constitutions can produce consciousness. The assumptions seems completely unsubstantiated given that thermodynamics are just states and states are replicable. The whole Chinese Room idea is pure sophistry as well. Both Dennett and Hofstadter address it quite well in Consciousness Explained and I am a Strange Loop respectively.
- emp17344 6mo ago
  You know that Dennett and Hofstadter aren’t the beginning and end of Philosophy of Mind, right? Calling Searle’s Room “complete sophistry” is hilariously misguided, considering the vast majority of academic philosophers consider it valid: https://survey2020.philpeople.org/survey/results/5002# https://survey2020.philpeople.org/survey/results/5002#
  - Kim_Bruning 6mo ago
    You'll need to unpack that survey for us a bit. There's a lot going on and the wording is very terse.
    - emp17344 6mo ago
      It’s a large survey of academic philosophers on famous philosophical arguments. In this case, the question is asking whether philosophers agree with Searle and believe the Chinese room does not understand Chinese, or disagree with Searle and believe the room does understand Chinese.
      - Kim_Bruning 6mo ago
        I actually agree that the room does not understand chinese too; because that's the only possible thing that could happen in real life. That doesn't mean I agree with Searle though! It depends on how the question is asked. Again, the wording is very terse so I can't determine what the people thought they were answering. Possibly you have a better insight?
        emp17344 6mo ago
        I think you’re possibly a bit confused… accepting Searle’s intuition on this thought experiment is agreeing with Searle. In light of this, I don’t understand your comment.
        Kim_Bruning 6mo ago
        The intuition pump surely works just fine as an intuition pump. But from a purely scientific view: if you were to try to stage it as an actual scientific experiment it fails to hold up. No control (there's just the one room), no single or double blind (the researcher self-reports) , and badly defined elements (the contents of the notebook are not specified). Of course you reach the conclusion that room doesn't understand. Compare to Turings imitation game experiment: Two participants (principal and control if you will), Double blind (they're in closed rooms so the judges can't see them), and you can have multiple people doing the scoring. We can conduct this IRL, and in fact if you've used IRC or discord, it's almost a natural experiment there. Looking at it as a scientific experiment isn't so strange, Searle is responding to an executable experiment with an intuition pump. Why should the latter win? Note that there are many ways to cut this, but this is one of mine.
diablozzq 6mo ago
Consciousness is a property of humans biology - and quite clearly not a requisite to intelligence. I say clearly as at some point we reach proof by construction. As in, we already built intelligence because the system already completes tasks that require intelligence. We are so far into what would have been science fiction five years ago and the goal posts have moved so far. For anyone who disagrees, I challenge you to prove deep learning systems cannot solve <task with specific outcome humans can solve but not AI> given sufficient data and compute. I think the strongest sign we have true intelligence already is no one has built any benchmark that AI cannot solve. Yes, our current robotics lags AI, so we don’t have the equivalent of the human body to give our deep learning systems. Thus, it’s expected AI will be limited in physical scenarios. Second, hallucinations are present in humans. We are highly biased to ignore all the misspoken words in everyday life as we have error correction built into normal conversations. How often do you have to have someone repeat or rephrase something? It just doesn’t make sense to me. It’s like there are people out there whose belief systems are incompatible with this tech existing. Sure, it has limitations due to training data. It has limitations with no physical body. It cannot combine training and inference the same way a human does. But none of those are measures of intelligence or required to be intelligent.
- lukev 6mo ago
  "intelligence" is not well defined. LLMs are throwing this into high relief with how "spiky" their capability curve is. Yes, they can solve some crazy hard problems with enough compute and thinking tokens. Yes, they also fall down in the dumbest ways without an ability to self-correct... despite how "smart" they are, human supervision remains absolutely critical for any system of importance. But I don't think the takeaway is "humans are intelligent and LLMs are not", it's that our vocabulary for talking about the intersection of language, cognition and compute is not up for the task.
  - diablozzq 6mo ago
    Intelligence was supposedly well defined, but folks kept getting their definitions wrecked by modern LLMs so we had to move the goal posts. No true Scottsman fallacy.
jdmoreira 6mo ago
This is the complete opposite of Hofstadter's "Strange Loop" hypothesis, which intuitively makes much more sense to me.
- defterGoose 6mo ago
  It's the pervasive theme in the book, but never really given a conceptual grounding further than "this sort of looks like recursion or can be modelled circularly so it's a strange loop". The vagueness of it reveals itself as being "more intuitive", because a vaguer pattern will have more matches. I don't remember Hofstadter digressing on whether these loops work "in reverse" either, which is sort of what the author here is denying. Basically positing that f doesn't have a well-defined inverse.
Kim_Bruning 6mo ago
I'm partial to bioinformatics as per Pauline Hogeweg's definition; which explicitly has computation as a property of life. This approach actually makes testable (and tested) scientific predictions. This makes Searle-derived papers super-weird for me; since from my perspective they seem to disprove the existence of life. (and it makes the name of the philosophy "biological naturalism" very ironic to me :-P ) (for extra irony, Turing actually went into biology late in his life. See: Turing 1952 "The Chemical Basis of Morphogenesis" )
- kbelder 6mo ago
  I'm disappointed that Searle's paper is still influential, at least out in the general culture. It's nonsense, and at face value, would disprove consciousness in humans unless you accept some mystic indefinable soul into the mix. Or quantum magic, which is just as mystic.
in-silico 6mo ago
From my observations, there are generally four camps in the machine consciousness discussion: 1. People who haven't really thought about it, and assume they're conscious because they talk like a human. 2. People who haven't really thought about it, and assume they can't be conscious because humans are obviously somehow special. This appears to be the largest group, and is linked to our religiously rooted culture in which human exceptionalism is the default. Those first two groups comprise the majority of people, and are not worth engaging with. 3. People who have thought about it, and came to the conclusion that they might be conscious, usually for computationalism/functionalism reasons. This is the group that I place myself in. 4. People who have thought about it, and came to the conclusion that they can't be conscious, usually for biological naturalist reasons. This seems to be the predominant group on Hacker News (among those who discuss it).
- sunrunner 6mo ago
  I'm not sure I'd agree that people in groups 1 and 2 aren't worth engaging with. The interesting bit to do for both cases is look at the 'they talk like a human' and 'are obviously somehow special' parts, separate the ideas of language, intelligence (memory, fluidity, abstract reasoning), _aliveness_ (as a biological process) and finally ideas about metacognition and theory of mind, and see whether their idea of consciousness as a super-bundle of the above (which is how I assume a lot of default ideas about consciousness are) actually sticks, or whether it falls apart when beings can have a subset of those properties but not all. Also, I nominate myself to be in the 'People who have thought about it and are becoming more doubtful that I myself am conscious, and the question might be moot.' group.
  - in-silico 6mo ago
    I'm curious about your doubting your own consciousness statement, given that "we humans are conscious" is pretty axiomic to its definition and one of the few pieces that most agree with.
    - Kim_Bruning 6mo ago
      Take a look at Daniel Dennett, for starters! If you're looking for one of the genuine angles on this: Consciousness is horrendously under-defined, to the point some people go something like "you know, at this point I figure we'd be better off not having this word at all. " Some days that's me, with a headache.
tmvphil 6mo ago
> To fully understand the difference between the embodied robot running an algorithm on a chip and the biological mapmaker, we need to remember that for the latter, subjective experience is a given, not because of abstract information processing, but because of a specific, metabolically constituted physical reality. Total drivel. Consciousness in biological systems is "a given" because of metabolism?
jwpapi 6mo ago
I think the question goes more into ourselves as it goes into AI, we don’t know exactly how our own intelligence and conciousness works and therefore it’s very tough to impossible to compare to AI intelligence and conciousness. Are we just autocomplete machines with sufficient enough variable pseudo-randomized input?
grantcas 6mo ago
[dead]

32 points by LopRabbit 6mo ago

Glad to see Searle's Chinese Room mentioned early on in the paper. "Syntax is not sufficient for semantics," no matter how much compute we throw at the problem. My very amateur view is that until the underlying compute architecture and substrate resembles artificial biology more than silicon, we wont get there. The latest advances in AI have given me even more appreciation of biology and evolution. It's incredible what the human brain can do with about 20 watts of power, barely enough to power a lightbulb, in comparison to what it takes to run even our most basic LLM models.

Kim_Bruning 6mo ago
Hofstadter and Dennett have taken great pains to try to debunk Searle. No love lost in that corner of the philosophical world.

yogthos 6mo ago

The paper makes a huge assumption that only thermodynamic constitutions can produce consciousness. The assumptions seems completely unsubstantiated given that thermodynamics are just states and states are replicable. The whole Chinese Room idea is pure sophistry as well. Both Dennett and Hofstadter address it quite well in Consciousness Explained and I am a Strange Loop respectively.

emp17344 6mo ago
You know that Dennett and Hofstadter aren’t the beginning and end of Philosophy of Mind, right? Calling Searle’s Room “complete sophistry” is hilariously misguided, considering the vast majority of academic philosophers consider it valid: https://survey2020.philpeople.org/survey/results/5002# https://survey2020.philpeople.org/survey/results/5002#
- Kim_Bruning 6mo ago
  You'll need to unpack that survey for us a bit. There's a lot going on and the wording is very terse.
  - emp17344 6mo ago
    It’s a large survey of academic philosophers on famous philosophical arguments. In this case, the question is asking whether philosophers agree with Searle and believe the Chinese room does not understand Chinese, or disagree with Searle and believe the room does understand Chinese.
    - Kim_Bruning 6mo ago
      I actually agree that the room does not understand chinese too; because that's the only possible thing that could happen in real life. That doesn't mean I agree with Searle though! It depends on how the question is asked. Again, the wording is very terse so I can't determine what the people thought they were answering. Possibly you have a better insight?
      - emp17344 6mo ago
        I think you’re possibly a bit confused… accepting Searle’s intuition on this thought experiment is agreeing with Searle. In light of this, I don’t understand your comment.
        Kim_Bruning 6mo ago
        The intuition pump surely works just fine as an intuition pump. But from a purely scientific view: if you were to try to stage it as an actual scientific experiment it fails to hold up. No control (there's just the one room), no single or double blind (the researcher self-reports) , and badly defined elements (the contents of the notebook are not specified). Of course you reach the conclusion that room doesn't understand. Compare to Turings imitation game experiment: Two participants (principal and control if you will), Double blind (they're in closed rooms so the judges can't see them), and you can have multiple people doing the scoring. We can conduct this IRL, and in fact if you've used IRC or discord, it's almost a natural experiment there. Looking at it as a scientific experiment isn't so strange, Searle is responding to an executable experiment with an intuition pump. Why should the latter win? Note that there are many ways to cut this, but this is one of mine.

diablozzq 6mo ago

Consciousness is a property of humans biology - and quite clearly not a requisite to intelligence. I say clearly as at some point we reach proof by construction. As in, we already built intelligence because the system already completes tasks that require intelligence. We are so far into what would have been science fiction five years ago and the goal posts have moved so far. For anyone who disagrees, I challenge you to prove deep learning systems cannot solve <task with specific outcome humans can solve but not AI> given sufficient data and compute. I think the strongest sign we have true intelligence already is no one has built any benchmark that AI cannot solve. Yes, our current robotics lags AI, so we don’t have the equivalent of the human body to give our deep learning systems. Thus, it’s expected AI will be limited in physical scenarios. Second, hallucinations are present in humans. We are highly biased to ignore all the misspoken words in everyday life as we have error correction built into normal conversations. How often do you have to have someone repeat or rephrase something? It just doesn’t make sense to me. It’s like there are people out there whose belief systems are incompatible with this tech existing. Sure, it has limitations due to training data. It has limitations with no physical body. It cannot combine training and inference the same way a human does. But none of those are measures of intelligence or required to be intelligent.

lukev 6mo ago
"intelligence" is not well defined. LLMs are throwing this into high relief with how "spiky" their capability curve is. Yes, they can solve some crazy hard problems with enough compute and thinking tokens. Yes, they also fall down in the dumbest ways without an ability to self-correct... despite how "smart" they are, human supervision remains absolutely critical for any system of importance. But I don't think the takeaway is "humans are intelligent and LLMs are not", it's that our vocabulary for talking about the intersection of language, cognition and compute is not up for the task.
- diablozzq 6mo ago
  Intelligence was supposedly well defined, but folks kept getting their definitions wrecked by modern LLMs so we had to move the goal posts. No true Scottsman fallacy.

jdmoreira 6mo ago

This is the complete opposite of Hofstadter's "Strange Loop" hypothesis, which intuitively makes much more sense to me.

defterGoose 6mo ago
It's the pervasive theme in the book, but never really given a conceptual grounding further than "this sort of looks like recursion or can be modelled circularly so it's a strange loop". The vagueness of it reveals itself as being "more intuitive", because a vaguer pattern will have more matches. I don't remember Hofstadter digressing on whether these loops work "in reverse" either, which is sort of what the author here is denying. Basically positing that f doesn't have a well-defined inverse.

Kim_Bruning 6mo ago

I'm partial to bioinformatics as per Pauline Hogeweg's definition; which explicitly has computation as a property of life. This approach actually makes testable (and tested) scientific predictions. This makes Searle-derived papers super-weird for me; since from my perspective they seem to disprove the existence of life. (and it makes the name of the philosophy "biological naturalism" very ironic to me :-P ) (for extra irony, Turing actually went into biology late in his life. See: Turing 1952 "The Chemical Basis of Morphogenesis" )

kbelder 6mo ago
I'm disappointed that Searle's paper is still influential, at least out in the general culture. It's nonsense, and at face value, would disprove consciousness in humans unless you accept some mystic indefinable soul into the mix. Or quantum magic, which is just as mystic.

in-silico 6mo ago

From my observations, there are generally four camps in the machine consciousness discussion: 1. People who haven't really thought about it, and assume they're conscious because they talk like a human. 2. People who haven't really thought about it, and assume they can't be conscious because humans are obviously somehow special. This appears to be the largest group, and is linked to our religiously rooted culture in which human exceptionalism is the default. Those first two groups comprise the majority of people, and are not worth engaging with. 3. People who have thought about it, and came to the conclusion that they might be conscious, usually for computationalism/functionalism reasons. This is the group that I place myself in. 4. People who have thought about it, and came to the conclusion that they can't be conscious, usually for biological naturalist reasons. This seems to be the predominant group on Hacker News (among those who discuss it).

sunrunner 6mo ago
I'm not sure I'd agree that people in groups 1 and 2 aren't worth engaging with. The interesting bit to do for both cases is look at the 'they talk like a human' and 'are obviously somehow special' parts, separate the ideas of language, intelligence (memory, fluidity, abstract reasoning), _aliveness_ (as a biological process) and finally ideas about metacognition and theory of mind, and see whether their idea of consciousness as a super-bundle of the above (which is how I assume a lot of default ideas about consciousness are) actually sticks, or whether it falls apart when beings can have a subset of those properties but not all. Also, I nominate myself to be in the 'People who have thought about it and are becoming more doubtful that I myself am conscious, and the question might be moot.' group.
- in-silico 6mo ago
  I'm curious about your doubting your own consciousness statement, given that "we humans are conscious" is pretty axiomic to its definition and one of the few pieces that most agree with.
  - Kim_Bruning 6mo ago
    Take a look at Daniel Dennett, for starters! If you're looking for one of the genuine angles on this: Consciousness is horrendously under-defined, to the point some people go something like "you know, at this point I figure we'd be better off not having this word at all. " Some days that's me, with a headache.

tmvphil 6mo ago

> To fully understand the difference between the embodied robot running an algorithm on a chip and the biological mapmaker, we need to remember that for the latter, subjective experience is a given, not because of abstract information processing, but because of a specific, metabolically constituted physical reality. Total drivel. Consciousness in biological systems is "a given" because of metabolism?

jwpapi 6mo ago

I think the question goes more into ourselves as it goes into AI, we don’t know exactly how our own intelligence and conciousness works and therefore it’s very tough to impossible to compare to AI intelligence and conciousness. Are we just autocomplete machines with sufficient enough variable pseudo-randomized input?

grantcas 6mo ago

[dead]