4 ms·

Neal Stephenson touches on this topic in several of his novels. Probably the most concrete of this is "Fall; or Dodge in Hell" which involves a simulation of pe

Neal Stephenson touches on this topic in several of his novels. Probably the most concrete of this is "Fall; or Dodge in Hell" which involves a simulation of people's scanned brains rediscovering qualia and constructing their own simulated world from scratch. In the book, two of the deceased digital "souls" eventually mate and produce digital offspring and the whole simulation starts consuming more and more resources. His Baroque Cycle series also touches on this in several places. One funny side plot involves a freed African slave (Dappa) who speaks dozens of different languages and is highly intelligent and an aristocratic person who maintains that of course this former slave (who is obviously a lot smarter than this aristocrat) is just a trained monkey that naturally is not conscious even though he is quite clever with language. The same books also have a lot of side plots involving Leibniz and various attempts to build thinking/computing machines. The Dappa plot is probably the closest to a lot of debates there will be around AGI with people likely to insist for all sorts of reasons rooted in philosophy, religion, etc. that even though the AGI walks, talks, etc. like a duck, it can't be a duck. At some point, we'll have AGI that pass any test we can think of and we'll still have people arguing that these cannot be conscious.

card_zero 5mo ago
The clinching thing for me is how the AGI is supposed to work. If it's the same as our theory of how we work, then fine, it's one of us. It wouldn't even have to work very well.
- vixen99 5mo ago
  I always like to remind myself that AI is trained on material fed to it, created and written (somewhere along the line) entirely by humans - as they perceive & interpret the universe around them.
  - gbgarbeb 5mo ago
    Not for long, if I have my way.
david-gpu 5mo ago
> At some point, we'll have AGI that pass any test we can think of and we'll still have people arguing that these cannot be conscious. Is there a distinction in your mind between consciousness and intelligence? Is it possible, for example, for a machine to solve complex problems but not be conscious? Or vice versa, can an animal or a person be very unintelligent yet still conscious?
- otikik 5mo ago
  A philosophical zombie [1] is essentially a mobile, autonomous human body devoid of consciousness. Critically, still giving all of the external cues that it has in fact consciousness. It is supposed to prove that physicalism doesn't work. "The sense of consciousness" is used like a "soul, with extra steps". In my humble opinion, which I have no way to prove or disprove, consciousness ("as a soul with extra steps") does not exist, and we are all philosophical zombies. Consciousness "as an amalgamation of complex biological signals and neural interactions that has evolved through millions of years as a successful survival strategy" does exist, and that is all that is needed. https://en.wikipedia.org/wiki/Philosophical_zombie https://en.wikipedia.org/wiki/Philosophical_zombie
  - otikik 5mo ago
    Serves me well for answering a comment before reading the article. This is basically what the author says, even pointing to the Philosophical zombie and all. Shame on me.
  - tim333 5mo ago
    I know I'm not a philosophical zombie based on >For example, if a philosophical zombie were poked with a sharp object, it would not feel any pain, but it would react exactly the way any conscious human would. in the Wikipedia article because I don't get on well with the sharp poking. On the basis of Occam's razor I assume all other humans and higher animals are similar. It would be odd if evolution had made me different from all the others.
    - otikik 5mo ago
lelanthran 5mo ago
> At some point, we'll have AGI that pass any test we can think of and we'll still have people arguing that these cannot be conscious. You've already drawn your line in the sand (i.e. they are conscious). In that case, you can't also claim that we should continue producing them by the millions at the flick of a switch. The AI-is-conscious crowd will have to choose - either they are conscious, in which case they should not be birthed, or they are not conscious in which case we can use them as tools. You can't have both and still be logically consistent.
- ticulatedspline 5mo ago
  I think you mean morally consistent. though even then humans don't have any real qualms about that. Dogs and livestock are conscious, we use those as tools.
  - lelanthran 5mo ago
    > Dogs and livestock are conscious, we use those as tools. Sure, but we don't create as many as we can, then kill them at the end of the day when the work is done. If you want to call AIs conscious, you can't also campaign for willy-nilly creation, even if they do get a status of a working tool (dogs, etc). If you think they are conscious, which implies laws protecting them, then the "owner" of them gets an obligation (you can't do whatever you want to a dog, for example).
    - wvenable 5mo ago
      > Sure, but we don't create as many as we can, then kill them at the end of the day when the work is done. So it's a matter of scale then? We breed pigs, for example, and kill them a mere fraction into their overall lifespan. So it's ok as long as we don't kill them faster? > If you want to call AIs conscious, you can't also campaign for willy-nilly creation, even if they do get a status of a working tool (dogs, etc). I don't see why this follows? Humans also exist for a period and then blink out of existence. Why does the timeframe matter? Our end was inevitable at the moment of our creation. > If you think they are conscious, which implies laws protecting them You're conflating conscious with personhood. Clearly in a future where you can create a conscious entity instantly and terminate it just as quickly these entities are unlikely to qualify for personhood.
an0malous 5mo ago
> Probably the most concrete of this is "Fall; or Dodge in Hell" which involves a simulation of people's scanned brains rediscovering qualia and constructing their own simulated world from scratch. This is a great example for a discussion about The Hard Problem. Here is the description from the book of the inner experience of this scanned brain as it gets booted up: > What came next could not, of course, be described without using words. But that was deceptive in a way since he no longer had words. Nor did he have memories, or coherent thoughts, or any other way to describe or think about the qualia he was experiencing. And those qualia were of miserably low quality. To the extent he was seeing, he was seeing incoherent patterns of fluctuating light. For people of a certain age, the closest descriptor for this was “static”: the sheets, waves, and bands of noise that had covered the screens of malfunctioning television sets. Static, in a sense, wasn’t real. It was simply what you got out of a system when it was unable to lock on to any strong signal—“Strong” meaning actually conveying useful, or at least understandable, information. Modern computer screens were smart enough to just shut down, or put up an error message, when the signal was lost. Old analog sets had no choice but to display something. The electron beam was forever scanning, a mindless beacon, and if you fed it nothing else it would produce a visual map of whatever was contingently banging around in its circuitry: some garbled mix of electrical noise from Mom’s vacuum cleaner, Dad’s shaver, solar flares, stray transmissions caroming off the ionosphere, and whatever happened when little feedback loops on the circuit board got out of hand. Likewise, to the extent he was hearing anything, it was just an inchoate hiss. The Hard Problem asks: who is experiencing this qualia and why is there an experiencer at all? Stephenson writes how this simulated brain is experiencing static as it condenses into meaningful patterns, but he implicitly starts with someone experiencing this static qualia. If this is the very beginning of the simulated brain booting up, where did this experiencer come from? > At some point, we'll have AGI that pass any test we can think of and we'll still have people arguing that these cannot be conscious. Because you're conflating intelligence with consciousness. There is no test for consciousness. In fact, you can't even prove that other human beings are conscious, you only know that you yourself are because it's self evident to you (cogito ergo sum, I think therefore I am). The whole point of the hard problem is that you can imagine something exactly like a human being that passes every test of being a human being (e.g. an AI) but still not be sure that it has any inner experience.

card_zero 5mo ago

The clinching thing for me is how the AGI is supposed to work. If it's the same as our theory of how we work, then fine, it's one of us. It wouldn't even have to work very well.

vixen99 5mo ago
I always like to remind myself that AI is trained on material fed to it, created and written (somewhere along the line) entirely by humans - as they perceive & interpret the universe around them.
- gbgarbeb 5mo ago
  Not for long, if I have my way.

david-gpu 5mo ago

> At some point, we'll have AGI that pass any test we can think of and we'll still have people arguing that these cannot be conscious. Is there a distinction in your mind between consciousness and intelligence? Is it possible, for example, for a machine to solve complex problems but not be conscious? Or vice versa, can an animal or a person be very unintelligent yet still conscious?

otikik 5mo ago
A philosophical zombie [1] is essentially a mobile, autonomous human body devoid of consciousness. Critically, still giving all of the external cues that it has in fact consciousness. It is supposed to prove that physicalism doesn't work. "The sense of consciousness" is used like a "soul, with extra steps". In my humble opinion, which I have no way to prove or disprove, consciousness ("as a soul with extra steps") does not exist, and we are all philosophical zombies. Consciousness "as an amalgamation of complex biological signals and neural interactions that has evolved through millions of years as a successful survival strategy" does exist, and that is all that is needed. https://en.wikipedia.org/wiki/Philosophical_zombie https://en.wikipedia.org/wiki/Philosophical_zombie
- otikik 5mo ago
  Serves me well for answering a comment before reading the article. This is basically what the author says, even pointing to the Philosophical zombie and all. Shame on me.
- tim333 5mo ago
  I know I'm not a philosophical zombie based on >For example, if a philosophical zombie were poked with a sharp object, it would not feel any pain, but it would react exactly the way any conscious human would. in the Wikipedia article because I don't get on well with the sharp poking. On the basis of Occam's razor I assume all other humans and higher animals are similar. It would be odd if evolution had made me different from all the others.
  - otikik 5mo ago

lelanthran 5mo ago

> At some point, we'll have AGI that pass any test we can think of and we'll still have people arguing that these cannot be conscious. You've already drawn your line in the sand (i.e. they are conscious). In that case, you can't also claim that we should continue producing them by the millions at the flick of a switch. The AI-is-conscious crowd will have to choose - either they are conscious, in which case they should not be birthed, or they are not conscious in which case we can use them as tools. You can't have both and still be logically consistent.

ticulatedspline 5mo ago
I think you mean morally consistent. though even then humans don't have any real qualms about that. Dogs and livestock are conscious, we use those as tools.
- lelanthran 5mo ago
  > Dogs and livestock are conscious, we use those as tools. Sure, but we don't create as many as we can, then kill them at the end of the day when the work is done. If you want to call AIs conscious, you can't also campaign for willy-nilly creation, even if they do get a status of a working tool (dogs, etc). If you think they are conscious, which implies laws protecting them, then the "owner" of them gets an obligation (you can't do whatever you want to a dog, for example).
  - wvenable 5mo ago
    > Sure, but we don't create as many as we can, then kill them at the end of the day when the work is done. So it's a matter of scale then? We breed pigs, for example, and kill them a mere fraction into their overall lifespan. So it's ok as long as we don't kill them faster? > If you want to call AIs conscious, you can't also campaign for willy-nilly creation, even if they do get a status of a working tool (dogs, etc). I don't see why this follows? Humans also exist for a period and then blink out of existence. Why does the timeframe matter? Our end was inevitable at the moment of our creation. > If you think they are conscious, which implies laws protecting them You're conflating conscious with personhood. Clearly in a future where you can create a conscious entity instantly and terminate it just as quickly these entities are unlikely to qualify for personhood.

an0malous 5mo ago

> Probably the most concrete of this is "Fall; or Dodge in Hell" which involves a simulation of people's scanned brains rediscovering qualia and constructing their own simulated world from scratch. This is a great example for a discussion about The Hard Problem. Here is the description from the book of the inner experience of this scanned brain as it gets booted up: > What came next could not, of course, be described without using words. But that was deceptive in a way since he no longer had words. Nor did he have memories, or coherent thoughts, or any other way to describe or think about the qualia he was experiencing. And those qualia were of miserably low quality. To the extent he was seeing, he was seeing incoherent patterns of fluctuating light. For people of a certain age, the closest descriptor for this was “static”: the sheets, waves, and bands of noise that had covered the screens of malfunctioning television sets. Static, in a sense, wasn’t real. It was simply what you got out of a system when it was unable to lock on to any strong signal—“Strong” meaning actually conveying useful, or at least understandable, information. Modern computer screens were smart enough to just shut down, or put up an error message, when the signal was lost. Old analog sets had no choice but to display something. The electron beam was forever scanning, a mindless beacon, and if you fed it nothing else it would produce a visual map of whatever was contingently banging around in its circuitry: some garbled mix of electrical noise from Mom’s vacuum cleaner, Dad’s shaver, solar flares, stray transmissions caroming off the ionosphere, and whatever happened when little feedback loops on the circuit board got out of hand. Likewise, to the extent he was hearing anything, it was just an inchoate hiss. The Hard Problem asks: who is experiencing this qualia and why is there an experiencer at all? Stephenson writes how this simulated brain is experiencing static as it condenses into meaningful patterns, but he implicitly starts with someone experiencing this static qualia. If this is the very beginning of the simulated brain booting up, where did this experiencer come from? > At some point, we'll have AGI that pass any test we can think of and we'll still have people arguing that these cannot be conscious. Because you're conflating intelligence with consciousness. There is no test for consciousness. In fact, you can't even prove that other human beings are conscious, you only know that you yourself are because it's self evident to you (cogito ergo sum, I think therefore I am). The whole point of the hard problem is that you can imagine something exactly like a human being that passes every test of being a human being (e.g. an AI) but still not be sure that it has any inner experience.