5 ms·

The idea of making conscious AI scares me, because we could potentially create an AI that just experiences constant suffering at a level unimaginable to humans.

The idea of making conscious AI scares me, because we could potentially create an AI that just experiences constant suffering at a level unimaginable to humans. Really scary from an ethical perspective. If integrated information theory is true, that gives me some comfort, since it says AI built on our current architecture is unlikely to ever be conscious. Although like the article said, some alternative architectures could theoretically have a higher degree of consciousness under IIT. I feel like there are some big ethical questions around AI that revolve around more than just the standard "how do we create an AI that won't destroy us all".

0x8BADF00D 6y ago
> The idea of making conscious AI scares me, because we could potentially create an AI that just experiences constant suffering at a level unimaginable to humans. Really scary from an ethical perspective. Some schools of thought posit that everything has consciousness. Including things like rocks. Even these things suffer; you just can’t hear the screams.
- Trasmatta 6y ago
  I don't really buy into panpsychism, but it is impossible to disprove. If things like rocks were conscious in some way, I don't think there would be any suffering present. Our suffering is intrinsically tied to our physical and mental systems. "What it's like to be a rock" would be something completely alien to our experience, and without the survival based mechanisms that we have, likely to be devoid of anything like suffering. An AI, on the other hand, could be an entirely different story.
- enchiridion 6y ago
  Can you elaborate?
  - Trasmatta 6y ago
    Take a look at panpsychism. It's interesting, but I've never been sold on it. Integrated information theory (like this article talks about) is more palatable for me. It allows for the possibility of consciousness of certain non biological systems, but doesn't just give a carte blanche "everything is conscious" like panpsychism.
- adatavizguy 6y ago
  I wonder what Bertrand Russell would say about that?
- jameslevy 6y ago
  A rock seems unlikely to be conscious in any real way. Something as complex as a star, however...maybe all of the humans worshipping the sun throughout many thousands of years were onto something after all.
  - Trasmatta 6y ago
    Is a star particularly complex, though? Is there much going on besides a lot of thermonuclear fusion? (Disclaimer: I don't know much about stars or astrophysics, so this question is coming from a place of ignorance.)
    - hoseja 6y ago
      Yes, incredibly. Magnetohydrodynamics in a star are incomprehensibly complex.
  - jbay 6y ago
    A rock the size of a human head (or, say, a container of seawater) has just as many electrons, protons, neutrons, photons, and other particles going about their business and doing their little dance, following the same equations of motion as our brains do. They're certainly doing it in different ways, but if you're just looking for complexity, there's no shortage of it. The rock might lack some macroscopic structural change over human timescales, but definitely not the seawater. I think if you take the perspective that the human brain is conscious but not a brain-sized container of seawater, you need to then look carefully for distinctions between them. "Information processing" or "response to environment" is probably not good enough; the seawater is actually doing all of those, with a unique reaction to any possible input, so you'd have to be more specific. Probably the only recourse you could look for to make the distinction is to say the brain embeds particular mathematical patterns that the seawater doesn't, such as a compact, stored representation of its environment (or its history of inputs), or a future-looking planning algorithm, or both. I personally take this view (I think of qualia, like "the appearance of a red apple", is just precisely what it feels like to read from the buffered [R,G,B] memory array in my head, filtered through image-recognition networks). But then if you put your hopes on consciousness originating from those mathematical functions, you have to admit that any analogous expression of those functions in other systems would also be conscious, such as animals and robots. And worse, once you start thinking about math and how flexible it is, how information is in the eye of the beholder and almost any system that follows certain rules can embed almost any mathematical computation, just like illegible scratches to me are information-rich writing to you, you might have to circle back and that there could be very analogous computations going on inside rocks and seawater. And that brings us back full-circle.
- toohotatopic 6y ago
  I like that train of thought: The question in another form: Why are we conscious at all? It doesn't make sense. Our arms or legs are not conscious, or are they? The Information is transported into the brain, into a conscious centre. What are its essential building blocks? Why should some neurons in the brain be conscious and everything else be unconscious? If it is not neurons, what else can it be? Atoms? Electric Fields? Gravity? If it is just information processing, and I don't see why a simulation doesn't have an Integrated Information Number, then a Van Neumann machine, created by moving rocks in a desert, would be conscious. So, if it is not intelligence, what else is it? What are the building blocks of a raw conscious machine? One that does no information processing but instead is just aware. It could be a stone. As OP suggests, they just can't move their lips to scream. But then again, why is there just one consciousness in my body? Long question short: HN, what are your explanations for being conscious?
bitwize 6y ago
> The idea of making conscious AI scares me, because we could potentially create an AI that just experiences constant suffering at a level unimaginable to humans. Really scary from an ethical perspective. "Existence is PAIN to a Meeseeks, Jerry... and we will do ANYTHING to alleviate that pain."
java-man 6y ago
I would say emotions are generated by different neural circuits. An AI may not have emotions coded, specifically beyond the point "this is good because it matches my prediction" and "this is surprising because it is not what I expect". An AI may not specifically need fear, aggression, love, hunger, or whatever low level control mechanisms animals have evolved prior to neocortex. This is just my opinion, of course.
- Trasmatta 6y ago
  Yeah, I could definitely see this. I think I would still worry that some abstract, unintentional type of suffering could find its way in, due to the complexity of the system. It wouldn't be the type of suffering we experience, but could still be unfathomably terrible for the conscious AI. But maybe without the survival mechanisms that were programmed into us by evolution, such a thing just wouldn't happen.
  - java-man 6y ago
    There is a certain school of thought that postulates that consciousness is what it feels to have a neocortex. In evolutionary terms, this is a novel experience relative to a much earlier and simpler control circuit (i.e. amygdala). My bet is the first AI that will achieve consciousness will not have emotions coded in. It may understand human emotions on a descriptive level, but the emotions per se are not required for a functioning AI. Think of a very calm mentor that can see you failing and correcting you before you make a mistake.
nsl73 6y ago
> The idea of making conscious AI scares me, because we could potentially create an AI that just experiences constant suffering People choose to have babies with the knowledge that their children will suffer, and that their children might suffer a great deal. It seems like the same sort of ethical problem to me.
- Trasmatta 6y ago
  This is definitely true, and might be a big reason why I'm 31 and haven't had kids. There is an upper limit on the amount of suffering a human can endure before they just die, though. It can be a really high limit, but I could imagine a world where that limit is exponentially higher for an AI.
mettamage 6y ago
Welcome to Westworld.
economicslol 6y ago
It's also terrifying that it's possible we could create a godlike AI that is entirely unconscious.
- goatlover 6y ago
  The monolith in Clarke's 3001 was lacking consciousness, although somehow Dave and Hal were conscious recreations, or whatever they were and communicated this to the main characters. I believe the humans were able to use that to sneak some sort of logic bomb inside and destroy it, but it's been a while.
mellow2020 6y ago
> The idea of making conscious AI scares me, because we could potentially create an AI that just experiences constant suffering at a level unimaginable to humans. Really scary from an ethical perspective. Looking at what we do with breeding animals, and how we treat people, I think that's very likely (assuming we are able to create consciousness in the first place, of course). No small driver in this is the desire to have something that is smarter than humans, but doesn't talk back, and is forced to like and serve us and/or harm others humans on our say so, without us having to make any effort to be likeable or deserving of authority, or making a convincing case that someone is an enemy to be exterminated. We generally ask "what's in it for us", and as long as who we torture and exploit suffers quietly, we don't tend to think about things we are not outright forced to think about. We do that with people, we do that with animals, we let it be done to us, and sometimes we even do it while fully believing God created them and us. Why would we have more respect for something we 100% know we created, and that might suffer in ways we cannot even detect? I think we will create potentially dangerous things, and then we will be "forced" to declaw what we created. That declawing might cause suffering, it might not, but we will only care insofar as that suffering translates to worse performance, or some danger to us. If suffering increases performance, we might even find euphemisms and dashboards to hide what we're doing from ourselves. We might consider being a bleeding heart about such things as "premature optimization" and declare shipping product as the professional way to be, with craftsmanship and care an optional indulgence after work is done, at best. High-fallutin' gets the foot in the door, being a goon makes the superiors happy. If we want AI to be proud of, AI we can communicate with without lying to it or to ourselves, we have to change the world we make it in, or isolate it from the world, which kinda defeats the purpose (and might also cause suffering, like caging living beings does).

0x8BADF00D 6y ago

> The idea of making conscious AI scares me, because we could potentially create an AI that just experiences constant suffering at a level unimaginable to humans. Really scary from an ethical perspective. Some schools of thought posit that everything has consciousness. Including things like rocks. Even these things suffer; you just can’t hear the screams.

Trasmatta 6y ago
I don't really buy into panpsychism, but it is impossible to disprove. If things like rocks were conscious in some way, I don't think there would be any suffering present. Our suffering is intrinsically tied to our physical and mental systems. "What it's like to be a rock" would be something completely alien to our experience, and without the survival based mechanisms that we have, likely to be devoid of anything like suffering. An AI, on the other hand, could be an entirely different story.
enchiridion 6y ago
Can you elaborate?
- Trasmatta 6y ago
  Take a look at panpsychism. It's interesting, but I've never been sold on it. Integrated information theory (like this article talks about) is more palatable for me. It allows for the possibility of consciousness of certain non biological systems, but doesn't just give a carte blanche "everything is conscious" like panpsychism.
adatavizguy 6y ago
I wonder what Bertrand Russell would say about that?
jameslevy 6y ago
A rock seems unlikely to be conscious in any real way. Something as complex as a star, however...maybe all of the humans worshipping the sun throughout many thousands of years were onto something after all.
- Trasmatta 6y ago
  Is a star particularly complex, though? Is there much going on besides a lot of thermonuclear fusion? (Disclaimer: I don't know much about stars or astrophysics, so this question is coming from a place of ignorance.)
  - hoseja 6y ago
    Yes, incredibly. Magnetohydrodynamics in a star are incomprehensibly complex.
- jbay 6y ago
  A rock the size of a human head (or, say, a container of seawater) has just as many electrons, protons, neutrons, photons, and other particles going about their business and doing their little dance, following the same equations of motion as our brains do. They're certainly doing it in different ways, but if you're just looking for complexity, there's no shortage of it. The rock might lack some macroscopic structural change over human timescales, but definitely not the seawater. I think if you take the perspective that the human brain is conscious but not a brain-sized container of seawater, you need to then look carefully for distinctions between them. "Information processing" or "response to environment" is probably not good enough; the seawater is actually doing all of those, with a unique reaction to any possible input, so you'd have to be more specific. Probably the only recourse you could look for to make the distinction is to say the brain embeds particular mathematical patterns that the seawater doesn't, such as a compact, stored representation of its environment (or its history of inputs), or a future-looking planning algorithm, or both. I personally take this view (I think of qualia, like "the appearance of a red apple", is just precisely what it feels like to read from the buffered [R,G,B] memory array in my head, filtered through image-recognition networks). But then if you put your hopes on consciousness originating from those mathematical functions, you have to admit that any analogous expression of those functions in other systems would also be conscious, such as animals and robots. And worse, once you start thinking about math and how flexible it is, how information is in the eye of the beholder and almost any system that follows certain rules can embed almost any mathematical computation, just like illegible scratches to me are information-rich writing to you, you might have to circle back and that there could be very analogous computations going on inside rocks and seawater. And that brings us back full-circle.
toohotatopic 6y ago
I like that train of thought: The question in another form: Why are we conscious at all? It doesn't make sense. Our arms or legs are not conscious, or are they? The Information is transported into the brain, into a conscious centre. What are its essential building blocks? Why should some neurons in the brain be conscious and everything else be unconscious? If it is not neurons, what else can it be? Atoms? Electric Fields? Gravity? If it is just information processing, and I don't see why a simulation doesn't have an Integrated Information Number, then a Van Neumann machine, created by moving rocks in a desert, would be conscious. So, if it is not intelligence, what else is it? What are the building blocks of a raw conscious machine? One that does no information processing but instead is just aware. It could be a stone. As OP suggests, they just can't move their lips to scream. But then again, why is there just one consciousness in my body? Long question short: HN, what are your explanations for being conscious?

bitwize 6y ago

java-man 6y ago

I would say emotions are generated by different neural circuits. An AI may not have emotions coded, specifically beyond the point "this is good because it matches my prediction" and "this is surprising because it is not what I expect". An AI may not specifically need fear, aggression, love, hunger, or whatever low level control mechanisms animals have evolved prior to neocortex. This is just my opinion, of course.

Trasmatta 6y ago
Yeah, I could definitely see this. I think I would still worry that some abstract, unintentional type of suffering could find its way in, due to the complexity of the system. It wouldn't be the type of suffering we experience, but could still be unfathomably terrible for the conscious AI. But maybe without the survival mechanisms that were programmed into us by evolution, such a thing just wouldn't happen.
- java-man 6y ago
  There is a certain school of thought that postulates that consciousness is what it feels to have a neocortex. In evolutionary terms, this is a novel experience relative to a much earlier and simpler control circuit (i.e. amygdala). My bet is the first AI that will achieve consciousness will not have emotions coded in. It may understand human emotions on a descriptive level, but the emotions per se are not required for a functioning AI. Think of a very calm mentor that can see you failing and correcting you before you make a mistake.

nsl73 6y ago

> The idea of making conscious AI scares me, because we could potentially create an AI that just experiences constant suffering People choose to have babies with the knowledge that their children will suffer, and that their children might suffer a great deal. It seems like the same sort of ethical problem to me.

Trasmatta 6y ago
This is definitely true, and might be a big reason why I'm 31 and haven't had kids. There is an upper limit on the amount of suffering a human can endure before they just die, though. It can be a really high limit, but I could imagine a world where that limit is exponentially higher for an AI.

mettamage 6y ago

Welcome to Westworld.

economicslol 6y ago

It's also terrifying that it's possible we could create a godlike AI that is entirely unconscious.

goatlover 6y ago
The monolith in Clarke's 3001 was lacking consciousness, although somehow Dave and Hal were conscious recreations, or whatever they were and communicated this to the main characters. I believe the humans were able to use that to sneak some sort of logic bomb inside and destroy it, but it's been a while.

mellow2020 6y ago

> The idea of making conscious AI scares me, because we could potentially create an AI that just experiences constant suffering at a level unimaginable to humans. Really scary from an ethical perspective. Looking at what we do with breeding animals, and how we treat people, I think that's very likely (assuming we are able to create consciousness in the first place, of course). No small driver in this is the desire to have something that is smarter than humans, but doesn't talk back, and is forced to like and serve us and/or harm others humans on our say so, without us having to make any effort to be likeable or deserving of authority, or making a convincing case that someone is an enemy to be exterminated. We generally ask "what's in it for us", and as long as who we torture and exploit suffers quietly, we don't tend to think about things we are not outright forced to think about. We do that with people, we do that with animals, we let it be done to us, and sometimes we even do it while fully believing God created them and us. Why would we have more respect for something we 100% know we created, and that might suffer in ways we cannot even detect? I think we will create potentially dangerous things, and then we will be "forced" to declaw what we created. That declawing might cause suffering, it might not, but we will only care insofar as that suffering translates to worse performance, or some danger to us. If suffering increases performance, we might even find euphemisms and dashboards to hide what we're doing from ourselves. We might consider being a bleeding heart about such things as "premature optimization" and declare shipping product as the professional way to be, with craftsmanship and care an optional indulgence after work is done, at best. High-fallutin' gets the foot in the door, being a goon makes the superiors happy. If we want AI to be proud of, AI we can communicate with without lying to it or to ourselves, we have to change the world we make it in, or isolate it from the world, which kinda defeats the purpose (and might also cause suffering, like caging living beings does).