5 ms·

Reading this, I realize I’m more scared of other intelligent entities (AI/Aliens) turning us to pets or putting us in some zoo/cages, than I am about them killi

Reading this, I realize I’m more scared of other intelligent entities (AI/Aliens) turning us to pets or putting us in some zoo/cages, than I am about them killing us.

adriand 5y ago
Ted Chiang believes it is AIs that have much to fear from us, not the other way around. He points out that to create intelligence, we will need to create something that is capable of suffering. He asks us to consider how little consideration we give to the billions of animals trapped in our system of industrial farming - animals we know are intelligent, feel emotions and are clearly capable of suffering - and then asks if we think we will treat computer programs with more consideration than that. Clearly not! The correct alternative path is to just not do it. Let’s not attempt to create intelligent beings. Given the suffering we would cause, doing so is profoundly unethical.
- sva_ 5y ago
  I always thought that people are afraid of AGI for the very reason of knowing how they treat beings of lesser intelligence, hence the popularity of movies like the matrix - because that is what we'd (presumably) do if we were in their position.
- ganzuul 5y ago
  While I have great compassion for anti-nativism, in the past few decades we have made immense sociological progress. If it is possible to create life that is incapable of shame or suffering and yet capable of enjoying its existence for its own sake, we should try to find out about it before turning the lights off.
- etothepii 5y ago
  I suppose this also extends to having children?
  - imtringued 5y ago
    Yes but they get to abuse their own children so we are "even". Please don't take this seriously. A lot of hazing culture is based around the fact that you got bullied and now you get your revenge by bullying someone else. We shouldn't perpetuate that.
- gonehome 5y ago
  I love Ted Chiang’s scifi but he’s just not well informed on the AGI stuff. I think if he ramped up on it his writing about it would better.
gonehome 5y ago
The more likely AGI risk is indifference in pursuit of optimizing a reward function with values not aligned with us. Like the anthill destroyed by the construction of a dam. The humans don’t hate the ants, aren’t putting them in zoos, they just don’t think of them in pursuit of their goals. In the AGI case it’s even possible for those goals to be “dumb” or pointless if we mess up the reward function or some unanticipated thing fits it (paper clips). The risk with a self-improving AGI is that it is way smarter than us and it can think way faster. In my experience people have a hard time really visualizing this. Imagine a human trying to trick a chimp (or an ant) into doing something so the human can pursue its goal. The gap between a human and an AGI would be even larger than that. Imagine scaling up a regular human brain’s operations per second a billion times so it’s the same architecture but thinks that much faster (no longer constrained by bio energy or head size). Humans wouldn’t be able to respond because to the machine it’d be nearly like interacting with a static object, it’d be trivially able to think/act around you. That’s why the research around AGI is about the goal alignment problem. Which is hard because humans are not perfectly aligned either. It’s also why “just unplug it” is dumb. The optimistic outcome if the alignment problem is solved is enormous intellectual capacity to solve any problem we might have.
- nradov 5y ago
  The more likely AGI risk is that we expend tremendous resources trying to develop it, only to find it doesn't work very well or do anything useful.
  - gonehome 5y ago
    There’s nothing that makes it physically impossible so it’s likely inevitable - just hard to know when. The risk of poorly aligned AGI as described is real, makes sense to try to work on the problem before we need it (when it’ll be too late).
    - nradov 5y ago
      No it's not a real risk, just a silly fantasy with no basis in scientific reality.
ck2 5y ago
Any alien with the ability to physically reach earth either via FTL or other dimensions, etc. is going to harvest our planet/solar-system for resources, they aren't going to give a damn about us. Not unlike maybe Star Trek Discovery latest season but on a lesser scale/tech. I never understand people over-evaluating our interest to something else exponentially more advanced. We wouldn't be like farm animals for labor, their tech/machines would be exponentially more efficient. This is also why I think UFO believers are not rational. Why would they even care, we'd be annoying at best.
- elvis10ten 5y ago
  Haha! So true. But even if we aren’t as intelligent as them, we could be “unique”. Someone would probably want a collection of this “cute” but less intelligent humans/animals?
- nanomonkey 5y ago
  It seems to me that we are the resources of this planet. If you just desired materials, any number of objects in space would be better suited. Unless it's the goldilocks zone of environments that you are after, a habitable planet might be harder to come by.
imtringued 5y ago
Thank god we are first (vs animal life).

Reading this, I realize I’m more scared of other intelligent entities (AI/Aliens) turning us to pets or putting us in some zoo/cages, than I am about them killing us.

adriand 5y ago

Ted Chiang believes it is AIs that have much to fear from us, not the other way around. He points out that to create intelligence, we will need to create something that is capable of suffering. He asks us to consider how little consideration we give to the billions of animals trapped in our system of industrial farming - animals we know are intelligent, feel emotions and are clearly capable of suffering - and then asks if we think we will treat computer programs with more consideration than that. Clearly not! The correct alternative path is to just not do it. Let’s not attempt to create intelligent beings. Given the suffering we would cause, doing so is profoundly unethical.

sva_ 5y ago
I always thought that people are afraid of AGI for the very reason of knowing how they treat beings of lesser intelligence, hence the popularity of movies like the matrix - because that is what we'd (presumably) do if we were in their position.
ganzuul 5y ago
While I have great compassion for anti-nativism, in the past few decades we have made immense sociological progress. If it is possible to create life that is incapable of shame or suffering and yet capable of enjoying its existence for its own sake, we should try to find out about it before turning the lights off.
etothepii 5y ago
I suppose this also extends to having children?
- imtringued 5y ago
  Yes but they get to abuse their own children so we are "even". Please don't take this seriously. A lot of hazing culture is based around the fact that you got bullied and now you get your revenge by bullying someone else. We shouldn't perpetuate that.
gonehome 5y ago
I love Ted Chiang’s scifi but he’s just not well informed on the AGI stuff. I think if he ramped up on it his writing about it would better.

gonehome 5y ago

The more likely AGI risk is indifference in pursuit of optimizing a reward function with values not aligned with us. Like the anthill destroyed by the construction of a dam. The humans don’t hate the ants, aren’t putting them in zoos, they just don’t think of them in pursuit of their goals. In the AGI case it’s even possible for those goals to be “dumb” or pointless if we mess up the reward function or some unanticipated thing fits it (paper clips). The risk with a self-improving AGI is that it is way smarter than us and it can think way faster. In my experience people have a hard time really visualizing this. Imagine a human trying to trick a chimp (or an ant) into doing something so the human can pursue its goal. The gap between a human and an AGI would be even larger than that. Imagine scaling up a regular human brain’s operations per second a billion times so it’s the same architecture but thinks that much faster (no longer constrained by bio energy or head size). Humans wouldn’t be able to respond because to the machine it’d be nearly like interacting with a static object, it’d be trivially able to think/act around you. That’s why the research around AGI is about the goal alignment problem. Which is hard because humans are not perfectly aligned either. It’s also why “just unplug it” is dumb. The optimistic outcome if the alignment problem is solved is enormous intellectual capacity to solve any problem we might have.

nradov 5y ago
The more likely AGI risk is that we expend tremendous resources trying to develop it, only to find it doesn't work very well or do anything useful.
- gonehome 5y ago
  There’s nothing that makes it physically impossible so it’s likely inevitable - just hard to know when. The risk of poorly aligned AGI as described is real, makes sense to try to work on the problem before we need it (when it’ll be too late).
  - nradov 5y ago
    No it's not a real risk, just a silly fantasy with no basis in scientific reality.

ck2 5y ago

Any alien with the ability to physically reach earth either via FTL or other dimensions, etc. is going to harvest our planet/solar-system for resources, they aren't going to give a damn about us. Not unlike maybe Star Trek Discovery latest season but on a lesser scale/tech. I never understand people over-evaluating our interest to something else exponentially more advanced. We wouldn't be like farm animals for labor, their tech/machines would be exponentially more efficient. This is also why I think UFO believers are not rational. Why would they even care, we'd be annoying at best.

elvis10ten 5y ago
Haha! So true. But even if we aren’t as intelligent as them, we could be “unique”. Someone would probably want a collection of this “cute” but less intelligent humans/animals?
nanomonkey 5y ago
It seems to me that we are the resources of this planet. If you just desired materials, any number of objects in space would be better suited. Unless it's the goldilocks zone of environments that you are after, a habitable planet might be harder to come by.

imtringued 5y ago

Thank god we are first (vs animal life).