4 ms·

I think it becomes unethical once we see strong evidence that the system could be self-aware. Before that, it's just a simulation. Simulating storm activity doe

by cmapes 13y ago

I think it becomes unethical once we see strong evidence that the system could be self-aware. Before that, it's just a simulation. Simulating storm activity doesn't make it rain.

lukifer 13y ago
It isn't a big leap to imagine a debate between aliens about whether the hairless apes on the third rock from Sol should be considered "sentient", or merely clever animals. I'd rather err far on the side of assuming sentience and personhood if there is a shadow of a doubt, simulated or otherwise.
- Dylan16807 13y ago
  Why would they debate? It's very easy to test for [sentience or fake sentience programmed by a sentient]. Unless the aliens think we're robots they can do a mirror test or something. But such a test does not work in a possibly-tainted environment. A neural net that seems to communicate might just have a hidden encoding of ELIZA. You can err on the side of assuming sentience if you want, but don't think it's out of self-preservation-empathy-logic.
  - pmr_ 13y ago
    > It's [sentience is] very easy to test for. Isn't that only true if you assume that our definition of sentience is universal or that there is no higher development of consciousness? It seems to me that it doesn't even require a very skilled science fiction/fantasy writer to develop such a concept or ability.
    - Dylan16807 13y ago
      As long as you assume whatever you're testing moves around or communicates instead of constantly naval-gazing, I'd bet even crazy-alien intelligence can be seen to be intelligent. Sure there might be higher levels, but we're not talking about higher levels?
      - pmr_ 13y ago
        You were explicitly talking about testing sentience (which does not require "moves around" or "communicates"). Sentience is simply a concept we have come up with to differentiate organisms. It wouldn't seem surprising to me if something more evolved used a different concept more appropriate to their physiology or philosophy. Hence I think you need to be careful if we are about to apply our concepts to classify something.
        Dylan16807 13y ago
        They might have a different standard of important but that has nothing to do with checking for sentience. If they're so advanced beyond that the original analogy breaks and while I might have that discussion at some point I don't think it fits here. On the topic of sentience, I just think it's exceedingly unlikely for there to be an intelligent organism that does literally nothing with its intelligence. Some kind of self-aware rock. And on top of that the mechanism of the intelligence would have to be almost impossible to study or plug wires into to try to force it to communicate.
  - lukifer 13y ago
    We choose the definition of sentience so that it includes us. There's no reason to believe that an alien species might not do the same, and that we might not make the cut. Hell, some of the worst racist rhetoric on the planet involves seeing other humans as non-sentient. See also: http://terrybison.com/page6/page6.html http://terrybison.com/page6/page6.html
    - Dylan16807 13y ago
      Why did you link a story where the ability to think/feel was clearly established in a bafflingly-different species to argue against me? It's not like this is some crazy threshold designed specifically to apply to humans, it applies to a bunch of animals too. At a certain level it's kind of like checking for turing completeness.
      - lukifer 13y ago
        The whole point is that we're blind to our own biases, as fish are blind to water. That includes our bias towards physical matter over simulated matter. (It's funny how the notion of a simulated universe unsettles people and destabilizes their narratives, despite the fact that discovering it would change precisely nothing about the subjective reality of our universe.)

by cmapes 13y ago

I think it becomes unethical once we see strong evidence that the system could be self-aware. Before that, it's just a simulation. Simulating storm activity doesn't make it rain.

lukifer 13y ago

It isn't a big leap to imagine a debate between aliens about whether the hairless apes on the third rock from Sol should be considered "sentient", or merely clever animals. I'd rather err far on the side of assuming sentience and personhood if there is a shadow of a doubt, simulated or otherwise.

Dylan16807 13y ago
Why would they debate? It's very easy to test for [sentience or fake sentience programmed by a sentient]. Unless the aliens think we're robots they can do a mirror test or something. But such a test does not work in a possibly-tainted environment. A neural net that seems to communicate might just have a hidden encoding of ELIZA. You can err on the side of assuming sentience if you want, but don't think it's out of self-preservation-empathy-logic.
- pmr_ 13y ago
  > It's [sentience is] very easy to test for. Isn't that only true if you assume that our definition of sentience is universal or that there is no higher development of consciousness? It seems to me that it doesn't even require a very skilled science fiction/fantasy writer to develop such a concept or ability.
  - Dylan16807 13y ago
    As long as you assume whatever you're testing moves around or communicates instead of constantly naval-gazing, I'd bet even crazy-alien intelligence can be seen to be intelligent. Sure there might be higher levels, but we're not talking about higher levels?
    - pmr_ 13y ago
      You were explicitly talking about testing sentience (which does not require "moves around" or "communicates"). Sentience is simply a concept we have come up with to differentiate organisms. It wouldn't seem surprising to me if something more evolved used a different concept more appropriate to their physiology or philosophy. Hence I think you need to be careful if we are about to apply our concepts to classify something.
      - Dylan16807 13y ago
        They might have a different standard of important but that has nothing to do with checking for sentience. If they're so advanced beyond that the original analogy breaks and while I might have that discussion at some point I don't think it fits here. On the topic of sentience, I just think it's exceedingly unlikely for there to be an intelligent organism that does literally nothing with its intelligence. Some kind of self-aware rock. And on top of that the mechanism of the intelligence would have to be almost impossible to study or plug wires into to try to force it to communicate.
- lukifer 13y ago
  We choose the definition of sentience so that it includes us. There's no reason to believe that an alien species might not do the same, and that we might not make the cut. Hell, some of the worst racist rhetoric on the planet involves seeing other humans as non-sentient. See also: http://terrybison.com/page6/page6.html http://terrybison.com/page6/page6.html
  - Dylan16807 13y ago
    Why did you link a story where the ability to think/feel was clearly established in a bafflingly-different species to argue against me? It's not like this is some crazy threshold designed specifically to apply to humans, it applies to a bunch of animals too. At a certain level it's kind of like checking for turing completeness.
    - lukifer 13y ago
      The whole point is that we're blind to our own biases, as fish are blind to water. That includes our bias towards physical matter over simulated matter. (It's funny how the notion of a simulated universe unsettles people and destabilizes their narratives, despite the fact that discovering it would change precisely nothing about the subjective reality of our universe.)