3 ms·

If the simulation decides to copy itself, and allows the copy to run independently, are there now two autonomous people? If the copy is allowed to advance one

If the simulation decides to copy itself, and allows the copy to run independently, are there now two autonomous people? If the copy is allowed to advance one nanosecond and is then deleted, leaving the original running, has a murder been committed? What if the copy never runs at all? What if the original is deleted instead of the copy? What if the copy receives identical input, and thus has identical state as the original, up to the point it is deleted? What if some aspects of the copy are "optimized" by reusing the results of the original? What if the entire copy just mirrors the state of the original, without recomputing anything? How can we say exactly how many copies actually exist? What if the original simulation modifies the copy so that it wants to kill itself, which it does, once it starts running? What if it's modified to just not care whether it lives or dies? What if the copy is modified in various other ways? Are some kinds of modifications ethical, while others are not? Point being, this course of AI development does not spare us having to completely rethink our models of ethics.

hastur 15y ago
Very interesting questions, I'll be happy to try to answer them: 1) yes 2a) yes 2b) It never lived, it was a "frozen corpse", and since you can make copies at will, just having a copy doesn't constitute a new life until it's run. 2c) If the original has been deleted at the same state as the copy is started, i.e. there's no interruption from the subjective point of view of the copy, then it's just a transfer. 3a) Since there's zero information difference, I'd say it is OK. 3b) They're two separate people with equal rights. 3c,d) No computing = no thinking = no life to speak of. 4a,b) The same as if you kidnapped someone from the street and convinced them to kill themselves (say, by drugging them). 4c,d) Very interesting. On the one hand, in case of many modifications, you can't really foresee the consequences, so all should be considered unethical to prevent unintended suffering. On the other hand - what is human parenting and bringing up if not shaping (or "modifying") a not-yet-sovereign mind. And we have no problem with that. But then, what if some psychotic parent brings up a kid with an implicit conviction that he should kill himself upon reaching adulthood (i.e. legal sovereignty). We'd put them for life in jail. But what if a parent has no bad intentions, but is just a very crappy parent and his kid grows up to be very miserable and in his 20's commits suicide because he can't deal with life. Well, we could pass moral judgment on the parent (if we somehow had the knowledge and thus certainty that it was a result of bad bringing up), but we couldn't really convict them in a court of law. So yeah, we're having a serious split here between moral judgment and what's practical. (Much like pro- and anti- abortion.) So I'd say playing with the brains of sentient beings is unethical in general. Practically though, there's no way to stop a "virtual" person from running all kinds of crazy stuff on their private machines. Unless you want to police the internals of all such brain-simulating computers, but then you're conducting mind-control of people, and that's totally unacceptable. 5) Agreed, but I think if we engage in it already, we can get to some very interesting conclusions that we might find useful even in today's world.
- extension 15y ago
  2b) It never lived, it was a "frozen corpse", and since you can make copies at will, just having a copy doesn't constitute a new life until it's run. How long does it have to run before it's alive? If the interval is short enough then there will be no change in state whatsoever. So in what way does the mind state have to change in order for it to have lived? What if that change is so simple that you can calculate it in your head? Have you created a new life just by thinking about what will happen next in the simulation? By writing it down? 3a) Since there's zero information difference, I'd say it is OK. So you would have no objection to being killed if there was an identical copy of you in existence somewhere? Why does that matter? Why do you care more about this copy of you than you do about anybody else? Why should a stranger care that there is at least one copy of you? 4c,d) Very interesting. On the one hand, in case of many modifications, you can't really foresee the consequences, so all should be considered unethical to prevent unintended suffering. But you are modifying your own mind all the time, by forming memories and learning skills. To some degree, you choose what you think and experience. Imagine that you have ultimate control over every facet of your mind. You live in a computer, so there is no "natural" mental life. You must choose exactly how your mind will progress. If you make a frozen copy of yourself, it hasn't diverged yet, so you can modify it however you like, since it's you. And if new, unique people are to be created at all, someone will have to choose how their mind works. There is no longer a "natural" reproductive process to decide this. How can choices be made in the new person's interests when it is those very interests that are being chosen?
  - hastur 15y ago
    2b) I would assume that the machine on which the AI is run is digital (i.e. discrete), so yeah, a single cycle (tick of the processor) could very well mean change. And since you've got to draw the line somewhere, I'd say that's enough. 3a) By killing me and starting an identical copy of me somewhere else, you're creating a discontinuity that infringes upon my identity (or same-ness) because there would be a very strong discontinuity in inputs simply because of the different location. Moreover, a third party exercising control over any of my copies (and especially the one that is running) is in violation of my sovereignty. [That's also why a stranger should care about my copies.] 4c,d) That's precisely it: I am doing it! I can do with myself whatever I want. And yes, I do imagine having the control about every facet of my mind. I actually dream about it. :) Now, when I manipulate a copy of me, that's a little different, because as I've stated earlier, a copy spun off to run independently is sovereign from its "original". That's why manipulating own copy should morally be considered equivalent to manipulating a third party, assuming that copy will be let run separately. To your last point, I agree, that's the most delicate part - creating new beings. As far as copies are concerned, my immediate instinct is that only identical copies should be considered ethical, since any modifications can have unpredictable consequences. But I'm not sure this should be enforced by the AI community / data center, because I value personal sovereignty and habeas corpus virtualis above all else. ;) I won't speak of creating completely new, original beings, because that topic deserves a ton of thought and a philosophical treatise of its own. But theoretically, you could try to mimic natural processes of child brain development. Of course, as you've pointed out, someone has to choose the parameters of that developmental algorithm, just as a search engine is never really objective, because people wrote it and made some choices in the process. But that last example also shows we can have some trust in such choices for practical reasons. On a side note: I would highly recommend the book "Diaspora" by Greg Egan. The first chapter (Orphanogenesis) speaks about the development of a virtual mind and to me is among the best works of science fiction of all time.

hastur 15y ago

Very interesting questions, I'll be happy to try to answer them: 1) yes 2a) yes 2b) It never lived, it was a "frozen corpse", and since you can make copies at will, just having a copy doesn't constitute a new life until it's run. 2c) If the original has been deleted at the same state as the copy is started, i.e. there's no interruption from the subjective point of view of the copy, then it's just a transfer. 3a) Since there's zero information difference, I'd say it is OK. 3b) They're two separate people with equal rights. 3c,d) No computing = no thinking = no life to speak of. 4a,b) The same as if you kidnapped someone from the street and convinced them to kill themselves (say, by drugging them). 4c,d) Very interesting. On the one hand, in case of many modifications, you can't really foresee the consequences, so all should be considered unethical to prevent unintended suffering. On the other hand - what is human parenting and bringing up if not shaping (or "modifying") a not-yet-sovereign mind. And we have no problem with that. But then, what if some psychotic parent brings up a kid with an implicit conviction that he should kill himself upon reaching adulthood (i.e. legal sovereignty). We'd put them for life in jail. But what if a parent has no bad intentions, but is just a very crappy parent and his kid grows up to be very miserable and in his 20's commits suicide because he can't deal with life. Well, we could pass moral judgment on the parent (if we somehow had the knowledge and thus certainty that it was a result of bad bringing up), but we couldn't really convict them in a court of law. So yeah, we're having a serious split here between moral judgment and what's practical. (Much like pro- and anti- abortion.) So I'd say playing with the brains of sentient beings is unethical in general. Practically though, there's no way to stop a "virtual" person from running all kinds of crazy stuff on their private machines. Unless you want to police the internals of all such brain-simulating computers, but then you're conducting mind-control of people, and that's totally unacceptable. 5) Agreed, but I think if we engage in it already, we can get to some very interesting conclusions that we might find useful even in today's world.

extension 15y ago
2b) It never lived, it was a "frozen corpse", and since you can make copies at will, just having a copy doesn't constitute a new life until it's run. How long does it have to run before it's alive? If the interval is short enough then there will be no change in state whatsoever. So in what way does the mind state have to change in order for it to have lived? What if that change is so simple that you can calculate it in your head? Have you created a new life just by thinking about what will happen next in the simulation? By writing it down? 3a) Since there's zero information difference, I'd say it is OK. So you would have no objection to being killed if there was an identical copy of you in existence somewhere? Why does that matter? Why do you care more about this copy of you than you do about anybody else? Why should a stranger care that there is at least one copy of you? 4c,d) Very interesting. On the one hand, in case of many modifications, you can't really foresee the consequences, so all should be considered unethical to prevent unintended suffering. But you are modifying your own mind all the time, by forming memories and learning skills. To some degree, you choose what you think and experience. Imagine that you have ultimate control over every facet of your mind. You live in a computer, so there is no "natural" mental life. You must choose exactly how your mind will progress. If you make a frozen copy of yourself, it hasn't diverged yet, so you can modify it however you like, since it's you. And if new, unique people are to be created at all, someone will have to choose how their mind works. There is no longer a "natural" reproductive process to decide this. How can choices be made in the new person's interests when it is those very interests that are being chosen?
- hastur 15y ago
  2b) I would assume that the machine on which the AI is run is digital (i.e. discrete), so yeah, a single cycle (tick of the processor) could very well mean change. And since you've got to draw the line somewhere, I'd say that's enough. 3a) By killing me and starting an identical copy of me somewhere else, you're creating a discontinuity that infringes upon my identity (or same-ness) because there would be a very strong discontinuity in inputs simply because of the different location. Moreover, a third party exercising control over any of my copies (and especially the one that is running) is in violation of my sovereignty. [That's also why a stranger should care about my copies.] 4c,d) That's precisely it: I am doing it! I can do with myself whatever I want. And yes, I do imagine having the control about every facet of my mind. I actually dream about it. :) Now, when I manipulate a copy of me, that's a little different, because as I've stated earlier, a copy spun off to run independently is sovereign from its "original". That's why manipulating own copy should morally be considered equivalent to manipulating a third party, assuming that copy will be let run separately. To your last point, I agree, that's the most delicate part - creating new beings. As far as copies are concerned, my immediate instinct is that only identical copies should be considered ethical, since any modifications can have unpredictable consequences. But I'm not sure this should be enforced by the AI community / data center, because I value personal sovereignty and habeas corpus virtualis above all else. ;) I won't speak of creating completely new, original beings, because that topic deserves a ton of thought and a philosophical treatise of its own. But theoretically, you could try to mimic natural processes of child brain development. Of course, as you've pointed out, someone has to choose the parameters of that developmental algorithm, just as a search engine is never really objective, because people wrote it and made some choices in the process. But that last example also shows we can have some trust in such choices for practical reasons. On a side note: I would highly recommend the book "Diaspora" by Greg Egan. The first chapter (Orphanogenesis) speaks about the development of a virtual mind and to me is among the best works of science fiction of all time.