I'd say that artilects, wholly artificial intellects built from first principles of cognition, are not where any anxiety should focus, since it looks vanishingl

I'd say that artilects, wholly artificial intellects built from first principles of cognition, are not where any anxiety should focus, since it looks vanishingly unlikely that we'll create any prior to the point of whole brain emulation. Whole brain emulation looks much more likely as a road to artificial intelligence; we'll start from there and tinker and edit our way to the construction of far greater intelligences. But it is worth thinking about what "tinker and edit" will mean for those entities involved, willingly and otherwise. Consider that at some point in the next few decades it will become possible to simulate and then emulate a human brain. That will enable related technological achievements as reverse engineering of memory, a wide range of brain-machine interfaces, and strong artificial intelligence. It will be possible to copy and alter an individual's mind: we are at root just data and operations on that data. It will be possible for a mind to run on computing hardware rather than in our present biology, for minds to be copied from a biological brain, and for arbitrary alterations of memory to be made near-immediately. This opens up all of the possibilities that have occupied science fiction writers for the past couple of decades: forking individuals, merging in memories from other forks, making backups, extending a human mind through commodity processing modules that provide skills or personality shards, and so on and so forth. There is already a population of folk who would cheerfully take on any or all of these options. I believe that this population will only grow: the economic advantages for someone who can edit, backup, and fork their own mind are enormous - let alone the ability to consistently take advantage of a marketplace of commodity products such as skills, personalities, or other fragments of the mind. But you'll notice I used what I regard as a malformed phrase there: "someone who can edit, backup, and fork their own mind." There are several sorts of people in the world; the first sort adhere to some form of pattern theory of identity, defining the self as a pattern, wherever that pattern may exists. Thus for these folk it makes sense to say that "my backup is me", or "my fork is me." The second sort, and I am in this camp, associate identity with the continuity of a slowly changing arrangement of mass and energy: I am this lump of flesh here, the one slowly shedding and rebuilding its cells and cellular components as it progresses. If you copy my mind and run it in software, that copy is not me. So in my view you cannot assign a single identity to forks and backups: every copy is an individual, large changes to the mind are equivalent to death, and it makes no sense to say something like "someone who can edit, backup, and fork their own mind." A copy of you is not you, but there is worse to consider: if the hardware that supports a running brain simulation is anything like present day computers, that copy isn't even particularly continuous. It is more like an ongoing set of individuals, each instantiated for a few milliseconds or less and then destroyed, to be replaced by yet another copy. If self is data associated with particular processing structures, such as an arrangement of neurons and their connections, then by comparison a simulation is absolute different: inside a modern computer or virtual machine that same data would be destroyed, changed, and copied at arbitrary times between physical structures - it is the illusion of a continuous entity, not the reality. That should inspire a certain sense of horror among folk in the continuity of identity camp, not just because it is an ugly thing to think about, but because it will almost certainly happen to many, many, many people before this century ends - and it will largely be by their own choice, or worse, inflicted upon them by the choice of the original from whom the copy was made. This is not even to think about the smaller third group of people who are fine with large, arbitrary changes to their state of mind: rewriting memories, changing the processing algorithms of the self, and so on. At the logical end of that road lie hives of software derived from human minds in which identity has given way to ever-changing assemblies of modules for specific tasks, things that transiently appear to be people but which are a different sort of entity altogether - one that has nothing we'd recognize as continuity of identity. Yet it would probably be very efficient and economically competitive. The existential threat here is that the economically better path to artificial minds, the one that involves lots of copying and next to no concern for continuity of identity, will be the one that dominates research and development. If successful and embedded in the cultural mainstream, it may squeeze out other roads that would lead to more robust agelessness for we biological humans - or more expensive and less efficient ways to build artificial brains that do have a continuity of structure and identity, such as a collection of artificial neurons that perform the same functions as natural ones. This would be a terrible, terrible tragedy: a culture whose tides are in favor of virtual, copied, altered, backed up and restored minds is to my eyes little different from the present culture that accepts and encourages death by aging. In both cases, personal survival requires research and development that goes against the mainstream, and thus proceeds more slowly. Sadly, given the inclinations of today's futurists - and, more importantly, the economic incentives involved - I see this future as far more likely than the alternatives. Given a way to copy, backup, and alter their own minds, people will use it and justify its use to themselves by adopting philosophies that state they are not in fact killing themselves over and again. I'd argue that they should be free to do so if they choose, just the same as I'd argue that anyone today should be free to determine the end of his or her life. Nonetheless, I suspect that this form of future culture may pose a sizable set of hurdles for those folk who emerge fresh from the decades in which the first early victories over degenerative aging take place.