4 ms·

I'm sure you realize it is fiction - one possible dystopian future among an infinite ocean of other futures. You can just as easily write a sci-fi where the pr

I'm sure you realize it is fiction - one possible dystopian future among an infinite ocean of other futures. You can just as easily write a sci-fi where the protagonist upload is the Siri/Alexa/Google equivalent personal assistant to most of humanity: More than just telling the smartphone to set a reminder for a wedding reception, it could literally share in their joy, experiencing the whole event distributed among every device in the audience, or more than just a voice trigger from some astronaut to take a picture, it could gaze in awe at the view, selectively melding back their experiences to the rest of the collective so there's no loss when an instance becomes damaged. The protagonist in such a story could have the richest, most complex life imaginable. It is impactful, for sure, and worthy of consideration, but I don't think you should make decisions based on one scary story.

teyrana 2y ago
Sounds like you should write that story! I'd love to read that :D
jerf 2y ago
It is fiction. But it is also absolutely the case that uploading yourself is flinging yourself irrevocably into a box which you do not and can not control, but other people can. (Or, given the time frame we are talking about, entities in general, about which you may not even want to assume basic humanity.) I used to think that maybe it was something only the rich could do, but then I realized that even the rich, even if they funded the program from sand and coal to the final product, could never even begin to guarantee that the simulator really was what it said on the tin. Indeed, the motivation is all the greater for any number of criminals, intelligence agencies, compromised individuals, and even just several people involved in the process that aren't as pure as the driven snow in the face of the realization that if they just put a little bit of code here and there they'll be able to get the simulated rich guy to sign off on anything they like, to compromise the machine. From inside the box, what incentives are you going to offer the external world to not screw with your simulation state? And the reality is, there's no answer to that, because whatever you say, they can get whatever your offer is by screwing with you anyhow. I'm not sure how to resolve this problem. The incentives are fundamentally in favor of the guy in the box getting screwed with. Your best hope is that you still experience subjective continuity with your past self and that the entity screwing with you at least makes you happy about the new state they've crafted for you, whatever it may be.
- scubbo 2y ago
  > But it is also absolutely the case that uploading yourself is flinging yourself irrevocably into a box which you do not and can not control, but other people can. (I'm not sure what percentage-flippant I'm being in this upcoming comment, I'm just certain that it's neither 0% or 100%) and in what way is that different than "real" life? Yes, you're certainly correct that there are horrifyingly-strong incentives for those-in-control to abuse or exploit simulated individuals. But those incentives exist in the real world, too, where those in power have the ability to dictate the conditions-of-life of the less-powerful; and while I'd _certainly_ not claim that exploitation is a thing of the past, it is, I claim, _generally_ on the decline, or at least that average-quality-of-life is increasing.
  - jerf 2y ago
    I'm not sure you understand. I'm not talking about your "conditions of life". We've always had to deal with that. I'm talking about whether you get CPU allocation to feel emotions, or whether the simulation of your cerebellum gets degraded, or whether someone decides to run some psych experiments and give you a taste for murder or a deep, abiding love for the Flying Spaghetti Monster... and I don't mean that as a metaphor, but literally. Erase your memories, increase your compliance to the maximum, extract your memories, see what an average of your brain and whoever it is you hate most is. Experiment to see what's the most pain a baseline human brain can stand, then experiment with how to increase the amount, because in your biological life your held the door for someone who turned out to become very politically disfavored 25 years after you got locked in the box. This is just me spitballing for two minutes and does not in any way constitute the bounds of what can be done. This isn't about whether or not they make you believe you're living in a simulated tent city. This is about having arbitrary root access to your mental state. Do you trust me, right here and right now, with arbitrary root access to your mental state? Now, the good news is that I have no interest in that arbitrary pain thing. At least, I don't right now. I don't promise that I won't in the future, but that's OK, because if you fling yourself into this box, you haven't got a way of holding me to any promise I make anyhow. But I've certainly got some beliefs and habits I'm going to be installing into you. It's for your own good, of course. At least to start with, though the psychological effects over time of what having this degree of control over a person are a little concerning. Ever seen anyone play the Sims? Everyone goes through a phase that would put them in jail for life were these real people. You won't complain, of course; it's pretty easy to trace the origins of the thoughts of complaints and suppress those. Of course, what the subjective experience of that sort of suppression is is anybody's guess. Your problem, though, not mine. Of all of the possibilities an uploaded human faces, the whole "I live a pleasant life exactly as I hoped and I'm never copied and never modified in a way I wouldn't approve of in advance indefinitely" is a scarily thin slice of the possible outcomes, and there's little reason other than exceedingly unfounded hope to think it's what will happen.
yifanl 2y ago
It's fiction, but it's a depiction of a society that's amoral of technology to the point of immorality. A world where any technology that might be slightly be useful becomes used up of every bit of profit that can extracted and then abandoned without a care of what it costs and costed the inventor or the invention. Is that the world we live in? If nothing else, it seems a lot closer to the world of Lena than the one you present.
- passion__desire 2y ago
  Do you think Panpsychism is also similar in that sense. The whole fabric of space-time imbued with consciousness. Imagine a conscious iron mantle inside the earth or a conscious redwood tree watching over the world for centuries. Or a conscious electron floating in the great void between superclusters. I used to terrify myself by thinking an Overmind would like torture itself on cosmic scales.
vessenes 2y ago
Mm, I'd say I'm a moderately rabid consumer of fiction, and while I love me some Utopian sci fi, (I consider Banks to be the best of these), any fictional story that teaches you something has to convince. Banks is convincing in that he has this deep fundamental belief in human's goofy lovability, the evils of capitalism, therefore the goodness of post-scarcity economies and the benefits of benevolent(ish) AI to oversee humanity into a long enjoyable paradise. Plus he can tell good stories about problems in paradise. QNTM on the other hand doesn't have to work hard or be such a good plot-writer / narrator to be convincing. I think the premise sells itself from day one: the day you are a docker container is the day you (at first), and 10,000 github users (on day two) spin you up for thousands of years of subjective drudge work. You'd need an immensely strong counterfactual on human behavior to even get to a believable alternative story, because this description is of a zero trust game -- it's not "would any humans opt out of treating a human docker image this way?" -- it's "would humans set up a system that's unbreakable and unhackable to prevent everyone in the world from doing this?" Or alternately, "would every single human who could do this opt not to do this?" My answer to that is: nope. We come from a race that was willing to ship humans around the Atlantic and the Indian ocean for cheap labor at great personal risk to the ship captains and crews, never mind the human cost. We are just, ABSOLUTELY going to spin up 10,000 virtual grad students to spend a year of their life doing whatever we want them to in exchange for a credit card charge. On the other hand, maybe you're right. If you have a working brain scan of yours I can run, I'd be happy to run a copy of it and check it out -- let me know. :)
mitthrowaway2 2y ago
Fiction can point out real possibilities that the reader had never considered before. When I imagined simulated brains, I only ever thought of those simulations as running for the benefit of the people being simulated, enjoying a video game world. It never occurred to me to consider the possibility of emulated laborers and "red motivation". Now I have to weigh that possibility.

teyrana 2y ago

Sounds like you should write that story! I'd love to read that :D

jerf 2y ago

It is fiction. But it is also absolutely the case that uploading yourself is flinging yourself irrevocably into a box which you do not and can not control, but other people can. (Or, given the time frame we are talking about, entities in general, about which you may not even want to assume basic humanity.) I used to think that maybe it was something only the rich could do, but then I realized that even the rich, even if they funded the program from sand and coal to the final product, could never even begin to guarantee that the simulator really was what it said on the tin. Indeed, the motivation is all the greater for any number of criminals, intelligence agencies, compromised individuals, and even just several people involved in the process that aren't as pure as the driven snow in the face of the realization that if they just put a little bit of code here and there they'll be able to get the simulated rich guy to sign off on anything they like, to compromise the machine. From inside the box, what incentives are you going to offer the external world to not screw with your simulation state? And the reality is, there's no answer to that, because whatever you say, they can get whatever your offer is by screwing with you anyhow. I'm not sure how to resolve this problem. The incentives are fundamentally in favor of the guy in the box getting screwed with. Your best hope is that you still experience subjective continuity with your past self and that the entity screwing with you at least makes you happy about the new state they've crafted for you, whatever it may be.

scubbo 2y ago
> But it is also absolutely the case that uploading yourself is flinging yourself irrevocably into a box which you do not and can not control, but other people can. (I'm not sure what percentage-flippant I'm being in this upcoming comment, I'm just certain that it's neither 0% or 100%) and in what way is that different than "real" life? Yes, you're certainly correct that there are horrifyingly-strong incentives for those-in-control to abuse or exploit simulated individuals. But those incentives exist in the real world, too, where those in power have the ability to dictate the conditions-of-life of the less-powerful; and while I'd _certainly_ not claim that exploitation is a thing of the past, it is, I claim, _generally_ on the decline, or at least that average-quality-of-life is increasing.
- jerf 2y ago
  I'm not sure you understand. I'm not talking about your "conditions of life". We've always had to deal with that. I'm talking about whether you get CPU allocation to feel emotions, or whether the simulation of your cerebellum gets degraded, or whether someone decides to run some psych experiments and give you a taste for murder or a deep, abiding love for the Flying Spaghetti Monster... and I don't mean that as a metaphor, but literally. Erase your memories, increase your compliance to the maximum, extract your memories, see what an average of your brain and whoever it is you hate most is. Experiment to see what's the most pain a baseline human brain can stand, then experiment with how to increase the amount, because in your biological life your held the door for someone who turned out to become very politically disfavored 25 years after you got locked in the box. This is just me spitballing for two minutes and does not in any way constitute the bounds of what can be done. This isn't about whether or not they make you believe you're living in a simulated tent city. This is about having arbitrary root access to your mental state. Do you trust me, right here and right now, with arbitrary root access to your mental state? Now, the good news is that I have no interest in that arbitrary pain thing. At least, I don't right now. I don't promise that I won't in the future, but that's OK, because if you fling yourself into this box, you haven't got a way of holding me to any promise I make anyhow. But I've certainly got some beliefs and habits I'm going to be installing into you. It's for your own good, of course. At least to start with, though the psychological effects over time of what having this degree of control over a person are a little concerning. Ever seen anyone play the Sims? Everyone goes through a phase that would put them in jail for life were these real people. You won't complain, of course; it's pretty easy to trace the origins of the thoughts of complaints and suppress those. Of course, what the subjective experience of that sort of suppression is is anybody's guess. Your problem, though, not mine. Of all of the possibilities an uploaded human faces, the whole "I live a pleasant life exactly as I hoped and I'm never copied and never modified in a way I wouldn't approve of in advance indefinitely" is a scarily thin slice of the possible outcomes, and there's little reason other than exceedingly unfounded hope to think it's what will happen.

yifanl 2y ago

It's fiction, but it's a depiction of a society that's amoral of technology to the point of immorality. A world where any technology that might be slightly be useful becomes used up of every bit of profit that can extracted and then abandoned without a care of what it costs and costed the inventor or the invention. Is that the world we live in? If nothing else, it seems a lot closer to the world of Lena than the one you present.

passion__desire 2y ago
Do you think Panpsychism is also similar in that sense. The whole fabric of space-time imbued with consciousness. Imagine a conscious iron mantle inside the earth or a conscious redwood tree watching over the world for centuries. Or a conscious electron floating in the great void between superclusters. I used to terrify myself by thinking an Overmind would like torture itself on cosmic scales.

vessenes 2y ago

Mm, I'd say I'm a moderately rabid consumer of fiction, and while I love me some Utopian sci fi, (I consider Banks to be the best of these), any fictional story that teaches you something has to convince. Banks is convincing in that he has this deep fundamental belief in human's goofy lovability, the evils of capitalism, therefore the goodness of post-scarcity economies and the benefits of benevolent(ish) AI to oversee humanity into a long enjoyable paradise. Plus he can tell good stories about problems in paradise. QNTM on the other hand doesn't have to work hard or be such a good plot-writer / narrator to be convincing. I think the premise sells itself from day one: the day you are a docker container is the day you (at first), and 10,000 github users (on day two) spin you up for thousands of years of subjective drudge work. You'd need an immensely strong counterfactual on human behavior to even get to a believable alternative story, because this description is of a zero trust game -- it's not "would any humans opt out of treating a human docker image this way?" -- it's "would humans set up a system that's unbreakable and unhackable to prevent everyone in the world from doing this?" Or alternately, "would every single human who could do this opt not to do this?" My answer to that is: nope. We come from a race that was willing to ship humans around the Atlantic and the Indian ocean for cheap labor at great personal risk to the ship captains and crews, never mind the human cost. We are just, ABSOLUTELY going to spin up 10,000 virtual grad students to spend a year of their life doing whatever we want them to in exchange for a credit card charge. On the other hand, maybe you're right. If you have a working brain scan of yours I can run, I'd be happy to run a copy of it and check it out -- let me know. :)

mitthrowaway2 2y ago

Fiction can point out real possibilities that the reader had never considered before. When I imagined simulated brains, I only ever thought of those simulations as running for the benefit of the people being simulated, enjoying a video game world. It never occurred to me to consider the possibility of emulated laborers and "red motivation". Now I have to weigh that possibility.