6 ms·

The author has also written a short horror story about simulated intelligence which I highly recommend: https://qntm.org/mmacevedo https://qntm.org/mmacevedo

crummy 2y ago
If you enjoyed this story, I cannot recommend enough the video game SOMA, which explores the concept very effectively from a first person perspective (which makes it all the more impactful).
vessenes 2y ago
Yess, that's a good one. It made me rethink my "sure I'd get scanned" plans, and put me in the "never allow my children to do that" camp. Extremely creepy.
- LeifCarrotson 2y ago
  I'm sure you realize it is fiction - one possible dystopian future among an infinite ocean of other futures. You can just as easily write a sci-fi where the protagonist upload is the Siri/Alexa/Google equivalent personal assistant to most of humanity: More than just telling the smartphone to set a reminder for a wedding reception, it could literally share in their joy, experiencing the whole event distributed among every device in the audience, or more than just a voice trigger from some astronaut to take a picture, it could gaze in awe at the view, selectively melding back their experiences to the rest of the collective so there's no loss when an instance becomes damaged. The protagonist in such a story could have the richest, most complex life imaginable. It is impactful, for sure, and worthy of consideration, but I don't think you should make decisions based on one scary story.
  - teyrana 2y ago
    Sounds like you should write that story! I'd love to read that :D
  - jerf 2y ago
    It is fiction. But it is also absolutely the case that uploading yourself is flinging yourself irrevocably into a box which you do not and can not control, but other people can. (Or, given the time frame we are talking about, entities in general, about which you may not even want to assume basic humanity.) I used to think that maybe it was something only the rich could do, but then I realized that even the rich, even if they funded the program from sand and coal to the final product, could never even begin to guarantee that the simulator really was what it said on the tin. Indeed, the motivation is all the greater for any number of criminals, intelligence agencies, compromised individuals, and even just several people involved in the process that aren't as pure as the driven snow in the face of the realization that if they just put a little bit of code here and there they'll be able to get the simulated rich guy to sign off on anything they like, to compromise the machine. From inside the box, what incentives are you going to offer the external world to not screw with your simulation state? And the reality is, there's no answer to that, because whatever you say, they can get whatever your offer is by screwing with you anyhow. I'm not sure how to resolve this problem. The incentives are fundamentally in favor of the guy in the box getting screwed with. Your best hope is that you still experience subjective continuity with your past self and that the entity screwing with you at least makes you happy about the new state they've crafted for you, whatever it may be.
    - scubbo 2y ago
      > But it is also absolutely the case that uploading yourself is flinging yourself irrevocably into a box which you do not and can not control, but other people can. (I'm not sure what percentage-flippant I'm being in this upcoming comment, I'm just certain that it's neither 0% or 100%) and in what way is that different than "real" life? Yes, you're certainly correct that there are horrifyingly-strong incentives for those-in-control to abuse or exploit simulated individuals. But those incentives exist in the real world, too, where those in power have the ability to dictate the conditions-of-life of the less-powerful; and while I'd _certainly_ not claim that exploitation is a thing of the past, it is, I claim, _generally_ on the decline, or at least that average-quality-of-life is increasing.
      - jerf 2y ago
  - yifanl 2y ago
    It's fiction, but it's a depiction of a society that's amoral of technology to the point of immorality. A world where any technology that might be slightly be useful becomes used up of every bit of profit that can extracted and then abandoned without a care of what it costs and costed the inventor or the invention. Is that the world we live in? If nothing else, it seems a lot closer to the world of Lena than the one you present.
    - passion__desire 2y ago
      Do you think Panpsychism is also similar in that sense. The whole fabric of space-time imbued with consciousness. Imagine a conscious iron mantle inside the earth or a conscious redwood tree watching over the world for centuries. Or a conscious electron floating in the great void between superclusters. I used to terrify myself by thinking an Overmind would like torture itself on cosmic scales.
  - vessenes 2y ago
    Mm, I'd say I'm a moderately rabid consumer of fiction, and while I love me some Utopian sci fi, (I consider Banks to be the best of these), any fictional story that teaches you something has to convince. Banks is convincing in that he has this deep fundamental belief in human's goofy lovability, the evils of capitalism, therefore the goodness of post-scarcity economies and the benefits of benevolent(ish) AI to oversee humanity into a long enjoyable paradise. Plus he can tell good stories about problems in paradise. QNTM on the other hand doesn't have to work hard or be such a good plot-writer / narrator to be convincing. I think the premise sells itself from day one: the day you are a docker container is the day you (at first), and 10,000 github users (on day two) spin you up for thousands of years of subjective drudge work. You'd need an immensely strong counterfactual on human behavior to even get to a believable alternative story, because this description is of a zero trust game -- it's not "would any humans opt out of treating a human docker image this way?" -- it's "would humans set up a system that's unbreakable and unhackable to prevent everyone in the world from doing this?" Or alternately, "would every single human who could do this opt not to do this?" My answer to that is: nope. We come from a race that was willing to ship humans around the Atlantic and the Indian ocean for cheap labor at great personal risk to the ship captains and crews, never mind the human cost. We are just, ABSOLUTELY going to spin up 10,000 virtual grad students to spend a year of their life doing whatever we want them to in exchange for a credit card charge. On the other hand, maybe you're right. If you have a working brain scan of yours I can run, I'd be happy to run a copy of it and check it out -- let me know. :)
  - mitthrowaway2 2y ago
    Fiction can point out real possibilities that the reader had never considered before. When I imagined simulated brains, I only ever thought of those simulations as running for the benefit of the people being simulated, enjoying a video game world. It never occurred to me to consider the possibility of emulated laborers and "red motivation". Now I have to weigh that possibility.
- sneak 2y ago
  What harm is there to the person so copied?
  - vessenes 2y ago
    Well you should read the story and find out some thoughts! QNTM refers to some people who think there's no harm, and some who do. It's short and great.
  - prepend 2y ago
    The harm is on the copies more so than the copied person.
ceejayoz 2y ago
Guessing that was based off https://en.wikipedia.org/wiki/Henrietta_Lacks https://en.wikipedia.org/wiki/Henrietta_Lacks a bit.
- stordoff 2y ago
  The author has said the title is a reference to the Lenna test image: https://en.wikipedia.org/wiki/Lenna https://en.wikipedia.org/wiki/Lenna. Possibly another influence though.
  - groby_b 2y ago
    I mean, the basic problem behind both is the same - taking without consent or compensation, and the entire field being OK with it. (And, in fact, happily leaning into it - even Playboy thought, hey, good for name recognition, we're not going to enforce our copyright)
    - TeMPOraL 2y ago
      Neither the test image nor the cell line are sentient, so they're nothing like MMAcedevo. Literally the one thing that's actually ethically significant about the latter does not exist in the former cases. Rights to information derived from someone is a boring first world problem of bickering about "lost revenue".
      - bee_rider 2y ago
        IIRC Lenna doesn’t want her picture used anymore because she was told it was making some young women in the field uncomfortable. I don’t think she’s complained about the revenue at all(?).
        throwanem 2y ago
        But not because the image itself is made to suffer by reuse. That it can't is why the comparison misses the point.
mmikeff 2y ago
Just read this last night! as part of the 'Valuable Humans in Transit and Other Stories' collection.
NameError 2y ago
I bought the short-story collection this is a part of and liked it a lot: https://qntm.org/vhitaos https://qntm.org/vhitaos A lot of the stories are free to read online without buying it but I thought the few dollars for the ebook was worth it
- jhbadger 2y ago
  And I really dig the cover design -- very much late 1960s "New Wave" SF vibes as if it were a collection of J.G. Ballard stories.
Ancapistani 2y ago
This story closely mirrors my (foggy) memory of “2012: The War for Souls” by Whitley Streiber. Without giving too much away, I recalled a specific story about a human consciousness being enslaved in a particular way, and ChatGPT confirmed that it was included in the book. I don’t think it is hallucinating, as it denied that similar stories I derived from that memory where in the book.
htk 2y ago
Reading mmacevedo was the only time that I actually felt dread related to AI. Excellent short story. Scarier in my opinion than the Roko's Basilisk theory that melted Yudkowsky's brain.
- digging 2y ago
  > Scarier in my opinion than the Roko's Basilisk theory that melted Yudkowsky's brain. Is that correct? I thought the Roko's Basilisk post was just seen as really stupid. Agreed that "Lena" is a great, chilling story though.
  - endtime 2y ago
    It's not correct. IIRC, Eliezer was mad that someone who thought they'd discovered a memetic hazard would be foolish enough to share it, and then his response to this unintentionally invoked the Streisand Effect. He didn't think it was a serious hazard. (Something something precommit to not cooperating with acausal blackmail)
    - deleted 2y ago
      [deleted]
    - wizzwizz4 2y ago
      > Something something precommit to not cooperating with acausal blackmail Acausal is a misnomer. It's atemporal, but TDT's atemporal blackmail requires common causation: namely, the mathematical truth "how would this agent behave in this circumstance?". So there's a simpler solution: be a human. Humans are incapable of simulating other agents simulating ourselves in the way that atemporal blackmail requires. Even if we were, we don't understand our thought processes well enough to instantiate our imagined AIs in software: we can't even write down a complete description of "that specific Roko's Basilisk you're imagining". The basic premises for TDT-style atemporal blackmail simply aren't there. The hypothetical future AI "being able to simulate you" is irrelevant. There needs to be a bidirectional causal link between that AI's algorithm, and your here-and-now decision-making process. You aren't actually simulating the AI, only imagining what might happen if it did, so any decision the future AI (is-the-sort-of-agent-that) makes does not affect your current decisions. Even if you built Roko's Basilisk as Roko specified it, it wouldn't choose to torture anyone. There is, of course, a stronger version of Roko's Basilisk, and one that's considerably older: evil Kantian ethics. See: any dictatorless dystopian society that harshly-punishes both deviance and non-punishment. There are plenty in fiction, though they don't seem to be all that stable in real life. (The obvious response to that idea is "don't set up a society that behaves that way".)
__MatrixMan__ 2y ago
I also like this one: https://qntm.org/responsibility https://qntm.org/responsibility
Chant-I-CRW 2y ago
This could just as easily be a history short from the Bobiverse.

crummy 2y ago

If you enjoyed this story, I cannot recommend enough the video game SOMA, which explores the concept very effectively from a first person perspective (which makes it all the more impactful).

vessenes 2y ago

Yess, that's a good one. It made me rethink my "sure I'd get scanned" plans, and put me in the "never allow my children to do that" camp. Extremely creepy.

LeifCarrotson 2y ago
I'm sure you realize it is fiction - one possible dystopian future among an infinite ocean of other futures. You can just as easily write a sci-fi where the protagonist upload is the Siri/Alexa/Google equivalent personal assistant to most of humanity: More than just telling the smartphone to set a reminder for a wedding reception, it could literally share in their joy, experiencing the whole event distributed among every device in the audience, or more than just a voice trigger from some astronaut to take a picture, it could gaze in awe at the view, selectively melding back their experiences to the rest of the collective so there's no loss when an instance becomes damaged. The protagonist in such a story could have the richest, most complex life imaginable. It is impactful, for sure, and worthy of consideration, but I don't think you should make decisions based on one scary story.
- teyrana 2y ago
  Sounds like you should write that story! I'd love to read that :D
- jerf 2y ago
  It is fiction. But it is also absolutely the case that uploading yourself is flinging yourself irrevocably into a box which you do not and can not control, but other people can. (Or, given the time frame we are talking about, entities in general, about which you may not even want to assume basic humanity.) I used to think that maybe it was something only the rich could do, but then I realized that even the rich, even if they funded the program from sand and coal to the final product, could never even begin to guarantee that the simulator really was what it said on the tin. Indeed, the motivation is all the greater for any number of criminals, intelligence agencies, compromised individuals, and even just several people involved in the process that aren't as pure as the driven snow in the face of the realization that if they just put a little bit of code here and there they'll be able to get the simulated rich guy to sign off on anything they like, to compromise the machine. From inside the box, what incentives are you going to offer the external world to not screw with your simulation state? And the reality is, there's no answer to that, because whatever you say, they can get whatever your offer is by screwing with you anyhow. I'm not sure how to resolve this problem. The incentives are fundamentally in favor of the guy in the box getting screwed with. Your best hope is that you still experience subjective continuity with your past self and that the entity screwing with you at least makes you happy about the new state they've crafted for you, whatever it may be.
  - scubbo 2y ago
    > But it is also absolutely the case that uploading yourself is flinging yourself irrevocably into a box which you do not and can not control, but other people can. (I'm not sure what percentage-flippant I'm being in this upcoming comment, I'm just certain that it's neither 0% or 100%) and in what way is that different than "real" life? Yes, you're certainly correct that there are horrifyingly-strong incentives for those-in-control to abuse or exploit simulated individuals. But those incentives exist in the real world, too, where those in power have the ability to dictate the conditions-of-life of the less-powerful; and while I'd _certainly_ not claim that exploitation is a thing of the past, it is, I claim, _generally_ on the decline, or at least that average-quality-of-life is increasing.
    - jerf 2y ago
- yifanl 2y ago
  It's fiction, but it's a depiction of a society that's amoral of technology to the point of immorality. A world where any technology that might be slightly be useful becomes used up of every bit of profit that can extracted and then abandoned without a care of what it costs and costed the inventor or the invention. Is that the world we live in? If nothing else, it seems a lot closer to the world of Lena than the one you present.
  - passion__desire 2y ago
    Do you think Panpsychism is also similar in that sense. The whole fabric of space-time imbued with consciousness. Imagine a conscious iron mantle inside the earth or a conscious redwood tree watching over the world for centuries. Or a conscious electron floating in the great void between superclusters. I used to terrify myself by thinking an Overmind would like torture itself on cosmic scales.
- vessenes 2y ago
  Mm, I'd say I'm a moderately rabid consumer of fiction, and while I love me some Utopian sci fi, (I consider Banks to be the best of these), any fictional story that teaches you something has to convince. Banks is convincing in that he has this deep fundamental belief in human's goofy lovability, the evils of capitalism, therefore the goodness of post-scarcity economies and the benefits of benevolent(ish) AI to oversee humanity into a long enjoyable paradise. Plus he can tell good stories about problems in paradise. QNTM on the other hand doesn't have to work hard or be such a good plot-writer / narrator to be convincing. I think the premise sells itself from day one: the day you are a docker container is the day you (at first), and 10,000 github users (on day two) spin you up for thousands of years of subjective drudge work. You'd need an immensely strong counterfactual on human behavior to even get to a believable alternative story, because this description is of a zero trust game -- it's not "would any humans opt out of treating a human docker image this way?" -- it's "would humans set up a system that's unbreakable and unhackable to prevent everyone in the world from doing this?" Or alternately, "would every single human who could do this opt not to do this?" My answer to that is: nope. We come from a race that was willing to ship humans around the Atlantic and the Indian ocean for cheap labor at great personal risk to the ship captains and crews, never mind the human cost. We are just, ABSOLUTELY going to spin up 10,000 virtual grad students to spend a year of their life doing whatever we want them to in exchange for a credit card charge. On the other hand, maybe you're right. If you have a working brain scan of yours I can run, I'd be happy to run a copy of it and check it out -- let me know. :)
- mitthrowaway2 2y ago
  Fiction can point out real possibilities that the reader had never considered before. When I imagined simulated brains, I only ever thought of those simulations as running for the benefit of the people being simulated, enjoying a video game world. It never occurred to me to consider the possibility of emulated laborers and "red motivation". Now I have to weigh that possibility.
sneak 2y ago
What harm is there to the person so copied?
- vessenes 2y ago
  Well you should read the story and find out some thoughts! QNTM refers to some people who think there's no harm, and some who do. It's short and great.
- prepend 2y ago
  The harm is on the copies more so than the copied person.

ceejayoz 2y ago

Guessing that was based off https://en.wikipedia.org/wiki/Henrietta_Lacks https://en.wikipedia.org/wiki/Henrietta_Lacks a bit.

stordoff 2y ago
The author has said the title is a reference to the Lenna test image: https://en.wikipedia.org/wiki/Lenna https://en.wikipedia.org/wiki/Lenna. Possibly another influence though.
- groby_b 2y ago
  I mean, the basic problem behind both is the same - taking without consent or compensation, and the entire field being OK with it. (And, in fact, happily leaning into it - even Playboy thought, hey, good for name recognition, we're not going to enforce our copyright)
  - TeMPOraL 2y ago
    Neither the test image nor the cell line are sentient, so they're nothing like MMAcedevo. Literally the one thing that's actually ethically significant about the latter does not exist in the former cases. Rights to information derived from someone is a boring first world problem of bickering about "lost revenue".
    - bee_rider 2y ago
      IIRC Lenna doesn’t want her picture used anymore because she was told it was making some young women in the field uncomfortable. I don’t think she’s complained about the revenue at all(?).
      - throwanem 2y ago
        But not because the image itself is made to suffer by reuse. That it can't is why the comparison misses the point.

mmikeff 2y ago

Just read this last night! as part of the 'Valuable Humans in Transit and Other Stories' collection.

NameError 2y ago

I bought the short-story collection this is a part of and liked it a lot: https://qntm.org/vhitaos https://qntm.org/vhitaos A lot of the stories are free to read online without buying it but I thought the few dollars for the ebook was worth it

jhbadger 2y ago
And I really dig the cover design -- very much late 1960s "New Wave" SF vibes as if it were a collection of J.G. Ballard stories.

Ancapistani 2y ago

This story closely mirrors my (foggy) memory of “2012: The War for Souls” by Whitley Streiber. Without giving too much away, I recalled a specific story about a human consciousness being enslaved in a particular way, and ChatGPT confirmed that it was included in the book. I don’t think it is hallucinating, as it denied that similar stories I derived from that memory where in the book.

htk 2y ago

Reading mmacevedo was the only time that I actually felt dread related to AI. Excellent short story. Scarier in my opinion than the Roko's Basilisk theory that melted Yudkowsky's brain.

digging 2y ago
> Scarier in my opinion than the Roko's Basilisk theory that melted Yudkowsky's brain. Is that correct? I thought the Roko's Basilisk post was just seen as really stupid. Agreed that "Lena" is a great, chilling story though.
- endtime 2y ago
  It's not correct. IIRC, Eliezer was mad that someone who thought they'd discovered a memetic hazard would be foolish enough to share it, and then his response to this unintentionally invoked the Streisand Effect. He didn't think it was a serious hazard. (Something something precommit to not cooperating with acausal blackmail)
  - deleted 2y ago
    [deleted]
  - wizzwizz4 2y ago
    > Something something precommit to not cooperating with acausal blackmail Acausal is a misnomer. It's atemporal, but TDT's atemporal blackmail requires common causation: namely, the mathematical truth "how would this agent behave in this circumstance?". So there's a simpler solution: be a human. Humans are incapable of simulating other agents simulating ourselves in the way that atemporal blackmail requires. Even if we were, we don't understand our thought processes well enough to instantiate our imagined AIs in software: we can't even write down a complete description of "that specific Roko's Basilisk you're imagining". The basic premises for TDT-style atemporal blackmail simply aren't there. The hypothetical future AI "being able to simulate you" is irrelevant. There needs to be a bidirectional causal link between that AI's algorithm, and your here-and-now decision-making process. You aren't actually simulating the AI, only imagining what might happen if it did, so any decision the future AI (is-the-sort-of-agent-that) makes does not affect your current decisions. Even if you built Roko's Basilisk as Roko specified it, it wouldn't choose to torture anyone. There is, of course, a stronger version of Roko's Basilisk, and one that's considerably older: evil Kantian ethics. See: any dictatorless dystopian society that harshly-punishes both deviance and non-punishment. There are plenty in fiction, though they don't seem to be all that stable in real life. (The obvious response to that idea is "don't set up a society that behaves that way".)

__MatrixMan__ 2y ago

I also like this one: https://qntm.org/responsibility https://qntm.org/responsibility

Chant-I-CRW 2y ago

This could just as easily be a history short from the Bobiverse.