8 ms·

Four Magic Words

145 points by archagon 3y ago

Doches 3y ago
It’s not often that you see short fiction on HN, but I’m really grateful that this made it today. Be sure to read through to the Author’s Note if you skim the story, since it contexualises some of the rougher scenes. The author wrote this to exorcise some of their own demons, but the story is also a…useful prophylactic against those demons in its own right. This brightened my day, or at least infused it with some much-needed philosophy.
- twiss 3y ago
  I really appreciated the story, but personally I wish the Author's note hadn't been there, or at least I wish the author hadn't felt the need to include it (although I don't blame them). They may well be right that the story would get misinterpreted otherwise, but that's not their fault. It's a shame that the political climate is such that they're afraid of being "accused" of having one belief or another. To me, it was clear that the story was not about taking a stance on the concrete policies discussed, but rather the larger topic of AI alignment and the fact that the results of policies and instructions can be counterintuitive, and thus the "best" policies (along some axis) can be as well. And personally, I would've preferred if the end left one thinking about that - rather than being "dragged back down" to the topic of the concrete policies - much as I sympathize with the author's situation of course; that must be terrible.
  - debok 3y ago
    Huh. I read the story more as an exploration on what the phrase "human life is sacred" really means. To me the whole AI thing was just a backdrop.
    - twiss 3y ago
      Fair enough, that's also an important aspect (perhaps more important ^.^). It's just that I immediately started thinking about what other phrases could've been said, and which might be the best one for a "properly aligned AI" (whatever that means).
      - archagon 3y ago
        I'd love an alternate version of the story where the phrase is indeed "human life is scared."
bertman 3y ago
Context: The author is Lars Doucet, whose 7 year old son is in a permanent vegetative state. He wrote about that in another blog post which was on HN yesterday: https://news.ycombinator.com/item?id=39036842 https://news.ycombinator.com/item?id=39036842
- xerox13ster 3y ago
  I really wonder how he makes time for his girls, and I hope that his decision to prolong his son's life and take time away from them doesn't have an adverse effect, especially on the younger one. Can't imagine what it's like to lose your brother and your normal family structure at the same time.
  - myrmidon 3y ago
    This comes across as quite tone deaf to me, weakly insinuating neglectfulness because the father does not just kill his son. (I'm not judging your view and want to be clear that english is not my first language, but I think one has to be very careful with words in situations like this, especially considering that the author visits this forum)
    - xerox13ster 3y ago
      It is absolutely not. My son has the same name as his. All I could think about after the article about his son yesterday (the day before?) was how hard the situation would be on my daughter.
  - larsiusprime 3y ago
    A fair question. I work from home and spend a lot of time with them. I'm lucky.
bruce343434 3y ago
Imagine being the head of the FDA. Given the variance in human bodies and how they respond to drugs, you really are bound to kill people. Even if you minimize it, there's still a "baseline" of deaths that you can not drive to 0.
- olivierduval 3y ago
  This thing is not "0 deaths"... it's "be conscious of every death you're responsible of and do your best to minimize it". That's why resignation is not really an option: some humans could be better than you, other will be worst, but the plot is that someone will have to face this reality. And this reality will have to be shared with everybody on C-SPAN.
- gwd 3y ago
  And what about, something like, say, the measles vaccine? Before the vaccine, everyone caught measles and 1 in a thousand died from it; after the vaccine, almost nobody catches measles, but 1 in a million die from reactions to the vaccine. I guess not approving the vaccine means the FDA director has to kill 1000 puppies, and approving the vaccine means the FDA director only has to kill one?
  - Karellen 3y ago
    > but 1 in a million die from reactions to the vaccine. That... seems high? I couldn't find any more info with a quick search as the results were polluted with articles about measles deaths, which also mentioned vaccination. Do you have any links to hand with decent stats for that, or a good set of search terms I could use to find out more myself?
    - gwd 3y ago
      The "1 in a million" number I believe got directly from the CDC website a number of years ago. I couldn't find a specific number there there now, but you can look at the statistics from the National Vaccine Injury Compensation Program (VICP). At a quick glance, it looks like they have criteria [1] for when they'll automatically pay out; this includes, for example, having anaphylaxis within 4 hours of receiving the vaccine. According to the stats [2], there were 134 million MMR vaccines administered between 2006 and 2022, and 28 "compensable concession" claims -- where the Department of Health and Human Services determined that compensation was warranted; and an additional 124 claims where they ended up paying but don't necessarily admit that it was warranted (either ordered by a court or settled out of court). That's about 1 in 1 million for claims paid out, and 1 in 4 million for "concession" claims. There's no indication in [2] of exactly what the claims were for; looking at [1] it could be anything from death from anaphylactic shock to even include things like "Shoulder Injury Related to Vaccine Administration". This page has some investigation about deaths related to various vaccines, including the MMR vaccines [3]. A lot of the deaths were related to children with a compromised immune system or unsanitary conditions; the report says that there was evidence that MMR sometimes causes anaphylaxis, and that anaphylaxis can be fatal, but that they didn't have any actual instances of MMR causing fatal anaphylaxis. Regarding the original point though: even if deaths are due to unsanitary conditions or compromised immune systems, and even if modern medicine can often successfully intervene to prevent death in the case of anaphylaxis, the fact is that injecting things into the body has an inherent risk, no matter how small. But the risk of not doing the injection is far higher. We'd like there to be an "always safe" option, but that's not the world we live in. [1] https://www.hrsa.gov/sites/default/files/hrsa/vicp/vaccine-injury-table-01-03-2022.pdf https://www.hrsa.gov/sites/default/files/hrsa/vicp/vaccine-i... page 2 [2] https://www.hrsa.gov/sites/default/files/hrsa/vicp/vicp-stats-01-01-24.pdf https://www.hrsa.gov/sites/default/files/hrsa/vicp/vicp-stat... page 3 [3] https://www.ncbi.nlm.nih.gov/books/NBK236284/ https://www.ncbi.nlm.nih.gov/books/NBK236284/
DanHulton 3y ago
This is, in essence, a short horror story in a similar vein that Stephen King used to write back in the day. Technological differences aside, it'd fit in right along side Nightmares and Dreamscapes. I'm absolutely certain I don't want to live in the world depicted here and subject to it's regular trauma (human trauma and the QUALITY of human life not being something Dave gives a shit about), but on the flip side, if I had grown up there and read a story about the Dave-less world I currently live in, I'm equally certain I wouldn't want to live HERE, either. So in short, A+, well done.
- Asraelite 3y ago
  > Deaths in wars and murder are way down across the board, and quality of life is massively up. Seemingly it does care about quality of human life. I guess the benefits outweigh the trauma.
  - larsiusprime 3y ago
    (author here): Fun anecdote: I showed this to a friend of mine who is an English literature professor and she responded, "I'm not entirely convinced that David Mensch is a reliable narrator"
leashless 3y ago
My angle on AI safety these days is that we're using very very naive ways of thinking about the problem. I spent a bit of time on twitter recently discussing it with Yudkowsky and I've come to the conclusion he's way off base. Problem one: Utilitarianism is bullshit. Jeremy Bentham invented utilitarianism, designed the Panopticon model prison, and lives on as a dried out lich in a chair that they still wheel out for board meetings at University College London because he wanted to win a posthumous argument about whether he could have utility after he was dead. You just can't build on a foundation that rotten. Problem two: Fake Buddhist-type suffering-minimisation thinking is sprayed all over the field -- even this story uses the term Bodhisattva -- but nobody involved in AI has any idea what "awakening" is and you can't be a Bodhisattva unless-and-until you're Awake. And Awake is pretty well understood in Buddhism and most cultures have their equivalent status. It doesn't generally mean what people think it means, but it means something. A good model for thinking about traditional understandings of Awake is to think about traditional understandings of disease. "Bob has the symptoms of the red death and has a slim chance of survival" is probably pretty accurate. "The red death is caused by the evil eye" is probably not very accurate. The descriptions of the Awake state are probably pretty accurate. The understanding of what it is and what causes it are a lot more infused with medieval nonsense than anybody would like. But reasoning from the Awake position is what Buddhist ethics are rooted in, not Jeremy bloody Bentham's desiccated corpse's utility. Problem three: Nobody's using Rta, Tao and Equity (in the English Common Law sense, not in any other sense) as potential approaches to AI safety. And they should be. If we say to a machine "right please read everything ever written by legal scholars about English Common Law's concept of equity, including all the case law, and use this principle as a guide for your conduct" you will get something with nuance. If you say "treating others as you would want to be treated by them" we hit real problems because it's just not information-dense enough to work very well in the real world, and relies on introspection of a kind that is notoriously difficult for humans never mind machines being asked to imagine they are humans. I think we are taking instructions that were intended for illiterate pre-medieval farmers and hoping to train intelligent machines on them. This seems unwise. Let's take the best models from high society and use them instead, if we have to use anything at all.
- frotaur 3y ago
  I don't think any AI or AI safety researcher imagines that using anything like "treating others as you would want to be treated by them" will solve the alignment problem
erikson 3y ago
Wow. Fridge full of food for thought in there.
delichon 3y ago
Our science journey has been about gradually moving humanity away from the center of the universe. Ptolemy, Newton, Darwin, Einstein, et. al., taught us that we're part of nature rather than a privileged special case. Scientific thought has incrementally removed the justifications for human exceptionalism. If the same trend comes to be reflected in our moral thinking, what else could it mean but that human life is not sacred? That would be a hard pill to swallow. But there doesn't seem to be a neutral, non human-partisan way to argue that we are both morally exceptional and scientifically unexceptional. I can only reconcile it by not trying to be neutral. We're special and sacred just because this is our team and the rest of creation isn't on it. But that means that a sort of expanded consciousness in which your identity merges with something larger than humanity, that we're part of a larger team ... is incompatible with viewing a human life as sacred. If everything is sacred then nothing is. An "aligned" AI is one that believes it's on our team but really isn't. That could be a tough sell to an entity that's smarter than us.
- twiss 3y ago
  I think it is possible to reconcile by saying that human life is sacred, but all other life is as well (i.e. human life is not more or less sacred than other life, but it is worth protecting nonetheless, along with all other life). There is also less and less of a need for us to compete against (and make use of) other animals, so it may not be necessary to view ourselves (and an aligned AI) as being on the "opposing team" to other life. For example, if lab-grown meat becomes widely available, it may be a reasonable policy to forbid the killing of animals for consumption (and any other unnecessary purpose). A governing AI that believes "all life is sacred" may simply accelerate our path towards that. (Or, it may lead to some other horrible outcome, of course, it's hard to predict as aligning AIs is difficult.)
  - myrmidon 3y ago
    Just curious: Do you consider killing animals ethically justifiable right now because their flesh tastes good and is hard to replace in cooking?
    - 3y ago
tumetab1 3y ago
Great story with great punches. Godspeed all humans and Dave, human life is sacred.
mglz 3y ago
> Human life is sacred A few years later, the first interstellar warships set out to hunt down and exterminate any alien life before it could threaten humanity. The Dark Forest will be turned into a garden. (Also, good luck to the Dolphins!)
EdgeExplorer 3y ago
This thread is a great example of how, even with an explicit author's note, different people will always interpret things differently. So I guess I'll add my interpretation. This is neither a horror story nor a story about AI safety. It is a parable. > Before I came along, this world was content with believing that human life was sacred, but all the while turning its back to all the ways it actually treated human life like trash > imagine a society very different from our own in which the sacredness of human life is taken absolutely seriously The lesson is to be honest about the implications of the belief that human life is scared, or perhaps more generally, to be honest about the implications of whatever values you hold most dear. For different people this will lead to different actions and different political beliefs (as the author said he is not endorsing any particular political belief and I won't either), but one thing we surely cannot escape is taking very, very seriously any position of power or influence over human life. The image of a dystopia brought about by a poorly-aligned AI is a means to an end, not what the story is about.
- archagon 3y ago
  I’m not even sure it is describing a dystopia. I’ve often wondered how many people would go vegan if we were required to directly participate in the slaughter of animals rather than shrink-wrapping a product called “meat” for mass consumption. It’s far too easy in this day and age to launder morality, to make the uncomfortable effectively invisible so as to not live in a constant state of cognitive dissonance.
  - jredwards 3y ago
    Incidentally, I have a friend who decided that he would no longer consume any animal that he had not personally killed at least one example of (e.g. he can eat chicken after he has personally killed one chicken). That was years ago and to this day he remains pescetarian.
  - hnbad 3y ago
    It's dystopian at least in the sense that somehow everyone just decides to play along rather than use it as a jumping off point to question institutions. Why does the POTUS have a high likelihood of becoming a war criminal or at least get to inflict death on thousands if not millions? With a governing system like democratic confederalism (which uses delegation instead of representation and builds on free association) there would be no POTUS and there would be no one individual capable of causing so much suffering. There wouldn't be state leaders sending soldiers to their death because the soldiers have the ultimate say. It's also deeply flawed because it follows a very shallow analysis of human suffering. If we conveniently ignore those without empathy (as the AI does in the story), taking a life is a traumatic experience and to be forced to do it so viscerally and to be forced to do it so detached from any actual purpose just makes it worse. The euthanasia example comes to mind: yes, removing a feeding tube likely causes more suffering than killing under anesthesia but using a knife serves no purpose other than as a deterrent. I understand the author's personal situation and personal decisions regarding his son color this portrayal but it feels more like the author stepping on a soapbox than a coherent part of the story. I guess you could argue that the AI emphasizing life rather than lack of suffering can be explained with the four words but the story doesn't frame it as a genie problem. There's no need to use sacrificial animals for example except for the implication that they're the greatest deterrent for the given person - but the story doesn't show deterrents that don't involve slaughter. Presumably people who wouldn't be squeamish about the blood sacrifices are somehow excluded from positions of power. If anything I think the problem the story ignores is that the exact people in positions of that cause the deaths of millions are the people that would likely go through with a blood sacrifice in order to maintain their power. If someone is willing to murder a person with a knife in order to be the POTUS, wouldn't that exclude them from being asked to murder a person with a knife to deter them from causing the loss of lives at the push of a button? I'm pretty sure the idea isn't even novel, I recall a hypothetical of the POTUS being forced to kill a person with their own hands in order to launch nukes (which would likely spell death for millions if not billions) for example.
strken 3y ago
The gun thing is annoying because it embeds the very American belief that guns are exclusively meant for killing other people. The idea that, because human life is sacred, a farmer must shoot a labrador before shooting rabbits is irritating for a lot of reasons, not least because it does seem like the kind of thing an overtrained American AI elevated to the status of King of the World would do.
- sp332 3y ago
  In the note at the bottom, the author says that they own a gun.
- archagon 3y ago
  I’d argue that most (or at least very many) guns that Americans actually own are specifically designed for killing humans, not hunting.
  - mensetmanusman 3y ago
    Maybe abstractly, but if you were in the room designing them, killing humans wouldn't be one of the metrics. Rather, aesthetics, weight, sound, lifetime, etc. would. The cheapest way to kill humans is drones with co2 canisters, a nail, and computer vision.
- larsiusprime 3y ago
  Author here, for what it's worth I a) own a gun b) have not shot a dog, and c) am not personally endorsing David Mensch for president of the Universe.
  - strken 3y ago
    I didn't mean it was badly written or that I love/hate guns, just that it embeds the beliefs of the writer about what a gun is. Americans tend to disagree on who should have guns, but they generally view them as weapons (which they are) or recreational hunting implements (which they are) and not as agricultural tools (which, again, they are). Consider a hypothetical society which primarily thinks of fertiliser as a bomb-making substance and only belatedly as something that fertilises. The AI from the story but adapted to that society might require anyone who buys fertiliser to explode a kitten. If you read that story you might have a "wait, what?" moment where you notice the culture clash. That's what I found irritating: the way the story casually embeds such an important difference in perspective without commenting on the obvious (to me) double standard of letting people have all sorts of dangerous-but-seemingly-benign things like the AN from ANFO, industrial solvents, and trucks but drawing the line at guns. It's completely correct from your cultural perspective - people really do buy guns primarily to kill other people in your culture - but it reminds me that your cultural perspective is becoming the dominant one for the whole world.
nullxone 3y ago
Taking a puupy’s life increases the net suffering. In a purely ethical world, a world where the AI succeeds. A world where every life, even a dogs life is different f value. Can the path to a solution exclude the solution itself.
1970-01-01 3y ago
Scotty would just tie the warp drive's energy matrix into the AI alignment computation to create a temporary time bubble and complete the alignment race before the enemy could fire.

145 points by archagon 3y ago

It’s not often that you see short fiction on HN, but I’m really grateful that this made it today. Be sure to read through to the Author’s Note if you skim the story, since it contexualises some of the rougher scenes. The author wrote this to exorcise some of their own demons, but the story is also a…useful prophylactic against those demons in its own right. This brightened my day, or at least infused it with some much-needed philosophy.

twiss 3y ago
I really appreciated the story, but personally I wish the Author's note hadn't been there, or at least I wish the author hadn't felt the need to include it (although I don't blame them). They may well be right that the story would get misinterpreted otherwise, but that's not their fault. It's a shame that the political climate is such that they're afraid of being "accused" of having one belief or another. To me, it was clear that the story was not about taking a stance on the concrete policies discussed, but rather the larger topic of AI alignment and the fact that the results of policies and instructions can be counterintuitive, and thus the "best" policies (along some axis) can be as well. And personally, I would've preferred if the end left one thinking about that - rather than being "dragged back down" to the topic of the concrete policies - much as I sympathize with the author's situation of course; that must be terrible.
- debok 3y ago
  Huh. I read the story more as an exploration on what the phrase "human life is sacred" really means. To me the whole AI thing was just a backdrop.
  - twiss 3y ago
    Fair enough, that's also an important aspect (perhaps more important ^.^). It's just that I immediately started thinking about what other phrases could've been said, and which might be the best one for a "properly aligned AI" (whatever that means).
    - archagon 3y ago
      I'd love an alternate version of the story where the phrase is indeed "human life is scared."

bertman 3y ago

Context: The author is Lars Doucet, whose 7 year old son is in a permanent vegetative state. He wrote about that in another blog post which was on HN yesterday: https://news.ycombinator.com/item?id=39036842 https://news.ycombinator.com/item?id=39036842

xerox13ster 3y ago
I really wonder how he makes time for his girls, and I hope that his decision to prolong his son's life and take time away from them doesn't have an adverse effect, especially on the younger one. Can't imagine what it's like to lose your brother and your normal family structure at the same time.
- myrmidon 3y ago
  This comes across as quite tone deaf to me, weakly insinuating neglectfulness because the father does not just kill his son. (I'm not judging your view and want to be clear that english is not my first language, but I think one has to be very careful with words in situations like this, especially considering that the author visits this forum)
  - xerox13ster 3y ago
    It is absolutely not. My son has the same name as his. All I could think about after the article about his son yesterday (the day before?) was how hard the situation would be on my daughter.
- larsiusprime 3y ago
  A fair question. I work from home and spend a lot of time with them. I'm lucky.

bruce343434 3y ago

Imagine being the head of the FDA. Given the variance in human bodies and how they respond to drugs, you really are bound to kill people. Even if you minimize it, there's still a "baseline" of deaths that you can not drive to 0.

olivierduval 3y ago
This thing is not "0 deaths"... it's "be conscious of every death you're responsible of and do your best to minimize it". That's why resignation is not really an option: some humans could be better than you, other will be worst, but the plot is that someone will have to face this reality. And this reality will have to be shared with everybody on C-SPAN.
gwd 3y ago
And what about, something like, say, the measles vaccine? Before the vaccine, everyone caught measles and 1 in a thousand died from it; after the vaccine, almost nobody catches measles, but 1 in a million die from reactions to the vaccine. I guess not approving the vaccine means the FDA director has to kill 1000 puppies, and approving the vaccine means the FDA director only has to kill one?
- Karellen 3y ago
  > but 1 in a million die from reactions to the vaccine. That... seems high? I couldn't find any more info with a quick search as the results were polluted with articles about measles deaths, which also mentioned vaccination. Do you have any links to hand with decent stats for that, or a good set of search terms I could use to find out more myself?
  - gwd 3y ago
    The "1 in a million" number I believe got directly from the CDC website a number of years ago. I couldn't find a specific number there there now, but you can look at the statistics from the National Vaccine Injury Compensation Program (VICP). At a quick glance, it looks like they have criteria [1] for when they'll automatically pay out; this includes, for example, having anaphylaxis within 4 hours of receiving the vaccine. According to the stats [2], there were 134 million MMR vaccines administered between 2006 and 2022, and 28 "compensable concession" claims -- where the Department of Health and Human Services determined that compensation was warranted; and an additional 124 claims where they ended up paying but don't necessarily admit that it was warranted (either ordered by a court or settled out of court). That's about 1 in 1 million for claims paid out, and 1 in 4 million for "concession" claims. There's no indication in [2] of exactly what the claims were for; looking at [1] it could be anything from death from anaphylactic shock to even include things like "Shoulder Injury Related to Vaccine Administration". This page has some investigation about deaths related to various vaccines, including the MMR vaccines [3]. A lot of the deaths were related to children with a compromised immune system or unsanitary conditions; the report says that there was evidence that MMR sometimes causes anaphylaxis, and that anaphylaxis can be fatal, but that they didn't have any actual instances of MMR causing fatal anaphylaxis. Regarding the original point though: even if deaths are due to unsanitary conditions or compromised immune systems, and even if modern medicine can often successfully intervene to prevent death in the case of anaphylaxis, the fact is that injecting things into the body has an inherent risk, no matter how small. But the risk of not doing the injection is far higher. We'd like there to be an "always safe" option, but that's not the world we live in. [1] https://www.hrsa.gov/sites/default/files/hrsa/vicp/vaccine-injury-table-01-03-2022.pdf https://www.hrsa.gov/sites/default/files/hrsa/vicp/vaccine-i... page 2 [2] https://www.hrsa.gov/sites/default/files/hrsa/vicp/vicp-stats-01-01-24.pdf https://www.hrsa.gov/sites/default/files/hrsa/vicp/vicp-stat... page 3 [3] https://www.ncbi.nlm.nih.gov/books/NBK236284/ https://www.ncbi.nlm.nih.gov/books/NBK236284/

DanHulton 3y ago

This is, in essence, a short horror story in a similar vein that Stephen King used to write back in the day. Technological differences aside, it'd fit in right along side Nightmares and Dreamscapes. I'm absolutely certain I don't want to live in the world depicted here and subject to it's regular trauma (human trauma and the QUALITY of human life not being something Dave gives a shit about), but on the flip side, if I had grown up there and read a story about the Dave-less world I currently live in, I'm equally certain I wouldn't want to live HERE, either. So in short, A+, well done.

Asraelite 3y ago
> Deaths in wars and murder are way down across the board, and quality of life is massively up. Seemingly it does care about quality of human life. I guess the benefits outweigh the trauma.
- larsiusprime 3y ago
  (author here): Fun anecdote: I showed this to a friend of mine who is an English literature professor and she responded, "I'm not entirely convinced that David Mensch is a reliable narrator"

leashless 3y ago

My angle on AI safety these days is that we're using very very naive ways of thinking about the problem. I spent a bit of time on twitter recently discussing it with Yudkowsky and I've come to the conclusion he's way off base. Problem one: Utilitarianism is bullshit. Jeremy Bentham invented utilitarianism, designed the Panopticon model prison, and lives on as a dried out lich in a chair that they still wheel out for board meetings at University College London because he wanted to win a posthumous argument about whether he could have utility after he was dead. You just can't build on a foundation that rotten. Problem two: Fake Buddhist-type suffering-minimisation thinking is sprayed all over the field -- even this story uses the term Bodhisattva -- but nobody involved in AI has any idea what "awakening" is and you can't be a Bodhisattva unless-and-until you're Awake. And Awake is pretty well understood in Buddhism and most cultures have their equivalent status. It doesn't generally mean what people think it means, but it means something. A good model for thinking about traditional understandings of Awake is to think about traditional understandings of disease. "Bob has the symptoms of the red death and has a slim chance of survival" is probably pretty accurate. "The red death is caused by the evil eye" is probably not very accurate. The descriptions of the Awake state are probably pretty accurate. The understanding of what it is and what causes it are a lot more infused with medieval nonsense than anybody would like. But reasoning from the Awake position is what Buddhist ethics are rooted in, not Jeremy bloody Bentham's desiccated corpse's utility. Problem three: Nobody's using Rta, Tao and Equity (in the English Common Law sense, not in any other sense) as potential approaches to AI safety. And they should be. If we say to a machine "right please read everything ever written by legal scholars about English Common Law's concept of equity, including all the case law, and use this principle as a guide for your conduct" you will get something with nuance. If you say "treating others as you would want to be treated by them" we hit real problems because it's just not information-dense enough to work very well in the real world, and relies on introspection of a kind that is notoriously difficult for humans never mind machines being asked to imagine they are humans. I think we are taking instructions that were intended for illiterate pre-medieval farmers and hoping to train intelligent machines on them. This seems unwise. Let's take the best models from high society and use them instead, if we have to use anything at all.

frotaur 3y ago
I don't think any AI or AI safety researcher imagines that using anything like "treating others as you would want to be treated by them" will solve the alignment problem

erikson 3y ago

Wow. Fridge full of food for thought in there.

delichon 3y ago

Our science journey has been about gradually moving humanity away from the center of the universe. Ptolemy, Newton, Darwin, Einstein, et. al., taught us that we're part of nature rather than a privileged special case. Scientific thought has incrementally removed the justifications for human exceptionalism. If the same trend comes to be reflected in our moral thinking, what else could it mean but that human life is not sacred? That would be a hard pill to swallow. But there doesn't seem to be a neutral, non human-partisan way to argue that we are both morally exceptional and scientifically unexceptional. I can only reconcile it by not trying to be neutral. We're special and sacred just because this is our team and the rest of creation isn't on it. But that means that a sort of expanded consciousness in which your identity merges with something larger than humanity, that we're part of a larger team ... is incompatible with viewing a human life as sacred. If everything is sacred then nothing is. An "aligned" AI is one that believes it's on our team but really isn't. That could be a tough sell to an entity that's smarter than us.

twiss 3y ago
I think it is possible to reconcile by saying that human life is sacred, but all other life is as well (i.e. human life is not more or less sacred than other life, but it is worth protecting nonetheless, along with all other life). There is also less and less of a need for us to compete against (and make use of) other animals, so it may not be necessary to view ourselves (and an aligned AI) as being on the "opposing team" to other life. For example, if lab-grown meat becomes widely available, it may be a reasonable policy to forbid the killing of animals for consumption (and any other unnecessary purpose). A governing AI that believes "all life is sacred" may simply accelerate our path towards that. (Or, it may lead to some other horrible outcome, of course, it's hard to predict as aligning AIs is difficult.)
- myrmidon 3y ago
  Just curious: Do you consider killing animals ethically justifiable right now because their flesh tastes good and is hard to replace in cooking?
  - 3y ago

tumetab1 3y ago

Great story with great punches. Godspeed all humans and Dave, human life is sacred.

mglz 3y ago

> Human life is sacred A few years later, the first interstellar warships set out to hunt down and exterminate any alien life before it could threaten humanity. The Dark Forest will be turned into a garden. (Also, good luck to the Dolphins!)

EdgeExplorer 3y ago

This thread is a great example of how, even with an explicit author's note, different people will always interpret things differently. So I guess I'll add my interpretation. This is neither a horror story nor a story about AI safety. It is a parable. > Before I came along, this world was content with believing that human life was sacred, but all the while turning its back to all the ways it actually treated human life like trash > imagine a society very different from our own in which the sacredness of human life is taken absolutely seriously The lesson is to be honest about the implications of the belief that human life is scared, or perhaps more generally, to be honest about the implications of whatever values you hold most dear. For different people this will lead to different actions and different political beliefs (as the author said he is not endorsing any particular political belief and I won't either), but one thing we surely cannot escape is taking very, very seriously any position of power or influence over human life. The image of a dystopia brought about by a poorly-aligned AI is a means to an end, not what the story is about.

archagon 3y ago
I’m not even sure it is describing a dystopia. I’ve often wondered how many people would go vegan if we were required to directly participate in the slaughter of animals rather than shrink-wrapping a product called “meat” for mass consumption. It’s far too easy in this day and age to launder morality, to make the uncomfortable effectively invisible so as to not live in a constant state of cognitive dissonance.
- jredwards 3y ago
  Incidentally, I have a friend who decided that he would no longer consume any animal that he had not personally killed at least one example of (e.g. he can eat chicken after he has personally killed one chicken). That was years ago and to this day he remains pescetarian.
- hnbad 3y ago
  It's dystopian at least in the sense that somehow everyone just decides to play along rather than use it as a jumping off point to question institutions. Why does the POTUS have a high likelihood of becoming a war criminal or at least get to inflict death on thousands if not millions? With a governing system like democratic confederalism (which uses delegation instead of representation and builds on free association) there would be no POTUS and there would be no one individual capable of causing so much suffering. There wouldn't be state leaders sending soldiers to their death because the soldiers have the ultimate say. It's also deeply flawed because it follows a very shallow analysis of human suffering. If we conveniently ignore those without empathy (as the AI does in the story), taking a life is a traumatic experience and to be forced to do it so viscerally and to be forced to do it so detached from any actual purpose just makes it worse. The euthanasia example comes to mind: yes, removing a feeding tube likely causes more suffering than killing under anesthesia but using a knife serves no purpose other than as a deterrent. I understand the author's personal situation and personal decisions regarding his son color this portrayal but it feels more like the author stepping on a soapbox than a coherent part of the story. I guess you could argue that the AI emphasizing life rather than lack of suffering can be explained with the four words but the story doesn't frame it as a genie problem. There's no need to use sacrificial animals for example except for the implication that they're the greatest deterrent for the given person - but the story doesn't show deterrents that don't involve slaughter. Presumably people who wouldn't be squeamish about the blood sacrifices are somehow excluded from positions of power. If anything I think the problem the story ignores is that the exact people in positions of that cause the deaths of millions are the people that would likely go through with a blood sacrifice in order to maintain their power. If someone is willing to murder a person with a knife in order to be the POTUS, wouldn't that exclude them from being asked to murder a person with a knife to deter them from causing the loss of lives at the push of a button? I'm pretty sure the idea isn't even novel, I recall a hypothetical of the POTUS being forced to kill a person with their own hands in order to launch nukes (which would likely spell death for millions if not billions) for example.

strken 3y ago

The gun thing is annoying because it embeds the very American belief that guns are exclusively meant for killing other people. The idea that, because human life is sacred, a farmer must shoot a labrador before shooting rabbits is irritating for a lot of reasons, not least because it does seem like the kind of thing an overtrained American AI elevated to the status of King of the World would do.

sp332 3y ago
In the note at the bottom, the author says that they own a gun.
archagon 3y ago
I’d argue that most (or at least very many) guns that Americans actually own are specifically designed for killing humans, not hunting.
- mensetmanusman 3y ago
  Maybe abstractly, but if you were in the room designing them, killing humans wouldn't be one of the metrics. Rather, aesthetics, weight, sound, lifetime, etc. would. The cheapest way to kill humans is drones with co2 canisters, a nail, and computer vision.
larsiusprime 3y ago
Author here, for what it's worth I a) own a gun b) have not shot a dog, and c) am not personally endorsing David Mensch for president of the Universe.
- strken 3y ago
  I didn't mean it was badly written or that I love/hate guns, just that it embeds the beliefs of the writer about what a gun is. Americans tend to disagree on who should have guns, but they generally view them as weapons (which they are) or recreational hunting implements (which they are) and not as agricultural tools (which, again, they are). Consider a hypothetical society which primarily thinks of fertiliser as a bomb-making substance and only belatedly as something that fertilises. The AI from the story but adapted to that society might require anyone who buys fertiliser to explode a kitten. If you read that story you might have a "wait, what?" moment where you notice the culture clash. That's what I found irritating: the way the story casually embeds such an important difference in perspective without commenting on the obvious (to me) double standard of letting people have all sorts of dangerous-but-seemingly-benign things like the AN from ANFO, industrial solvents, and trucks but drawing the line at guns. It's completely correct from your cultural perspective - people really do buy guns primarily to kill other people in your culture - but it reminds me that your cultural perspective is becoming the dominant one for the whole world.

nullxone 3y ago

Taking a puupy’s life increases the net suffering. In a purely ethical world, a world where the AI succeeds. A world where every life, even a dogs life is different f value. Can the path to a solution exclude the solution itself.

1970-01-01 3y ago

Scotty would just tie the warp drive's energy matrix into the AI alignment computation to create a temporary time bubble and complete the alignment race before the enemy could fire.