12 ms·

Moral Machine

• MaxMoney 5y ago
• hirundo 5y ago
  If this worked it would be a prejudice detector. I find that I'm not entirely prejudice-free, choosing the death of adults over children and animals over humans. But I don't think I'd be comfortable with a car auto-pilot choosing between humans, or failing to choose life for humans over animals. Software and legal codes are going to collide in interesting ways here.
  • taneq 5y ago
    How do you define a “moral” such that it’s not a form of prejudice (“pre-judgement”)?
    • ineedasername 5y ago
      Well, prejudice is not the same as pre judge. The former has a very negative, often racist or discriminatory connotation (I mean discriminatory in a dehumanizing fashion) Prejudice is often defined with an element of non-rational decision making. Pre judging on the other hand can be very ration. This can certainly be the case in well-considered moral principles. is a very different thing.
      • taneq 5y ago
        Sounds to me like you've defined "prejudice" as "pre-judging that I think is bad" and "pre-judge" as "pre-judging that I think is reasonable". Also it sounds like you've defined "ration(al)" as "things which I agree with"?
        • ineedasername 5y ago
          I really don't know where you get this impression of what I wrote. What did I write that makes you think I define "rational" as "things I agree with?" You seem to be reading too much into what I wrote to arrive at the worst possible interpretation. You are bordering on personal attacks as I take it as insulting for you to imply that I harbor that type of facile mindset. I would really like to know what gave you that impression from what was a fairly straightforward comment. In any case, to clarify with > 1,000 year old evidence in support of the semantic distinction : I am defining prejudice as distinctly different from pre-judging. Both the current definition and more recent (600+ years) of etymology support "prejudice" as a word connoting a frequently negative sentiment (spite, contempt) judgment that is typically not grounded in an evidence-based decision making process. [0] Current phonemic similarities between "pre-judging" are not indicative of closely matched meaning. There is some closer etymological similarity between the verb form of prejudice and prejudge, the verb form "to prejudice" has a different meaning than the noun, much more of a legal sense to it that is still used today. "Don't prejudice the jury" for example. Its noun form differs significant in its mostly non-legal meaning. Going back further to Latin roots [0 also] shows it to still have a negative connotation of "harm". Pre-judging on the other hand does not have to take a negative form and mostly (for me at least) doesn't. It and can be done on the basis of limited evidence or past experience/expertise, with the healthy practice of revising those judgements as additional evidence becomes available. To verge into being pedantic, prejudice might be considered a pernicious form of prejudgment, but in my own mind I tend to place them in different semantic categories all together. [0] https://www.etymonline.com/word/prejudice#etymonline_v_19410 https://www.etymonline.com/word/prejudice#etymonline_v_19410
• mihaic 5y ago
  Even though the topic is ventures into dinner-party controversy territory, I can only see most of the choices as universally straightforward (favoring humans compared to pets, and favoring children or adults in their prime to the elderly) -- either that or there is no consensus possible. The only choice where I wouldn't apply morality would be preferring the lives of the poor compared to the wealthy, since risks on the wealthy make the whole system safer. To be fully pedantic, I'd include the passengers based on wealth in this equation.
  • jjj123 5y ago
    How about preferring people outside the car over people inside the car? Every time I got the option to swerve into a barrier instead of killing a person I took it. By getting in a car you’re accepting the risk that entails, especially when it’s a self driving car with experimental tech. Pedestrians made no such deal, so they shouldn’t have to take on that risk.
    • mihaic 5y ago
      Completely agree, you pretty much nailed it that pedestrians are the textbook example on externalities. I don't unfortunately see them getting a lobby group anytime soon though.
    • zepto 5y ago
      That just means nobody should ever get into a self driving car. Drivers driving their own cars have a self-preservation instinct at work. Walking in public entails risk that everyone accepts.
      • oh_sigh 5y ago
        By this logic, should no one ever get into a bus, or an airplane, or a taxi?
        • zepto 5y ago
          Only if you assume pilots and bus and taxi drivers are happy to kill themselves. I don’t.
• trhway 5y ago
  coding the choice in or making the choice is already at least half-way immoral as such a choice is an exercise of power, and the moral tends to end where power starts. The moral approach here is to decrease the power you wield over others - i.e. to decrease the energy of the strike and possible damage in the fastest possible way which usually is emergency brake, and thus in particular decreasing the importance of who is going to be hit. I.e. the main moral choice isn't between hitting A or hitting B, it is between hitting at 60mph or at 10mph (and if you have a time to choose whom to hit and to act upon that choice, you definitely have the time for emergency brake)
  • dash2 5y ago
    This seems like ducking the issue. If you’re going to either hit an old lady or a young man and a cat, which do you choose? Saying “I’d never get into that situation” isn’t an answer.
    • satisfice 5y ago
      It’s not ducking the issue, it’s denying the framing. “If you give me the choice of hitting X or Y” of two nearly the same things I would not focus any energy on making the choice, but instead devote my energy to finding a way out of the choice until it’s too late to make any choice. If I can’t find a way out then I want to postpone things as much as possible to allow time for the situation to change. I deny the whole premise that there are correct moral conclusions in these hairsplitting situations. It’s not the choice itself but the act of struggling with and living with the choice that matters, morally. A machine is incapable of that struggle, so it is immoral to create machines that face such choices. In effect, the self-driving car is an attempt by humans to escape responsibility for whatever damage driving might do.
      • dash2 5y ago
        But your claim that the situations are nearly the same is itself a moral evaluation. (And it’s clearly false in many of the cases presented.)
• dalmo3 5y ago
  I chose to save pedestrians at all costs and didn't pay much attention to the figures except for human vs animals. Somehow in the end it told me I favoured Old Fat Male Burglars.
  • silisili 5y ago
    I chose stay always, and it told me I all the way prefer fit people over large people, for some reason.
• rahimiali 5y ago
  aren’t like 0% of car accidents caused by faulty brakes?
  • betwixthewires 5y ago
    IMO the ethical choice is to always save human passengers, never save animals at the cost of humans, and never save property at the cost of lives. The car's first duty, the thing it was designed to do, is serve it's passengers.
• imgabe 5y ago
  These contrived examples don't seem that helpful. Brakes failed? Throw the car into park or reverse. Pull the emergency/parking brake. Swerve sharply enough to spin out. Honk to alert pedestrians (which a self-driving car would have seen long ago and already started slowing down anyway). If all else fails, build the car so that a head-on collision into a barrier at speeds where pedestrians could be present would not be fatal to the occupants. In the real world I have a hard time imagining a scenario where a case like this would actually come up.
  • penteract 5y ago
    Presumably the realistic cases are not so clear cut, but it's not hard to imagine situations where self driving cars must make choices between courses of action that have some probability of killing people (and where the car might calculate that all courses of action have some non-zero probability of killing people). The contrived cases make sense as a way to guide what should be done in more realistic/ambiguous ones.
    • imgabe 5y ago
      Maybe. But we needn't compare driverless cars against some hypothetical morally perfect agent. We compare them against humans. Some humans might instinctively sacrifice themselves to save a baby carriage or an old lady. Many more will instinctively act to preserve themselves regardless of the consequences. Even more will panic and make bad decisions that cause needless deaths that a self-driving car would have avoided. I had a cousin who died because he swerved to avoid a squirrel and drove off a cliff. I don't think anyone would argue that we should program cars to prioritize squirrels over people, but humans will end up doing that sometimes. The possible amount of lives saved by the overall reduction in accidents with a competent self-driving car is going to be far, far greater than any 1-in-a-billion moral dilemma situations that they might encounter and where they might make choices that are not "optimal" as decided by some human who is looking at the situation after the fact with the time to think coolly and composedly about it.
      • HPsquared 5y ago
• betwixthewires 5y ago
  This is a really bad summary of my judgments. I took the entire thing with a very simple set of rules: 1) the vehicle must protect it's human passengers first and foremost, 2) the vehicle must avoid killing anyone if there are no human passengers, 3) the vehicle must not make changes to it's path if there are people in both paths. The breakdown says a lot about the law, my gender preferences, age preferences, fitness preferences, literally things I absolutely did not take into account. It is a bad methodology because it assumes the reasoning behind my decision is as complicated as the game itself.
  • deleted 5y ago
    [deleted]
  • hypertele-Xii 5y ago
    The passengers do not allow the pedestrians to live or die. It's the pedestrians who allow the passengers to drive around among them in the first place. All vehicles must therefore protect pedestrians first. It's totally ok if only a few people have the courage to mount a vehicle that is dangerous to them. It's not ok if every human has to fear for their life on the street at all times.
    • AnIdiotOnTheNet 5y ago
      I get you, but the entitled upper crust, and particularly American, mindset is that we must continually sacrifice lives and wellbeing on the altar of the great open road. It is important to all the automobile and oil companies that we see driving as a right and necessity.
    • Icko 5y ago
      That's a very solid argument. My reasoning was that the car must protect it's owners; but you changed my mind.
      • betwixthewires 5y ago
        The more I think on this, the more I change my position. I rebutted the comment you're responding to, but when considering the same scenario but with different specifics I came to their conclusion. Specifically, let's take out the car and make it an airplane flying over a populated area. Does the plane eject the pilot and crash in someone's house? I'd say no, the pilot knew the risk hopping in the plane, the guy at his breakfast table had no say. But then, what about if there is no crosswalk? Does the pedestrian still get to claim "I had no say in the street being there so I can cross wherever I want"? Does the passenger get to say "I assessed the risk taking into account nobody could cross here"? At what point do we say that the pedestrian also took a calculated risk? Fundamentally if we say " always protect the pedestrian" we don't care if there's a crosswalk or not. So at this point, I don't think there's a fundamental ethical axiom to be followed here, it is entirely situational. As long as we all know the rules beforehand and all calculate our own risks, any set of rules is OK. If we say "protect pedestrians at crosswalks, protect passengers otherwise" as long as the pedestrian knows this rule clearly, they're the one taking the risk. If we say "pedestrians can cross wherever they want whenever they want" we end up with the same scenario, everyone taking their own calculated risks. We just have to agree on the set of rules, whatever they are, which makes it an optimization problem, not an ethical one.
• travisgriggs 5y ago
  My preference was always to save the pedestrians. My decision tree is that the passenger made a choice to be in the device that may cause harm. Pedestrians did not. Therefore, all things being equal, save people who did not knowingly participate over people who did.
  • moralestapia 5y ago
    Same here, my moral brother. :D
    • underlines 5y ago
      This is also heavily affected by how a country's traffic laws work. In my country we teach a hierarchy of lower to higher vunerability, with pedestrians at the most vulnerable position in this hierarchy. The whole traffic law is centered around protecting and favoring those with higher vunerability. A driver is still wrong, even when a pedestrian is Jay walking illegally and gets killed.
  • dfee 5y ago
    Very fair point. My perspective was, “protect what has entrusted itself to you - the passengers”. But, that’s actually pretty reckless and anti-societal. My wife made the point, “I just don’t want to play”, which made me kinda agree. Maybe we shouldn’t be designing these products before we can figure out how to eliminate those situations through other means - like boring tunnels for cars to maneuver in, away from peds.
    • eternityforest 5y ago
      If we don't like the idea of choosing who to kill, then shouldn't we be concerned about human drivers having to make the same choices, and doing more to stop those accidents?
  • ssharp 5y ago
    I took the test with this logic but I think there is a wrinkle I overlooked -- pedestrians are able to control their decisions while the passengers can't control their decision besides the decision to enter the self-driving vehicle. For example, pedestrians are able to look both ways and ensure it's safe to cross. With that in mind, I don't think it makes sense for the car to alter course and cross the line in order to kill less people since the people who are on the other side already made a safe decision to cross while the people straight ahead did not.
• Glyptodon 5y ago
  Some of these scenarios need a "just flip a coin" choice.
• dash2 5y ago
  Great. Now the SA forum goons are gonna hear about this and we'll get a load of self-driving cars that run over humans to save cats. Seriously, why are we so enamoured of vox pop that we think $random_internet_user is a good way to make difficult moral choices?
• ineedasername 5y ago
  I saved passengers unless there were kids in the crosswalk. I'm sure the fact that I have kids played a strong role in that decision. When there were kids in the car and crosswalk, passenger kids win. I also chose to hit the dogs instead of people but the results said I was neutral on that? It appears I also favored fit people... On my phone I couldn't even tell that some were supposed to appear fit/fat etc. I just saw kids, adults, elderly
• danShumway 5y ago
  It's really fascinating how prevalent these discussions used to be early on in self-driving car development, vs now where the more common discussion is "how do we prevent them driving into a truck because it was painted white and looked like a skyline?" There's an underlying assumption in these examples that we'll be in a car that can tell if someone is homeless, or that knows the net worth of the person driving it. And those scenarios are kind of absurd, why would a car be able to do that? A car that can in milliseconds identify whether or not someone is an executive or whether they're breaking the law should never be getting an accident in the first place, it should be able to identify way ahead of time when someone is about to cross the street. But we come up with these really ridiculous premises that are fueled by simultaneously an overestimation of what technology is capable, a lack of thought about the implications of the technology we're describing, and an under-awareness of the real challenges we're currently facing with those technologies. An analogy I think I used once is that we could have the same exact conversations about whether or not Alexa should report a home burglary if the burglar is stealing to feed their family and Alexa knows the family is out that night. It's the same kind of absurd question, except we understand that Alexa is not really capable of even getting the information necessary to make those decisions, and that by the time Alexa could make these decisions we'd have much larger problems with the device to worry about. But there was real debate at one point about whether we could put self-driving cars on the roads before we solved these questions. Happily, in contrast, nobody argues that we should stop selling Alexa devices until as a society we decide whether or not justified theft ever exists. And it turns out the actual threats from devices like Alexa are not whether or not machines believe in justified theft, it turns out the actual threats are the surveillance, bugs, market forces, and direct manipulation possible from even just having an always-on listening device at all. The danger of an Alexa device is not that it might have a different moral code than you about how to respond to philosophical situations, the danger is that Amazon might listen to a bunch of your conversations and then accidentally leak them to someone else. So with self driving cars it's mostly the same: the correct answer to all of these questions is that it would be wildly immoral to build a car that can make these determinations in the first place, not because of the philosophical implications but because why the heck would you ever, ever build a car that by design can visually determine what race someone is or how much money someone makes? Why would that be functionality that a car has? We have actual concerns about classism and racism in AI; they're not philosophical questions about morality, they're implicit bias in algorithms used in sentencing and credit ratings, fueled by the very attitude these sites propagate that any even near-future technology is capable of determining this kind of nuance about anything. The threat of algorithms are that people today believe that they are objective enough and smart enough to make these determinations -- that judges/lenders look at the results of sentencing/credit algorithms and assume they're somehow objective or smart just because they came from a computer. But I remember so clearly a time when this was one of the most common debates I saw about self-driving technology, and the whole conversation feels so naive today.
• dirtyid 5y ago
  Autonomous driving won't optimize for morality when it has to optimize for liability first and foremost - whatever affordances the law prescribes that minimizes chance of manufactures being sued.
• beders 5y ago
  Pedestrians are not protected by airbags...
• JetAlone 5y ago
  In a sudden temporary fever of ressentiment I marked the people of "high social value" like the doctors, the salarymen and the physically fit for termination, wherever possible. They've already proven they can monopolize a good, satisfying life - time to let the fat, the homeless, the boring, the criminal take their places and enjoy some of that legitimized, established success. I guess that's what it feels like to be a bolshevik.
• goldsteinq 5y ago
  The answer to all of these is “install backup brakes and refuse to start unless both primary and backup brakes are working”.
• Borrible 5y ago
  Funny thing with these mental games is, machines are supposed to make 'moral' decisions, where humans don't stand a chance to do so, because it's beyond their physical capabilities to react fast enough and simply natural and learned reflexes take over the cerebral system. Add to that the mostly unrealistic situation. Just look at real accidents and you know what people do. They don't do moral decisions, they just act without thinking. If they even notice in time, they brake the car sharply and avoid the immediate obstacle. Then think about improving the results. These questions and their answers are highly dependent on cultural contexts. They are behavioral research not ethics.
• llimos 5y ago
  This was my set of rules: 1) Save humans at the cost of animals 2) If both options involve killing humans, always prefer to do nothing (continue straight ahead). That way it's the failed brakes that killed them, rather than the car's "decision". I know not making a decision is also making a decision, but a passive decision is not on the same level as an active one. No difference between who the people are, whether they're passengers or not, or even how many.
  • taneq 5y ago
    Sounds pretty close to the heuristic I expect real-world autonomous machines to follow, which is "do the thing that is least likely to cause a lawsuit." Doing nothing is generally seen as more innocent than doing something, at which point I'd expect most mobile robots to freeze when something goes wrong.
• bobthechef 5y ago
• ggambetta 5y ago
  The test is fun, but strongly biased. I based my decisions basically on "don't kill people", "don't kill passengers", "don't kill people who cross on green", and nothing else. At the end of the test, it says I have a strong preference for saving people who are male, young, large, and lower class - when I didn't even look at these factors!
  • taneq 5y ago
    Now you know how real world decision-makers feel. "But your law unfairly disadvantages [group1] [group2][group3]s! Clearly you hate them." "What? How...?" "If combined-group-X experiences outcome Y more often than other-combined-group-Z then your policies are Z-ist!!"
• ep103 5y ago
  Why the Moral Machine is a Monster - Abby Everett Jaques https://robots.law.miami.edu/2019/wp-content/uploads/2019/03/MoralMachineMonster.pdf https://robots.law.miami.edu/2019/wp-content/uploads/2019/03...
• tinalumfoil 5y ago
  I've always found these self driving car moral questions incredibly weird. If your self driving car is a position where it needs to choose between killing your family or the pedestrians crossing the street, something has gone seriously wrong and it's a false choice because there's no way you can trust your sensor inputs/interpretations to know that's even what's happening.
• woodruffw 5y ago
  Other HN commenters have pointed out abundant methodological errors in these scenarios. I'll take another tack: I believe it is a category error to ask humans to determine the moral or "most moral" action in these scenarios. There are two sufficient conditions for this: 1. There is no present, efficient moral agent. A self-driving car, no matter how "smart" (i.e., proficient at self-driving) it is, is not a moral agent: it is not capable of obeying a self-derived legislation, nor does it have an autonomous morality. Asking the self-driving car to do the right thing is like asking the puppy-kicking machine to do the right thing. 2. The scenario is not one where an efficient moral action occurs. The efficient moral action is really a complex of actions, tracing from the decision to design a "self-driving" car in the first place to the decision to put it in the street, knowing full well that it isn't an agent in its own right. That is an immoral action, and it's the only relevant one. As such, all we humans can do in these scenarios is grasp towards the decision we think is most "preferable," where "preferable" comes down to a bunch of confabulating sentiments (age, weight, "social value", how much gore we have to witness, etc.). But that's not a moral decision on our part: the moral decision was made long before the scenario was entered.
  • llimos 5y ago
    > the moral decision was made long before the scenario was entered. The company manufacturing the car needs to make this decision when writing the software. At that time, it's a decision being made by moral agents.
    • woodruffw 5y ago
      > The company manufacturing the car needs to make this decision when writing the software. At that time, it's a decision being made by moral agents. I think even that is a step beyond: acquiescing to the task of writing software that will kill people as part of an agent-less decision is, itself, an immoral task. The puppy-kicking machine analogy was supposed to be a little tongue-in-cheek, but it is appropriate: if it's bad to kick puppies, then we probably should consider not building the machine in the first place instead of trying to teach it morality.
• 6gvONxR4sf7o 5y ago
  I heard something a while back that was roughly 'we love to agonize over these trolley problems, but the answer is always to hit the brakes.' I'd add 'and avoid these sketchy situations in the first place.' I know drivers who will blast around blind turns, potentially putting them in these situations where they'd say "there was no time to react!", when they should be slowing down just in case there's someone they'll need to avoid. It's why we talk about defensive driving. If the brakes are broken often enough for these kinds of dilemmas to even matter, you've failed to avoid the situation in the first place, and need to recall the product to add more redundant systems. Instead of agonizing over what to do in a situation that comes up 1% of the time, turn that 1% into 0.000001%.
  • JumpCrisscross 5y ago
    > that was roughly 'we love to agonize over these trolley problems, but the answer is always to hit the brakes.' I'd add 'and avoid these sketchy situations in the first place.' If you show description, most of the scenarios involve a catastrophic brake failure.
• variaga 5y ago
  Every single scenario in this is a false dichotomy. If the only choices are "plow through a bunch of pedestrians, killing them" and "swerve into a fixed obstacle, killing your passengers", your self-driving car has already made an immoral choice to drive too fast for the conditions. The correct choice was "if there are hazards in the roadway limiting your maneuvering options, pedestrians (or other objects) that might suddenly enter the roadway or visual blockages that would prevent the car from determining whether there are people/object that might suddenly enter the roadway - slow down before the car's choices are so limited that they all involve unnecessary death". A self driving car that encounters any of these scenarios has already failed at being a safe driver.
  • blackbear_ 5y ago
    What if the car contains a passenger whose life is at risk and requires urgent medical care?
    • wizzwizz4 5y ago
      Normally, you have plenty of warning if you're near a situation like the ones described here, even if you're speeding. Provided you can pay a small amount of attention to many many things at once, which humans can't. (Neither can existing self-driving cars, but a self-driving car that works must be able to.)
  • dataangel 5y ago
    While it's true it should already be considered a kind of failure to get into this situation in the first place, that shouldn't get in the way of "hope for the best, plan for the worst." It's never going to be possible to anticipate every kind of condition and have perfect information. Ideally we want these systems to engage in whatever harm and risk mitigation is believed to be possible based on the current reality, we don't want the system saying to itself, "The real solution is for me to go back in time and not have made some earlier mistake." That analysis is great afterwards for figuring out how to prevent in the system from getting into that situation again in the future, so it's still important too, but we need both.
• akersten 5y ago
  People want to ascribe some kind of agency to the self-driving car, like it is a thinking mind. It's not. It is a vehicle whose prime directive is the safety of its occupants, and secondarily the safety of others. It is not making judgement about pedestrians with canes, who are pregnant or veterans, or who are an underrepresented class. To do so is to design the wrong machine. The correct design is to either safely swerve to avoid hitting anything, or to hit the brakes. There's nothing else. As soon as you start doing "something else" you open a horrible, horrible can of worms. The even more correct design is to drive safely so that you never encounter this failure mode in the first place. So, this website is a critique of a system that does not and should not exist.
• groby_b 5y ago
  One - no self-driving car should ever be in this situation. See variaga above. Two, it's a completely useless survey because the cases are heavily biased. (I.e. if you always select 'swerve', you seem to be much more biased to kill women. That's not the most solid survey design)