I know it's not your intent to present one, but I love moral puzzles and you have just generated a delicious one. The core of your comment is that the "decisio

I know it's not your intent to present one, but I love moral puzzles and you have just generated a delicious one. The core of your comment is that the "decisions" of the car should be based on the community to which it is sold; i.e. a car sold in, say, France should prioritize saving the lives of women, while one sold in Saudi Arabia should save the lives of men (for information on preference data, see [0]). Further, that there should be laws mandating these cultural standards such that (I assume) it would be illegal to sell or manufacture cars not in accordance with said standards. Two scenarios immediately spring to mind. What if I do not agree with the standards, either communal or legal? For example, Burmese standards strongly prefer inaction of the car; what if I consider that wrong, and that there would likely be fewer deaths if the car was allowed more autonomy in its decision-making? Would I be morally culpable if I was the car's programmer and my boss asked me to make the car less responsive (i.e. in my opinion more likely to kill people) so that it could be sold in Burma? Other countries are more indifferent to the car trying to save more lives; would I be a murderer for designing a car for those markets? The second scenario is the more obvious one; your argument hinges on community-derived morals being superior to abstract, though-experiment morals. But that's not a given. There are communities that don't consider certain people as people. There are communities that tolerate infanticide and slavery. There are communities that, in short, have pretty objectively abhorrent moral philosophies. Sometimes those communities have legislative power in a nation. If we concede that the ethics of autonomous systems should be derived by the specific morality of particular groups in power, we will likely end up with some extremely harmful systems. This obviously goes beyond the initial question of self-driving cars into matters of surveillance, social control, AI-driven weaponry, and so on. I just find it important to ask whether you believe that as an engineer, you believe that your own moral beliefs should be completely irrelevant when designing a product - that the manufacturers of Agent Orange or Zyklon B carry no culpability for the use of those products. [0]: http://moralmachineresults.scalablecoop.org/ http://moralmachineresults.scalablecoop.org/