12 ms·

It's interesting that you say that when we can clearly see trolley-problem style questions affecting real engineering decisions. For example, take self-driving

It's interesting that you say that when we can clearly see trolley-problem style questions affecting real engineering decisions. For example, take self-driving cars. We can take it as a given that at some point such a car will be in a state that it will likely collide with a bystander or else risk the lives of it's passengers. At that point, which lives should it prioritize? Or alternately, the question raised by an international study; if such a car were in a situation where it would either collide with a child or an elder, which outcome would be preferable? Interestingly, different cultures have overwhelmingly different answers to that question. Every moral event has particulars and context, but it's impossible to take in that context at the stage of designing systems, so we need a way of analyzing and reasoning about them in the abstract. Do you disagree with this?

coldtea 6y ago
>For example, take self-driving cars. We can take it as a given that at some point such a car will be in a state that it will likely collide with a bystander or else risk the lives of it's passengers. At that point, which lives should it prioritize? Those which the community will deem as acceptable, based on its historical ideas, tendencies, morals, etc (and adjusted to taste later on, e.g. when the victims pile up, or don't). And that will solidify into a set of laws. There wont be a pick based on some "perfect" abstract theoritical answer on this or that thought experiment.
- endominus 6y ago
  I know it's not your intent to present one, but I love moral puzzles and you have just generated a delicious one. The core of your comment is that the "decisions" of the car should be based on the community to which it is sold; i.e. a car sold in, say, France should prioritize saving the lives of women, while one sold in Saudi Arabia should save the lives of men (for information on preference data, see [0]). Further, that there should be laws mandating these cultural standards such that (I assume) it would be illegal to sell or manufacture cars not in accordance with said standards. Two scenarios immediately spring to mind. What if I do not agree with the standards, either communal or legal? For example, Burmese standards strongly prefer inaction of the car; what if I consider that wrong, and that there would likely be fewer deaths if the car was allowed more autonomy in its decision-making? Would I be morally culpable if I was the car's programmer and my boss asked me to make the car less responsive (i.e. in my opinion more likely to kill people) so that it could be sold in Burma? Other countries are more indifferent to the car trying to save more lives; would I be a murderer for designing a car for those markets? The second scenario is the more obvious one; your argument hinges on community-derived morals being superior to abstract, though-experiment morals. But that's not a given. There are communities that don't consider certain people as people. There are communities that tolerate infanticide and slavery. There are communities that, in short, have pretty objectively abhorrent moral philosophies. Sometimes those communities have legislative power in a nation. If we concede that the ethics of autonomous systems should be derived by the specific morality of particular groups in power, we will likely end up with some extremely harmful systems. This obviously goes beyond the initial question of self-driving cars into matters of surveillance, social control, AI-driven weaponry, and so on. I just find it important to ask whether you believe that as an engineer, you believe that your own moral beliefs should be completely irrelevant when designing a product - that the manufacturers of Agent Orange or Zyklon B carry no culpability for the use of those products. [0]: http://moralmachineresults.scalablecoop.org/ http://moralmachineresults.scalablecoop.org/
walleeee 6y ago
You've posed the trolley problem to someone explicitly refusing to answer it! I'm joking :') thanks for the thoughtful comment. I agree that we see utilitarian thought experiments influencing engineering decisions. But as you imply, there is no perfect answer to the trolley problem. Wander far beyond difficult-to-dispute trivia like "it's less bad for fewer people to die" and it rapidly becomes an exercise in justification rather than critique. Also, as you state, it's impossible to predict every possibility in an engineered system's deployment environment. We cannot preclude unknown unknowns. Imo this should motivate caution. Why should we develop technology so quickly and haphazardly if we don't understand its consequences? Uncritical utilitarianism can too easily collapse the is-ought distinction. "Hypothetically, what should we do if X?" tends to snowball, like a self-fulfilling prophecy. Technological forms in the present haunt the future insofar as they render their (always-contingent) existence permanent in the human mind. We plan cities around the automobile because it carries techno-cultural inertia, not because it best promotes human health and well-being. So I don't dispute your argument, taken out of context. Engineered systems require utility functions: this is inescapable, I agree. Groups of humans must come to some consensus on how to construct these, and different groups will reach different conclusions. I question the premise, and the long chain of assumptions that lead to it.
- endominus 6y ago
  I agree fully on the tendency for toy philosophical problems to devolve into bikeshedding and useless arguments over irrelevant particulars. I also agree that ignorance of unknown harms and second-order effects deeply harms the potential good that engineered systems can do. I worry, however, that selectively-applied caution leads to equally unfavorable results in the opposite direction; to whit, it encourages sticking to the status quo. It is often complex to the point of impossibility to predict the wider effects of technological or social changes. The requirement that we understand those consequences would, if applied consistently, basically halt any technological progress. As distasteful as it is to admit it, we learn much more quickly by failure than success, and often the quickest, cheapest, and most thorough way to understand the consequences of a technology is to implement it. In other words, while I agree that applying implementing new technology without sufficient caution is often disastrous, I make note of that camel's nose "sufficient." The cost of excessive caution is that certain ideas (cars, fossil fuel energy, destructive economic models, etc) are allowed to continue far past the point where we can implement better alternatives. The cost of trying to do that is some measure of failure; there will always be some cost, and some of that measured in human lives, with trying to improve the state of the world.

coldtea 6y ago

>For example, take self-driving cars. We can take it as a given that at some point such a car will be in a state that it will likely collide with a bystander or else risk the lives of it's passengers. At that point, which lives should it prioritize? Those which the community will deem as acceptable, based on its historical ideas, tendencies, morals, etc (and adjusted to taste later on, e.g. when the victims pile up, or don't). And that will solidify into a set of laws. There wont be a pick based on some "perfect" abstract theoritical answer on this or that thought experiment.

endominus 6y ago
I know it's not your intent to present one, but I love moral puzzles and you have just generated a delicious one. The core of your comment is that the "decisions" of the car should be based on the community to which it is sold; i.e. a car sold in, say, France should prioritize saving the lives of women, while one sold in Saudi Arabia should save the lives of men (for information on preference data, see [0]). Further, that there should be laws mandating these cultural standards such that (I assume) it would be illegal to sell or manufacture cars not in accordance with said standards. Two scenarios immediately spring to mind. What if I do not agree with the standards, either communal or legal? For example, Burmese standards strongly prefer inaction of the car; what if I consider that wrong, and that there would likely be fewer deaths if the car was allowed more autonomy in its decision-making? Would I be morally culpable if I was the car's programmer and my boss asked me to make the car less responsive (i.e. in my opinion more likely to kill people) so that it could be sold in Burma? Other countries are more indifferent to the car trying to save more lives; would I be a murderer for designing a car for those markets? The second scenario is the more obvious one; your argument hinges on community-derived morals being superior to abstract, though-experiment morals. But that's not a given. There are communities that don't consider certain people as people. There are communities that tolerate infanticide and slavery. There are communities that, in short, have pretty objectively abhorrent moral philosophies. Sometimes those communities have legislative power in a nation. If we concede that the ethics of autonomous systems should be derived by the specific morality of particular groups in power, we will likely end up with some extremely harmful systems. This obviously goes beyond the initial question of self-driving cars into matters of surveillance, social control, AI-driven weaponry, and so on. I just find it important to ask whether you believe that as an engineer, you believe that your own moral beliefs should be completely irrelevant when designing a product - that the manufacturers of Agent Orange or Zyklon B carry no culpability for the use of those products. [0]: http://moralmachineresults.scalablecoop.org/ http://moralmachineresults.scalablecoop.org/

walleeee 6y ago

You've posed the trolley problem to someone explicitly refusing to answer it! I'm joking :') thanks for the thoughtful comment. I agree that we see utilitarian thought experiments influencing engineering decisions. But as you imply, there is no perfect answer to the trolley problem. Wander far beyond difficult-to-dispute trivia like "it's less bad for fewer people to die" and it rapidly becomes an exercise in justification rather than critique. Also, as you state, it's impossible to predict every possibility in an engineered system's deployment environment. We cannot preclude unknown unknowns. Imo this should motivate caution. Why should we develop technology so quickly and haphazardly if we don't understand its consequences? Uncritical utilitarianism can too easily collapse the is-ought distinction. "Hypothetically, what should we do if X?" tends to snowball, like a self-fulfilling prophecy. Technological forms in the present haunt the future insofar as they render their (always-contingent) existence permanent in the human mind. We plan cities around the automobile because it carries techno-cultural inertia, not because it best promotes human health and well-being. So I don't dispute your argument, taken out of context. Engineered systems require utility functions: this is inescapable, I agree. Groups of humans must come to some consensus on how to construct these, and different groups will reach different conclusions. I question the premise, and the long chain of assumptions that lead to it.

endominus 6y ago
I agree fully on the tendency for toy philosophical problems to devolve into bikeshedding and useless arguments over irrelevant particulars. I also agree that ignorance of unknown harms and second-order effects deeply harms the potential good that engineered systems can do. I worry, however, that selectively-applied caution leads to equally unfavorable results in the opposite direction; to whit, it encourages sticking to the status quo. It is often complex to the point of impossibility to predict the wider effects of technological or social changes. The requirement that we understand those consequences would, if applied consistently, basically halt any technological progress. As distasteful as it is to admit it, we learn much more quickly by failure than success, and often the quickest, cheapest, and most thorough way to understand the consequences of a technology is to implement it. In other words, while I agree that applying implementing new technology without sufficient caution is often disastrous, I make note of that camel's nose "sufficient." The cost of excessive caution is that certain ideas (cars, fossil fuel energy, destructive economic models, etc) are allowed to continue far past the point where we can implement better alternatives. The cost of trying to do that is some measure of failure; there will always be some cost, and some of that measured in human lives, with trying to improve the state of the world.