5 ms·

You've posed the trolley problem to someone explicitly refusing to answer it! I'm joking :') thanks for the thoughtful comment. I agree that we see utilitarian

You've posed the trolley problem to someone explicitly refusing to answer it! I'm joking :') thanks for the thoughtful comment. I agree that we see utilitarian thought experiments influencing engineering decisions. But as you imply, there is no perfect answer to the trolley problem. Wander far beyond difficult-to-dispute trivia like "it's less bad for fewer people to die" and it rapidly becomes an exercise in justification rather than critique. Also, as you state, it's impossible to predict every possibility in an engineered system's deployment environment. We cannot preclude unknown unknowns. Imo this should motivate caution. Why should we develop technology so quickly and haphazardly if we don't understand its consequences? Uncritical utilitarianism can too easily collapse the is-ought distinction. "Hypothetically, what should we do if X?" tends to snowball, like a self-fulfilling prophecy. Technological forms in the present haunt the future insofar as they render their (always-contingent) existence permanent in the human mind. We plan cities around the automobile because it carries techno-cultural inertia, not because it best promotes human health and well-being. So I don't dispute your argument, taken out of context. Engineered systems require utility functions: this is inescapable, I agree. Groups of humans must come to some consensus on how to construct these, and different groups will reach different conclusions. I question the premise, and the long chain of assumptions that lead to it.

endominus 6y ago
I agree fully on the tendency for toy philosophical problems to devolve into bikeshedding and useless arguments over irrelevant particulars. I also agree that ignorance of unknown harms and second-order effects deeply harms the potential good that engineered systems can do. I worry, however, that selectively-applied caution leads to equally unfavorable results in the opposite direction; to whit, it encourages sticking to the status quo. It is often complex to the point of impossibility to predict the wider effects of technological or social changes. The requirement that we understand those consequences would, if applied consistently, basically halt any technological progress. As distasteful as it is to admit it, we learn much more quickly by failure than success, and often the quickest, cheapest, and most thorough way to understand the consequences of a technology is to implement it. In other words, while I agree that applying implementing new technology without sufficient caution is often disastrous, I make note of that camel's nose "sufficient." The cost of excessive caution is that certain ideas (cars, fossil fuel energy, destructive economic models, etc) are allowed to continue far past the point where we can implement better alternatives. The cost of trying to do that is some measure of failure; there will always be some cost, and some of that measured in human lives, with trying to improve the state of the world.
- walleeee 6y ago
  > As distasteful as it is to admit it, we learn much more quickly by failure than success, and often the quickest, cheapest, and most thorough way to understand the consequences of a technology is to implement it. I totally agree. This is a fundamental paradox. As you state it's impossible to fully understand the consequences of any technological development. We can't learn without taking risks. > The cost of excessive caution is that certain ideas (cars, fossil fuel energy, destructive economic models, etc) are allowed to continue far past the point where we can implement better alternatives. I'm not sure about your use of "caution" here. I don't think fossil fuel extraction persists because we are too cautious (i.e., too risk-averse) to pursue alternatives. This would imply that biogeochemical realities are absent from our risk-benefit analysis, leading us to conclude burning carbon is less risky than other options. This may have been the case 50 years ago but "climate consciousness" is well within the mainstream now. I would argue instead that humans are uniquely skilled at ignoring unpleasant potentialities. This is socially useful: we coordinate our activities by genuflecting to fictions. This was a better strategy when we were incapable of terraforming the planet. Perhaps we can salvage it, but insulating ourselves from consequences as we seem so keen to do is a great way to walk off a cliff before the loss function kicks in. If you feed a system too-thoroughly sanitized data it will not form an accurate world model. > trying to improve the state of the world. I would submit that most existentially threatening technologies were born out of a desire to do just this. It seems the arc of development finds its way to entrenchment, if not the trenches, with little to no regard for inventors' intentions. As you say, perverse economic incentives don't help. I don't mean to condemn tool-making wholesale. But if we're going to build tools, imo they must be easy to dismantle and supersede. This principle is not so uncommon in software but the opposite tends to obtain more generally. An atmosphere full of carbon is not an easy thing to deconstruct, nor are stockpiles of nuclear weapons, or soils exhausted by years of improper care in pursuit of yield-maximization, or car-centric urban megalopoli.
  - endominus 6y ago
    >I'm not sure about your use of "caution" here. I was alluding to caution in reference to fossil fuels specifically because nuclear power causes less deaths per megawatt than any fossil fuel (and, interestingly, hydroelectric, if only because of the catastrophic failure of the Three Gorges Dam), and the extra scrutiny it receives in comparison is thus unjustified in my opinion. Over 100 people are killed in car crashes every day in the US alone, which helps put the potential value of self-driving cars in better perspective (although, of course, the switchover to self-driving cars opens us up to the possibility of malicious actors suborning the entire fleet; a worthwhile hypothetical to consider, I think you'll agree, given the... lackadaisical information security standards both IoT and vehicle manufacturers adhere to). The example of economic models is more speculative, but I could well imagine any potential new system being derided for being unproven or too risky even as our current one encourages and rewards looting the planet and poisoning the communal well. To your point on ignoring unpleasant potentialities, I couldn't agree more. The optimists have always led the way, and as long as there is more commons to translate into immediate success, it shall be so. We are living in a time of whalefall, with a surplus of resources sufficient to support a maximally extractive lifestyle for perhaps one generation more, before it is gone forever. I have grown increasingly disillusioned with our potential to avert the coming global ecological and subsequent economic catastrophe; it seems that our Neros would rather fiddle while Rome burns. >But if we're going to build tools, imo they must be easy to dismantle and supersede. An interesting point, and one that I never considered before. My assumption is generally that for physical goods, robustness and long lifetimes are strongly favored over upgradability. Or rather, that "ease of transition away from" is basically never considered in the design or proliferation of new tools. That's a difficult point; is a meme that provides a path to a better meme more adaptive than one that locks the target in? Can it be made so?

endominus 6y ago

I agree fully on the tendency for toy philosophical problems to devolve into bikeshedding and useless arguments over irrelevant particulars. I also agree that ignorance of unknown harms and second-order effects deeply harms the potential good that engineered systems can do. I worry, however, that selectively-applied caution leads to equally unfavorable results in the opposite direction; to whit, it encourages sticking to the status quo. It is often complex to the point of impossibility to predict the wider effects of technological or social changes. The requirement that we understand those consequences would, if applied consistently, basically halt any technological progress. As distasteful as it is to admit it, we learn much more quickly by failure than success, and often the quickest, cheapest, and most thorough way to understand the consequences of a technology is to implement it. In other words, while I agree that applying implementing new technology without sufficient caution is often disastrous, I make note of that camel's nose "sufficient." The cost of excessive caution is that certain ideas (cars, fossil fuel energy, destructive economic models, etc) are allowed to continue far past the point where we can implement better alternatives. The cost of trying to do that is some measure of failure; there will always be some cost, and some of that measured in human lives, with trying to improve the state of the world.

walleeee 6y ago
> As distasteful as it is to admit it, we learn much more quickly by failure than success, and often the quickest, cheapest, and most thorough way to understand the consequences of a technology is to implement it. I totally agree. This is a fundamental paradox. As you state it's impossible to fully understand the consequences of any technological development. We can't learn without taking risks. > The cost of excessive caution is that certain ideas (cars, fossil fuel energy, destructive economic models, etc) are allowed to continue far past the point where we can implement better alternatives. I'm not sure about your use of "caution" here. I don't think fossil fuel extraction persists because we are too cautious (i.e., too risk-averse) to pursue alternatives. This would imply that biogeochemical realities are absent from our risk-benefit analysis, leading us to conclude burning carbon is less risky than other options. This may have been the case 50 years ago but "climate consciousness" is well within the mainstream now. I would argue instead that humans are uniquely skilled at ignoring unpleasant potentialities. This is socially useful: we coordinate our activities by genuflecting to fictions. This was a better strategy when we were incapable of terraforming the planet. Perhaps we can salvage it, but insulating ourselves from consequences as we seem so keen to do is a great way to walk off a cliff before the loss function kicks in. If you feed a system too-thoroughly sanitized data it will not form an accurate world model. > trying to improve the state of the world. I would submit that most existentially threatening technologies were born out of a desire to do just this. It seems the arc of development finds its way to entrenchment, if not the trenches, with little to no regard for inventors' intentions. As you say, perverse economic incentives don't help. I don't mean to condemn tool-making wholesale. But if we're going to build tools, imo they must be easy to dismantle and supersede. This principle is not so uncommon in software but the opposite tends to obtain more generally. An atmosphere full of carbon is not an easy thing to deconstruct, nor are stockpiles of nuclear weapons, or soils exhausted by years of improper care in pursuit of yield-maximization, or car-centric urban megalopoli.
- endominus 6y ago
  >I'm not sure about your use of "caution" here. I was alluding to caution in reference to fossil fuels specifically because nuclear power causes less deaths per megawatt than any fossil fuel (and, interestingly, hydroelectric, if only because of the catastrophic failure of the Three Gorges Dam), and the extra scrutiny it receives in comparison is thus unjustified in my opinion. Over 100 people are killed in car crashes every day in the US alone, which helps put the potential value of self-driving cars in better perspective (although, of course, the switchover to self-driving cars opens us up to the possibility of malicious actors suborning the entire fleet; a worthwhile hypothetical to consider, I think you'll agree, given the... lackadaisical information security standards both IoT and vehicle manufacturers adhere to). The example of economic models is more speculative, but I could well imagine any potential new system being derided for being unproven or too risky even as our current one encourages and rewards looting the planet and poisoning the communal well. To your point on ignoring unpleasant potentialities, I couldn't agree more. The optimists have always led the way, and as long as there is more commons to translate into immediate success, it shall be so. We are living in a time of whalefall, with a surplus of resources sufficient to support a maximally extractive lifestyle for perhaps one generation more, before it is gone forever. I have grown increasingly disillusioned with our potential to avert the coming global ecological and subsequent economic catastrophe; it seems that our Neros would rather fiddle while Rome burns. >But if we're going to build tools, imo they must be easy to dismantle and supersede. An interesting point, and one that I never considered before. My assumption is generally that for physical goods, robustness and long lifetimes are strongly favored over upgradability. Or rather, that "ease of transition away from" is basically never considered in the design or proliferation of new tools. That's a difficult point; is a meme that provides a path to a better meme more adaptive than one that locks the target in? Can it be made so?