11 ms·

In most complex systems the reality is people will encounter harm. The commercial airline industry is a good example of progress through crashing. In 2017 ther

In most complex systems the reality is people will encounter harm. The commercial airline industry is a good example of progress through crashing. In 2017 there were zero reported accidentally deaths [1]. Compared to 1972, one of the most dangerous years to fly, where more than 2,300 accidental deaths occurred on commercial flights. How did we go from thousands of deaths to zero? Primarily through crashing. Then working to evaluate what went wrong and improve the design, processes, and programs around commercial aviation. As one pilot explains [2]: "One of the ways we’ve become so safe was to realize that our efforts were never going to be good enough." So yes, I think technologies like autopilot will save lives. But it won't be able to do it magically, unfortunately some harm will occur because we—in all our human wisdom—just can never accurately predict all possibilities. It's the same problem most self-driving companies are facing today, and the reason promises of self-driving being here today have fallen short. The world is far, far more complex than we realize... especially when we put a computer out into it and tell it to "go forth." https://www.reuters.com/article/us-aviation-safety/2017-safest-year-on-record-for-commercial-passenger-air-travel-groups-idUSKBN1EQ17L https://www.reuters.com/article/us-aviation-safety/2017-safe... https://www.usatoday.com/story/travel/columnist/cox/2018/01/07/ask-captain-why-aviation-so-safe-2017/1005183001/ https://www.usatoday.com/story/travel/columnist/cox/2018/01/...

brosinante 7y ago
I guess this is how people justify killing for peace. Tesla stock holders must be celebrating that their cars get to run a few people over in the name of "progress". (Well ok, they get to celebrate because their failed pet project gets to have another go - potato/potato)
- everdev 7y ago
  If you find something that's pure good, let me know. There's always a price to pay with any activity. The question is just if the positive outweighs the negative. If you can't see both sides the you're probably polarized and biased to one approach. I think everyone wants better transportation and fewer deaths. But 0 deaths with an emerging driving technology sounds really hard. There are just so many edge cases and physics is pretty unforgiving on our bodies past 40mph.
- Diederich 7y ago
  > I guess this is how people justify killing for peace. Extreme but (hopefully) relevant counter-example: was the killing of tens of thousands of people (soldiers and civilians) by American citizens in Europe during World War II justified? This is a very complex question, and any reasonable answer will be even more complex and nuanced, so I'm not really asking anyone to directly answer it in this venue. To be clear, I'm not talking about Tesla here, only the first sentence in your response, in isolation. I think there are some circumstances where some deaths as payment for far fewer deaths that are morally and ethically sound and defensible. If we can agree to that, then we can move on and analyze what Tesla is trying to do.
  - Piskvorrr 7y ago
    Are we in a war with Machines now? Did Tesla state that it needs more Lebensraum, which, regrettably, requires purging the globe of everyone else? I'm not trying to invoke Godwin here: just that war happens be somewhat hard to compare - "people die, ergo same thing" seems a rather forced comparison. Perhaps the only fitting parallel seems "history will be written by the victors."
    - Piskvorrr 7y ago
      Also note how well all the other USian Wars On A Concept turned out...
  - brosinante 7y ago
    Of course there are, and you're absolutely correct in your example - fighting against war is a noble cause. But this is not a war in any way - there's already options in place to save lives - better policing, stricter speed limits, safer vehicles/roads/crossings. Allowing new technology on the road without the proper safeguards, (and the in my opinion abhorent justification that a few unlucky people might just get run over for the greater good) is not one of these options. The "dangerous self driving cars on the road" option is only helping the pockets of Tesla's investors, not the public.
  - modestmasse 7y ago
    Only on HN would somebody argue that Teslas killing people is just like the Allied powers fighting the Axis (who were engaged in genocide). You don't get to choose when we move on. You have a broken brain. Your opinions are worse than useless.
Someone1234 7y ago
If Tesla's AutoPilot was regulated like the airline industry with post-crash reviews, mandatory improvements, and open sharing of data we likely would see that. At the current time Tesla's system is proprietary/secret and their oversight is lax. If we look at the history of airline safety, going back further, we can see that it took many years and numerous crashes to get to the point where it became well regulated and crashes were independently investigated. Before government stepped in, private companies weren't doing a great job and their biases (inc. bad headlines/liability/cost) got in the way of improving passenger safety. So let's absolutely keep with the airline analogy and regulate Tesla's AutoPilot like the airline industry.
- xVedun 7y ago
  Hopefully it takes less than ~50 years which the airline industry took to get to that point. Seems that this would need to become a popular opinion before becoming reality.
- standyro 7y ago
  Regulation saves lives. We don't even have to look at the airline industry. Seat belts saved lives. Tailpipe emission controls saved lives. Tesla's Enhanced Autopilot and other Level 2 (partial autonomy) solutions will save lives.
  - dantheman 7y ago
    Not always - it has to be appropriate - there are cases where regulation costs lives by locking in suboptimal solutions because the costs to re-certify can be too high. We need to have the appropriate balance.
- scarejunba 7y ago
  The NTSB does actually investigate their stuff. And then it does issue recommends.
- mywacaday 7y ago
  I see a cost issue, easy to justify a full investigation every time a commercial plane crashes, how do you do a full investigation every time somebody crashes a car or is crashed into. How do you get the improvement loop that the airline industry had?
jeromegv 7y ago
Except the airline industry got there because of regulation and federal agencies investigating every single death (or even incidents, near crash without any deaths!) Right now, there isn't any equivalent in the car world.
- zaroth 7y ago
  I would like to introduce you to the National Highway Traffic Safety Administration (NHTSA) at https://www.nhtsa.gov/ https://www.nhtsa.gov/. Here's an example of one of their famous investigative reports on a Tesla Model S crash while operating under AutoPilot back in 2016: https://static.nhtsa.gov/odi/inv/2016/INCLA-PE16007-7876.PDF https://static.nhtsa.gov/odi/inv/2016/INCLA-PE16007-7876.PDF
deleted 7y ago
[deleted]
twic 7y ago
> How did we go from thousands of deaths to zero? Primarily through crashing. No, the reduction in deaths was through not crashing. The airline industry would not have got safer faster by deliberately doing more crashing. "Okay, we need to increase safety, so let's crash as much as we can!" is something they didn't say, because that wouldn't have made any sense. What you've done here, and the reason you've been upvoted to the top spot, is you've applied the Silicon Valley mantra of "failure is good" to airlines, and by extension, to autonomous cars. That mantra isn't really even true for software - it's not that failures are good, it's that learning is good - but it just about works in a world where people turn can turn free VC dollars into Medium posts and acqui-hires by failing, and hence where you might as well learn that way, rather than proceeding slowly with painstaking care; this approach doesn't really work in worlds where failures are underlined in blood.
- double0jimb0 7y ago
  I’d really like to see the details of the “proceeding slowly with painstaking care” looks like. Airplanes never had airbags, crumple zones, etc. The odds of surviving a car crash are much higher than odds of surviving a plane crash. It seems therefore that testing in cars can be done with lower risk of loss of life. Yes, there will be fender benders and some deaths.
  - mcguire 7y ago
    NASA manned space flight.
    - DuskStar 7y ago
      > NASA manned space flight. NASA is horrible at keeping astronauts alive. Worse than the Soviets/Russians, as far as I know. Apollo 1 and the two Shuttle disasters do not paint a good picture for NASA's safety in practice.
      - mcguire 7y ago
        NASA is great at keeping astronauts alive. It has no launch vehicles.
Piskvorrr 7y ago
Very interesting to not have chosen 2019, a.k.a. The Year Of 737 Max. #cherrypicking
Scoundreller 7y ago
The airline industry also « gives up » under a lot of conditions. Scenarios where a plane would refuse to takeoff or divest to another airport... are conditions where employers would still expect their employees to get to work, or their deliveries.
- ncallaway 7y ago
  I'm not interested in having a self-driving car until I don't have to pay attention to how it's driving. If I have to be alert and paying attention the entire time, then it doesn't provide a benefit. That said, I'm totally fine with a self-driving car that gives up when it can't handle something. As long as it can safely remove itself from traffic and come to a stop (e.g. pull over to the side of the road), then alert me that I need to take over, that's fine.
deleted 7y ago
[deleted]
mannykannot 7y ago
To be clear, the impossibility of perfection is no excuse for reckless behavior. For example, if your system cannot reliably detect stationary obstacles blocking its path, and you are relying on the driver to save the situation, you should be using all the available technology to ensure that the driver is, at all times, alert and paying attention to the road.
- zaroth 7y ago
  You have to appreciate that a system which increases your overall safety but that comes with so much overbearing annoyance that users refuse to use it is itself useless. If NHTSA mandates driver attention management systems in all cars then this calculus changes. If the driver attention management system is only required while AutoPilot is on, then the requirement is actually counter-productive to overall safety, e.g. results in increased deaths.
  - mannykannot 7y ago
    This reply ignores the very significant fact that Tesla's autopilot, and any other with similar capabilities, both enables and invites the driver to not pay attention to the extent that Tesla itself insists is necessary for the safe operation of the vehicle. If actually conforming to what Tesla says is necessary is too much of an annoyance, it is not ready to be on the road.

brosinante 7y ago

I guess this is how people justify killing for peace. Tesla stock holders must be celebrating that their cars get to run a few people over in the name of "progress". (Well ok, they get to celebrate because their failed pet project gets to have another go - potato/potato)

everdev 7y ago
If you find something that's pure good, let me know. There's always a price to pay with any activity. The question is just if the positive outweighs the negative. If you can't see both sides the you're probably polarized and biased to one approach. I think everyone wants better transportation and fewer deaths. But 0 deaths with an emerging driving technology sounds really hard. There are just so many edge cases and physics is pretty unforgiving on our bodies past 40mph.
Diederich 7y ago
> I guess this is how people justify killing for peace. Extreme but (hopefully) relevant counter-example: was the killing of tens of thousands of people (soldiers and civilians) by American citizens in Europe during World War II justified? This is a very complex question, and any reasonable answer will be even more complex and nuanced, so I'm not really asking anyone to directly answer it in this venue. To be clear, I'm not talking about Tesla here, only the first sentence in your response, in isolation. I think there are some circumstances where some deaths as payment for far fewer deaths that are morally and ethically sound and defensible. If we can agree to that, then we can move on and analyze what Tesla is trying to do.
- Piskvorrr 7y ago
  Are we in a war with Machines now? Did Tesla state that it needs more Lebensraum, which, regrettably, requires purging the globe of everyone else? I'm not trying to invoke Godwin here: just that war happens be somewhat hard to compare - "people die, ergo same thing" seems a rather forced comparison. Perhaps the only fitting parallel seems "history will be written by the victors."
  - Piskvorrr 7y ago
    Also note how well all the other USian Wars On A Concept turned out...
- brosinante 7y ago
  Of course there are, and you're absolutely correct in your example - fighting against war is a noble cause. But this is not a war in any way - there's already options in place to save lives - better policing, stricter speed limits, safer vehicles/roads/crossings. Allowing new technology on the road without the proper safeguards, (and the in my opinion abhorent justification that a few unlucky people might just get run over for the greater good) is not one of these options. The "dangerous self driving cars on the road" option is only helping the pockets of Tesla's investors, not the public.
- modestmasse 7y ago
  Only on HN would somebody argue that Teslas killing people is just like the Allied powers fighting the Axis (who were engaged in genocide). You don't get to choose when we move on. You have a broken brain. Your opinions are worse than useless.

Someone1234 7y ago

If Tesla's AutoPilot was regulated like the airline industry with post-crash reviews, mandatory improvements, and open sharing of data we likely would see that. At the current time Tesla's system is proprietary/secret and their oversight is lax. If we look at the history of airline safety, going back further, we can see that it took many years and numerous crashes to get to the point where it became well regulated and crashes were independently investigated. Before government stepped in, private companies weren't doing a great job and their biases (inc. bad headlines/liability/cost) got in the way of improving passenger safety. So let's absolutely keep with the airline analogy and regulate Tesla's AutoPilot like the airline industry.

xVedun 7y ago
Hopefully it takes less than ~50 years which the airline industry took to get to that point. Seems that this would need to become a popular opinion before becoming reality.
standyro 7y ago
Regulation saves lives. We don't even have to look at the airline industry. Seat belts saved lives. Tailpipe emission controls saved lives. Tesla's Enhanced Autopilot and other Level 2 (partial autonomy) solutions will save lives.
- dantheman 7y ago
  Not always - it has to be appropriate - there are cases where regulation costs lives by locking in suboptimal solutions because the costs to re-certify can be too high. We need to have the appropriate balance.
scarejunba 7y ago
The NTSB does actually investigate their stuff. And then it does issue recommends.
mywacaday 7y ago
I see a cost issue, easy to justify a full investigation every time a commercial plane crashes, how do you do a full investigation every time somebody crashes a car or is crashed into. How do you get the improvement loop that the airline industry had?

jeromegv 7y ago

Except the airline industry got there because of regulation and federal agencies investigating every single death (or even incidents, near crash without any deaths!) Right now, there isn't any equivalent in the car world.

zaroth 7y ago
I would like to introduce you to the National Highway Traffic Safety Administration (NHTSA) at https://www.nhtsa.gov/ https://www.nhtsa.gov/. Here's an example of one of their famous investigative reports on a Tesla Model S crash while operating under AutoPilot back in 2016: https://static.nhtsa.gov/odi/inv/2016/INCLA-PE16007-7876.PDF https://static.nhtsa.gov/odi/inv/2016/INCLA-PE16007-7876.PDF

deleted 7y ago

[deleted]

twic 7y ago

> How did we go from thousands of deaths to zero? Primarily through crashing. No, the reduction in deaths was through not crashing. The airline industry would not have got safer faster by deliberately doing more crashing. "Okay, we need to increase safety, so let's crash as much as we can!" is something they didn't say, because that wouldn't have made any sense. What you've done here, and the reason you've been upvoted to the top spot, is you've applied the Silicon Valley mantra of "failure is good" to airlines, and by extension, to autonomous cars. That mantra isn't really even true for software - it's not that failures are good, it's that learning is good - but it just about works in a world where people turn can turn free VC dollars into Medium posts and acqui-hires by failing, and hence where you might as well learn that way, rather than proceeding slowly with painstaking care; this approach doesn't really work in worlds where failures are underlined in blood.

double0jimb0 7y ago
I’d really like to see the details of the “proceeding slowly with painstaking care” looks like. Airplanes never had airbags, crumple zones, etc. The odds of surviving a car crash are much higher than odds of surviving a plane crash. It seems therefore that testing in cars can be done with lower risk of loss of life. Yes, there will be fender benders and some deaths.
- mcguire 7y ago
  NASA manned space flight.
  - DuskStar 7y ago
    > NASA manned space flight. NASA is horrible at keeping astronauts alive. Worse than the Soviets/Russians, as far as I know. Apollo 1 and the two Shuttle disasters do not paint a good picture for NASA's safety in practice.
    - mcguire 7y ago
      NASA is great at keeping astronauts alive. It has no launch vehicles.

Piskvorrr 7y ago

Very interesting to not have chosen 2019, a.k.a. The Year Of 737 Max. #cherrypicking

Scoundreller 7y ago

The airline industry also « gives up » under a lot of conditions. Scenarios where a plane would refuse to takeoff or divest to another airport... are conditions where employers would still expect their employees to get to work, or their deliveries.

ncallaway 7y ago
I'm not interested in having a self-driving car until I don't have to pay attention to how it's driving. If I have to be alert and paying attention the entire time, then it doesn't provide a benefit. That said, I'm totally fine with a self-driving car that gives up when it can't handle something. As long as it can safely remove itself from traffic and come to a stop (e.g. pull over to the side of the road), then alert me that I need to take over, that's fine.

deleted 7y ago

[deleted]

mannykannot 7y ago

To be clear, the impossibility of perfection is no excuse for reckless behavior. For example, if your system cannot reliably detect stationary obstacles blocking its path, and you are relying on the driver to save the situation, you should be using all the available technology to ensure that the driver is, at all times, alert and paying attention to the road.

zaroth 7y ago
You have to appreciate that a system which increases your overall safety but that comes with so much overbearing annoyance that users refuse to use it is itself useless. If NHTSA mandates driver attention management systems in all cars then this calculus changes. If the driver attention management system is only required while AutoPilot is on, then the requirement is actually counter-productive to overall safety, e.g. results in increased deaths.
- mannykannot 7y ago
  This reply ignores the very significant fact that Tesla's autopilot, and any other with similar capabilities, both enables and invites the driver to not pay attention to the extent that Tesla itself insists is necessary for the safe operation of the vehicle. If actually conforming to what Tesla says is necessary is too much of an annoyance, it is not ready to be on the road.