6 ms·

The problem is that the autopilot almost works, so drivers dont need to pay attention most of the time.

dmitrygr 7y ago
This is actually a phenomenon that is well studied (as it happens, by NTSB). Automation that is 99% reliable - kills! Less is unrelible enough that you notice and watch it carefully, more is reliable enough to actually rely on. It is like the uncanny valley of automation.
- rcMgD2BwE72F 7y ago
  >Automation that is 99% reliable kills! It also saves more lives than it kills people. So?
  - djannzjkzxn 7y ago
    Depending on the numbers, the “saves more than it kills” standard may be 99.9% or 99.9999%. You need data to answer the question and any claims should be backed up by numbers, followed by a bunch of analysis. It’s definitely not something we can prove about currently sold technology with a one-line quip.
  - falcolas 7y ago
    > It also saves more lives than it kills people. So? Do you have data that the rest of us don't - that is data which identifies incursions (that is near-crash interactions which do not end in a crash) where vehicular automation acts correctly and human drivers do not? If not, than you really can't make that statement. You can say that "Tesla's luxury cars running under autopilot show fewer crashes per mile than average for all cars", and be accurate, but that's not "saving lives".
  - LoSboccacc 7y ago
    it's not quite there yet, unless you believe at the stats published by Tesla that compares apples and oranges.
- spullara 7y ago
  Sadly, humans that are 99% reliable kill more.
- ajross 7y ago
  Is there a cite for that study? Everyone wants to scream about these things being dangerous, but we're still stuck in the noise floor. There are still only four Tesla autopilot fatalities over the 5.5 years it's been available, the last just shy of a year ago. And the number of cars sold (and presumably autopilot miles driven) has been rising rapidly during that period. I mean, seriously: if this technology "kills!" we really should have some statistics on that by now, right?
  - dmitrygr 7y ago
    Google "handoff latency". It is a measure of how long it takes a human suddenly dumped into a situation they were not previously paying much attention to, to become fully aware of it and act accordingly. In aviation this is measured in Tens of seconds. This is ok since in aviation you're far from everything when automation is on and thus have tense of seconds to act. With cars you may need a response in seconds or less in case of a disconnect. Humans suck at that. Some sources: https://www.google.com/search?q=autopilot+disconnect+airplane+time+human+response https://www.google.com/search?q=autopilot+disconnect+airplan...
    - deleted 7y ago
      [deleted]
pengaru 7y ago
It seems like there should already be an industry term for this, it's like the autonomous systems version of the "uncanny valley" [0]. https://en.wikipedia.org/wiki/Uncanny_valley https://en.wikipedia.org/wiki/Uncanny_valley
- missosoup 7y ago
  There is in aviation. It's called handoff latency. It's the time between autonomy giving up and asking for intervention, and the time when something bad might happen. In aviation this is measured in minutes, worst case tens of seconds. In cars, it can be sub-second and humanly impossible. This latency is why an autonomous car that works 95% of the time is more dangerous than driving manually. I haven't seen any indication that this problem is possible to solve until we get cars to 99.999%.
  - lukebuehler 7y ago
    Very interesting, I did not know that! Although, having flown planes myself, it makes sense. You can afford a few second of inattentiveness. Same with boats. But not in cars.
  - ghaff 7y ago
    If you haven't listened, it's worth checking out Children of the Magenta on the 99 Percent Invisible podcast. There's also a lot of interesting research from Missy Cummings' Humans and Autonomy Lab at Duke. https://hal.pratt.duke.edu/ https://hal.pratt.duke.edu/ She's been very critical of a lot of self-driving systems that mostly work but still require drivers to pay attention. (She's also a former US Navy fighter pilot.)
  - falcolas 7y ago
    Well, I think that one problem is that the automation is not giving up; there is not handoff occurring. It's reporting "all's well" until fractions of a second before the crash, if that. This means that it's up to the human to remain constantly vigilant and identify abnormal situations. Of course, when you consider that an overhead road sign and a stopped semi trailer "look" the same to the systems involved, it's not surprising that they think that everything's OK.

This is actually a phenomenon that is well studied (as it happens, by NTSB). Automation that is 99% reliable - kills! Less is unrelible enough that you notice and watch it carefully, more is reliable enough to actually rely on. It is like the uncanny valley of automation.

rcMgD2BwE72F 7y ago
>Automation that is 99% reliable kills! It also saves more lives than it kills people. So?
- djannzjkzxn 7y ago
  Depending on the numbers, the “saves more than it kills” standard may be 99.9% or 99.9999%. You need data to answer the question and any claims should be backed up by numbers, followed by a bunch of analysis. It’s definitely not something we can prove about currently sold technology with a one-line quip.
- falcolas 7y ago
  > It also saves more lives than it kills people. So? Do you have data that the rest of us don't - that is data which identifies incursions (that is near-crash interactions which do not end in a crash) where vehicular automation acts correctly and human drivers do not? If not, than you really can't make that statement. You can say that "Tesla's luxury cars running under autopilot show fewer crashes per mile than average for all cars", and be accurate, but that's not "saving lives".
- LoSboccacc 7y ago
  it's not quite there yet, unless you believe at the stats published by Tesla that compares apples and oranges.
spullara 7y ago
Sadly, humans that are 99% reliable kill more.
ajross 7y ago
Is there a cite for that study? Everyone wants to scream about these things being dangerous, but we're still stuck in the noise floor. There are still only four Tesla autopilot fatalities over the 5.5 years it's been available, the last just shy of a year ago. And the number of cars sold (and presumably autopilot miles driven) has been rising rapidly during that period. I mean, seriously: if this technology "kills!" we really should have some statistics on that by now, right?
- dmitrygr 7y ago
  Google "handoff latency". It is a measure of how long it takes a human suddenly dumped into a situation they were not previously paying much attention to, to become fully aware of it and act accordingly. In aviation this is measured in Tens of seconds. This is ok since in aviation you're far from everything when automation is on and thus have tense of seconds to act. With cars you may need a response in seconds or less in case of a disconnect. Humans suck at that. Some sources: https://www.google.com/search?q=autopilot+disconnect+airplane+time+human+response https://www.google.com/search?q=autopilot+disconnect+airplan...
  - deleted 7y ago
    [deleted]

pengaru 7y ago

It seems like there should already be an industry term for this, it's like the autonomous systems version of the "uncanny valley" [0]. https://en.wikipedia.org/wiki/Uncanny_valley https://en.wikipedia.org/wiki/Uncanny_valley

missosoup 7y ago
There is in aviation. It's called handoff latency. It's the time between autonomy giving up and asking for intervention, and the time when something bad might happen. In aviation this is measured in minutes, worst case tens of seconds. In cars, it can be sub-second and humanly impossible. This latency is why an autonomous car that works 95% of the time is more dangerous than driving manually. I haven't seen any indication that this problem is possible to solve until we get cars to 99.999%.
- lukebuehler 7y ago
  Very interesting, I did not know that! Although, having flown planes myself, it makes sense. You can afford a few second of inattentiveness. Same with boats. But not in cars.
- ghaff 7y ago
  If you haven't listened, it's worth checking out Children of the Magenta on the 99 Percent Invisible podcast. There's also a lot of interesting research from Missy Cummings' Humans and Autonomy Lab at Duke. https://hal.pratt.duke.edu/ https://hal.pratt.duke.edu/ She's been very critical of a lot of self-driving systems that mostly work but still require drivers to pay attention. (She's also a former US Navy fighter pilot.)
- falcolas 7y ago
  Well, I think that one problem is that the automation is not giving up; there is not handoff occurring. It's reporting "all's well" until fractions of a second before the crash, if that. This means that it's up to the human to remain constantly vigilant and identify abnormal situations. Of course, when you consider that an overhead road sign and a stopped semi trailer "look" the same to the systems involved, it's not surprising that they think that everything's OK.