6 ms·

I'm not sure I like the focus on the driver here. I do think he was a fool for trusting the device. It genuinely sounds like the autopilot failed to prevent a

I'm not sure I like the focus on the driver here. I do think he was a fool for trusting the device. It genuinely sounds like the autopilot failed to prevent a collision that, whether right or wrong, we would expect it to before broadly deploying these devices. A semi is a very large and pretty reasonably defined visual mass that intersected with the path of the vehicle. What did the autopilot do and when did it do it seems to be missing from this round of reports on this incident and thats pretty lame. Edit- Notice there is no mention of a COLLISION warning being issued? Only nagging "hands on wheel".

MBCook 9y ago
Despite the terrible name Tesla's system is supposed to assistive, the driver is still supposed to be in the loop. If the driver totally abdicated all involvement, the accident is his fault. He's the final safety system and it sounds like he didn't even try to do his job.
- CydeWeys 9y ago
  Names are important. Calling it "autopilot" implies that it does the driving for you, and people will take that at face value. It should be named in a way that accurately describes its abilities. If it is actually just assistive, and isn't autopilot, then calling it such is incredibly dangerous. It'd be like calling homeopathic junk a "cancer medicine", which is illegal mind you.
  - WhatIsDukkha 9y ago
    This is a case where bad word choice in marketing could literally kill people.
  - Robotbeat 9y ago
    "Autopilot" in an aircraft doesn't imply it flies for you. And in an aircraft, there are multiple levels. "Autopilot" is no worse than "cruise control" in this regard.
    - dragonwriter 9y ago
      “Autopilot” is worse, not because of what “autopilot” actually means in aviation, but because of what “autopilot” suggests to the car-buying public, who are, generally, not aviators.
      - hsod 9y ago
        What does the car-buying public think those folks in blue suits are doing at the front of the plane?
        neumann 9y ago
        Giving each other handskys!
        deleted 9y ago
        [deleted]
      - Robotbeat 9y ago
        No, I think many people just don't like the term and so decided everyone is confused by it. They're not. It's not self-driving, it's autopilot. That's the contrast they're drawing, and it actually IS clear to the car-buying public. A car is the second biggest single purchase you're going to make in your life, so people aren't going to /not/ figure out what the term means before buying the car.
    - exclusiv 9y ago
      If the market is confused about the term and it's not clear in practice - it's still not a good term to use. I definitely think it's worse than cruise control as you suggest. Cruise control systems control your cruising speed. Autopilot has way too many asterisks right now.
    - CydeWeys 9y ago
      Pilots require years of specialized training. Airplane autopilots have been around for decades and you aren't going to have an issue with unskilled pilots attempting to rely on them for things they can't handle based on a name alone. Also, airplane autopilots do fly the plane for you (including take-off and landing). They can certainly do so more safely than Tesla's "autopilot" does. A lot of this has to do with the aviation environment being much more controlled. This whole comment thread seems like a distraction in the form of a non sequitur, though. If Tesla is falsely claiming capabilities that their system does not actually possess right in the name, and that causes deaths, then they could easily be liable, and at a minimum they need to stop.
- LeifCarrotson 9y ago
  Agreed, with a couple caveats: 1. Any system that abstracts away human control bears some fault in failures of the human-machine system. Perhaps Tesla can get out of this legally through some fine print, or even ethically because the driver bears the final responsibility in the system. But they ought not design a system that is pleasant and convenient to use irresponsibly, which is what they have here. 2. Autopilot is supposed to handle issues like this. The Ars article says: > ...a Florida highway crash when the Tesla he was driving struck a tractor-trailer as the semi was crossing an intersection of a divided highway that did not have a traffic signal. So what if the intersection is unmarked. So what if the semi pulled out in front of him. So what if he ignored warnings. Autopilot should have seen the hazard and braked.
- cc439 9y ago
  That's also the fundamental problem with all existing forms of autonomous vehicle AI. Google was the furthest along with Level 4 and 5 autonomy but even they have backed off that development track as they couldn't muster the resources (a massive fleet collecting data) necessary to make the next leap in progress. Anything less than Level 4 autonomy will require constant driver attention. While there is technically Level 3 autonomy, the premise of a system which can operate with the operator intervening within some "limited span of time" in an emergency is fundamentally flawed. The situations in which an AI will have to kick the decision making back to the human driver are inherently time sensitive. Any sort of time-sensitive response from the driver would have to occur in such a short window that one could never safely remove their attention from the driving process for any meaningful amount of time. Even a time window as short as 5 seconds is too long to react to a situation at highway speeds (the car would have traveled 440 feet if it's at 60mph). My point is that this is an existential problem in the field of autonomous vehicles to a point that even the most ambitious players are admitting defeat. Most, if not all, the commonly discussed benefits of an autonomous car (all the things you can do with a comfy seat and an internet connection in the time one spent driving, also no worrying about DUIs) are unachievable with Level 0-3 technologies. They might make driving safer if they're implemented as a fail-safe for bad drivers but if they're allowed to take full command for any period of time, they are fundamentally unsafe and undesirable in terms of what the public is demanding from the promise of a truly driverless car.
- legolas2412 9y ago
  I am going to disagree there. Tesla names its system autopilot. Uses the fact that autopilot that includes collision avoidance reduces accidents by 40%, and then claims that there system is safer than a human (assisted human safer than a human, genius!). Elon musk has a statement every month about self driving autonomous driverless cars. They even have the gall to say that new cars have "full self driving hardware", is proof of outrageous statements even required? And all this hype/misleading advertising is fine because Tesla showed a few warnings? And the driver didn't have sufficient reason to believe the car was capable of complete autonomy. A simple Google search will tell you that this driver wasn't the only one.
deleted 9y ago
[deleted]
gitpusher 9y ago
You're saying this is a matter of "driver shaming"? Could be, especially given the stakes for Tesla. It would be helpful to know exactly what these warnings look/sound like, because as a previous poster noted, many drivers learn to ignore warnings if they're persistent, such as a check engine light (or "low washer fluid", which I receive in my BMW, and whose sound is identical to that for more serious issues.)
- fivestar 9y ago
  I think Tesla rolled it out too soon and marketed it as something it is not yet capable of and people died because of it.
- WhatIsDukkha 9y ago
  There is no mention of a "collision" warning is there? That is very disturbing and why I'm bothered by this round of reporting. The warnings seem to be just "hey you aren't supposed to hands off for this long" which, as others point out, are just the type of notifications that people start to ignore as they grow more comfortable with technologies.
- FooHentai 9y ago
  >Could be, especially given the stakes for Tesla. This is a really important thing to remain vigilant about - Tesla and other self-driving car developers have an absolute incentive to blame/excuse/deny when it comes to fault for any autonomous vehicle accident. My money is on us seeing a LOT of 'this dead driver was a total idiot' and 'this totally crazy thing happened that the autopilot could never have predicted and 100% of human drivers would have failed too'. The hard truth is a lot of people are going to die in the process of self-driving vehicles reaching maturity. This truth is not politically tenable for any party to admit. The real risk in this pattern of accidents followed by blame avoidance is that valuable lessons might not be learned from incidents. The aviation industry accident investigation process went through a very similar early life and many lives were lost unnecessarily before it matured into the robust, root-cause-driven, appropriately powerful system it is today. That was what it took for aviation safety to reach what I think most would agree is a suitable level of governance, and TBH I think autonomous vehicles are going to need to go through this same process and will hopefully get to a similar level of maturity without the loss of too many lives.
gshulegaard 9y ago
> What did the autopilot do and when did it do it seems to be missing from this round of reports on this incident and thats pretty lame. I am pretty sure this is because it has been reported on pretty thoroughly. https://www.tesla.com/blog/tragic-loss https://www.tesla.com/blog/tragic-loss https://www.nytimes.com/interactive/2016/07/01/business/inside-tesla-accident.html https://www.nytimes.com/interactive/2016/07/01/business/insi... The general sense is that the system is designed to apply brake when both radar and the camera system agree there was an obstacle (this may no longer be true since there was indication that decoupling was being worked on). It has been suggested that the white trailer against a bright blue sky didn't register on visual detection. Thereby there was not a consensus between radar and visual that there was an obstacle so the brake was not applied. > we would expect it to before broadly deploying these devices. And this is why there is focus on the driver. What is reasonable expectation? Are there adequate warnings and documentation about the capabilities of the system? Autopilot is not marketed as an autonomous driving system...it (at the moment) explicitly says drivers should continue to monitor the system to manually intervene if required. Further NHTSA research suggest as high as a 40% crash reduction since the wide introduction of Tesla's autopilot mode: https://techcrunch.com/2017/01/19/nhtsas-full-final-investigation-into-teslas-autopilot-shows-40-crash-rate-reduction/ https://techcrunch.com/2017/01/19/nhtsas-full-final-investig... This is a single accident...and arguably the driver was the final safety gate and they failed to apply corrective action (in addition to the automated safety systems). It does not appear as if the driver had sufficient reason to believe the system was capable of complete autonomous operation. And it does not appear as if there was a critical flaw or bug in the system that was putting consumers at risk. And just briefly going over the items in the roughly 500 page report there does appear to be a lot of detailed examination of the systems.
- WhatIsDukkha 9y ago
  Awesome reply, thanks. I'm actually even more bothered by this. This looks like a super common scenario people in cars face daily. I acknowledge the technical obstacles to "wide view" forward sensors but wow, this system seems very very limited. ""“The limited time the target is in the field of view prior to impact challenges the system’s ability to perform threat assessment and apply the CIB system. A target is usually recognized very late or not at all prior to impact.”""" It seems like the FOV of the sensors is not adequate. The system should be looking backward to assess the danger of a sudden braking? Sensor fusion should resolve ambiguity via probability... one line in the reporting suggested it needs both systems to "agree" which is a binary, this is probably just reporter confusion. This system doesn't seem ready for anyone younger then 21 full stop. If it doesn't radically improve it's going to have many more problems as it gets in the hands of a wider range of drivers beyond affluent people in middle age+. These systems are "lane keepers" and really the marketing and driver education should be tightly focused around what they DO NOT do. I genuinely look forward to less humans in control of cars but this is going to be a rough road.

MBCook 9y ago

Despite the terrible name Tesla's system is supposed to assistive, the driver is still supposed to be in the loop. If the driver totally abdicated all involvement, the accident is his fault. He's the final safety system and it sounds like he didn't even try to do his job.

CydeWeys 9y ago
Names are important. Calling it "autopilot" implies that it does the driving for you, and people will take that at face value. It should be named in a way that accurately describes its abilities. If it is actually just assistive, and isn't autopilot, then calling it such is incredibly dangerous. It'd be like calling homeopathic junk a "cancer medicine", which is illegal mind you.
- WhatIsDukkha 9y ago
  This is a case where bad word choice in marketing could literally kill people.
- Robotbeat 9y ago
  "Autopilot" in an aircraft doesn't imply it flies for you. And in an aircraft, there are multiple levels. "Autopilot" is no worse than "cruise control" in this regard.
  - dragonwriter 9y ago
    “Autopilot” is worse, not because of what “autopilot” actually means in aviation, but because of what “autopilot” suggests to the car-buying public, who are, generally, not aviators.
    - hsod 9y ago
      What does the car-buying public think those folks in blue suits are doing at the front of the plane?
      - neumann 9y ago
        Giving each other handskys!
      - deleted 9y ago
        [deleted]
    - Robotbeat 9y ago
      No, I think many people just don't like the term and so decided everyone is confused by it. They're not. It's not self-driving, it's autopilot. That's the contrast they're drawing, and it actually IS clear to the car-buying public. A car is the second biggest single purchase you're going to make in your life, so people aren't going to /not/ figure out what the term means before buying the car.
  - exclusiv 9y ago
    If the market is confused about the term and it's not clear in practice - it's still not a good term to use. I definitely think it's worse than cruise control as you suggest. Cruise control systems control your cruising speed. Autopilot has way too many asterisks right now.
  - CydeWeys 9y ago
    Pilots require years of specialized training. Airplane autopilots have been around for decades and you aren't going to have an issue with unskilled pilots attempting to rely on them for things they can't handle based on a name alone. Also, airplane autopilots do fly the plane for you (including take-off and landing). They can certainly do so more safely than Tesla's "autopilot" does. A lot of this has to do with the aviation environment being much more controlled. This whole comment thread seems like a distraction in the form of a non sequitur, though. If Tesla is falsely claiming capabilities that their system does not actually possess right in the name, and that causes deaths, then they could easily be liable, and at a minimum they need to stop.
LeifCarrotson 9y ago
Agreed, with a couple caveats: 1. Any system that abstracts away human control bears some fault in failures of the human-machine system. Perhaps Tesla can get out of this legally through some fine print, or even ethically because the driver bears the final responsibility in the system. But they ought not design a system that is pleasant and convenient to use irresponsibly, which is what they have here. 2. Autopilot is supposed to handle issues like this. The Ars article says: > ...a Florida highway crash when the Tesla he was driving struck a tractor-trailer as the semi was crossing an intersection of a divided highway that did not have a traffic signal. So what if the intersection is unmarked. So what if the semi pulled out in front of him. So what if he ignored warnings. Autopilot should have seen the hazard and braked.
cc439 9y ago
That's also the fundamental problem with all existing forms of autonomous vehicle AI. Google was the furthest along with Level 4 and 5 autonomy but even they have backed off that development track as they couldn't muster the resources (a massive fleet collecting data) necessary to make the next leap in progress. Anything less than Level 4 autonomy will require constant driver attention. While there is technically Level 3 autonomy, the premise of a system which can operate with the operator intervening within some "limited span of time" in an emergency is fundamentally flawed. The situations in which an AI will have to kick the decision making back to the human driver are inherently time sensitive. Any sort of time-sensitive response from the driver would have to occur in such a short window that one could never safely remove their attention from the driving process for any meaningful amount of time. Even a time window as short as 5 seconds is too long to react to a situation at highway speeds (the car would have traveled 440 feet if it's at 60mph). My point is that this is an existential problem in the field of autonomous vehicles to a point that even the most ambitious players are admitting defeat. Most, if not all, the commonly discussed benefits of an autonomous car (all the things you can do with a comfy seat and an internet connection in the time one spent driving, also no worrying about DUIs) are unachievable with Level 0-3 technologies. They might make driving safer if they're implemented as a fail-safe for bad drivers but if they're allowed to take full command for any period of time, they are fundamentally unsafe and undesirable in terms of what the public is demanding from the promise of a truly driverless car.
legolas2412 9y ago
I am going to disagree there. Tesla names its system autopilot. Uses the fact that autopilot that includes collision avoidance reduces accidents by 40%, and then claims that there system is safer than a human (assisted human safer than a human, genius!). Elon musk has a statement every month about self driving autonomous driverless cars. They even have the gall to say that new cars have "full self driving hardware", is proof of outrageous statements even required? And all this hype/misleading advertising is fine because Tesla showed a few warnings? And the driver didn't have sufficient reason to believe the car was capable of complete autonomy. A simple Google search will tell you that this driver wasn't the only one.

deleted 9y ago

[deleted]

gitpusher 9y ago

You're saying this is a matter of "driver shaming"? Could be, especially given the stakes for Tesla. It would be helpful to know exactly what these warnings look/sound like, because as a previous poster noted, many drivers learn to ignore warnings if they're persistent, such as a check engine light (or "low washer fluid", which I receive in my BMW, and whose sound is identical to that for more serious issues.)

fivestar 9y ago
I think Tesla rolled it out too soon and marketed it as something it is not yet capable of and people died because of it.
WhatIsDukkha 9y ago
There is no mention of a "collision" warning is there? That is very disturbing and why I'm bothered by this round of reporting. The warnings seem to be just "hey you aren't supposed to hands off for this long" which, as others point out, are just the type of notifications that people start to ignore as they grow more comfortable with technologies.
FooHentai 9y ago
>Could be, especially given the stakes for Tesla. This is a really important thing to remain vigilant about - Tesla and other self-driving car developers have an absolute incentive to blame/excuse/deny when it comes to fault for any autonomous vehicle accident. My money is on us seeing a LOT of 'this dead driver was a total idiot' and 'this totally crazy thing happened that the autopilot could never have predicted and 100% of human drivers would have failed too'. The hard truth is a lot of people are going to die in the process of self-driving vehicles reaching maturity. This truth is not politically tenable for any party to admit. The real risk in this pattern of accidents followed by blame avoidance is that valuable lessons might not be learned from incidents. The aviation industry accident investigation process went through a very similar early life and many lives were lost unnecessarily before it matured into the robust, root-cause-driven, appropriately powerful system it is today. That was what it took for aviation safety to reach what I think most would agree is a suitable level of governance, and TBH I think autonomous vehicles are going to need to go through this same process and will hopefully get to a similar level of maturity without the loss of too many lives.

gshulegaard 9y ago

> What did the autopilot do and when did it do it seems to be missing from this round of reports on this incident and thats pretty lame. I am pretty sure this is because it has been reported on pretty thoroughly. https://www.tesla.com/blog/tragic-loss https://www.tesla.com/blog/tragic-loss https://www.nytimes.com/interactive/2016/07/01/business/inside-tesla-accident.html https://www.nytimes.com/interactive/2016/07/01/business/insi... The general sense is that the system is designed to apply brake when both radar and the camera system agree there was an obstacle (this may no longer be true since there was indication that decoupling was being worked on). It has been suggested that the white trailer against a bright blue sky didn't register on visual detection. Thereby there was not a consensus between radar and visual that there was an obstacle so the brake was not applied. > we would expect it to before broadly deploying these devices. And this is why there is focus on the driver. What is reasonable expectation? Are there adequate warnings and documentation about the capabilities of the system? Autopilot is not marketed as an autonomous driving system...it (at the moment) explicitly says drivers should continue to monitor the system to manually intervene if required. Further NHTSA research suggest as high as a 40% crash reduction since the wide introduction of Tesla's autopilot mode: https://techcrunch.com/2017/01/19/nhtsas-full-final-investigation-into-teslas-autopilot-shows-40-crash-rate-reduction/ https://techcrunch.com/2017/01/19/nhtsas-full-final-investig... This is a single accident...and arguably the driver was the final safety gate and they failed to apply corrective action (in addition to the automated safety systems). It does not appear as if the driver had sufficient reason to believe the system was capable of complete autonomous operation. And it does not appear as if there was a critical flaw or bug in the system that was putting consumers at risk. And just briefly going over the items in the roughly 500 page report there does appear to be a lot of detailed examination of the systems.

WhatIsDukkha 9y ago
Awesome reply, thanks. I'm actually even more bothered by this. This looks like a super common scenario people in cars face daily. I acknowledge the technical obstacles to "wide view" forward sensors but wow, this system seems very very limited. ""“The limited time the target is in the field of view prior to impact challenges the system’s ability to perform threat assessment and apply the CIB system. A target is usually recognized very late or not at all prior to impact.”""" It seems like the FOV of the sensors is not adequate. The system should be looking backward to assess the danger of a sudden braking? Sensor fusion should resolve ambiguity via probability... one line in the reporting suggested it needs both systems to "agree" which is a binary, this is probably just reporter confusion. This system doesn't seem ready for anyone younger then 21 full stop. If it doesn't radically improve it's going to have many more problems as it gets in the hands of a wider range of drivers beyond affluent people in middle age+. These systems are "lane keepers" and really the marketing and driver education should be tightly focused around what they DO NOT do. I genuinely look forward to less humans in control of cars but this is going to be a rough road.