10 ms·

Tesla's 'Autopilot' Will Make Mistakes. Humans Will Overreact

163 points by svenfaw 10y ago

tptacek 10y ago
This entire article is based on the premise that two very different situations are directly comparable. In the former, people sometimes accidentally stomp the gas instead of the brakes, then report the opposite. In the latter, Tesla markets a feature as "Autopilot". It pilots a car into another vehicle. The feature had apparently been demonstrated at events with key safety features disabled: attendees are able to use it without their hands on the wheel. "Autopilot" fails to account for a common real-world traffic occurrence and its driver has an inadequate mental model of the feature, which is made up of a complicated mesh of sensors and software. There may be something to the argument that self-driving car features will reduce traffic fatalities even as they become responsible for a distinctive (but hopefully small) subset of those fatalities. But this article doesn't make that point. The problem people have with features like Autopilot is that it may not be reasonable --- at least, as those features are currently designed --- to expect human drivers to model their behavior accurately enough to use them safely. This is an HCI problem that people in the computer industry should be very familiar with. I'm surprised to see it dismissed so easily.
- lisper 10y ago
  > without key safety features disabled Did you intend this to be a double-negative? It's pretty confusing as written, but would make sense as a single-negaitve.
  - tptacek 10y ago
    Nope, thanks. I shuffled the sentences around in that paragraph but should have just rewritten it.
  - deleted 10y ago
    [deleted]
- joosters 10y ago
  The author is comparing them in the context of the risk of uproar that they create. There's no comparison with the exact circumstances of the two problems. The article is describing how a past issue with Audi caused a huge sales problem for the company, and is suggesting that 'autopilot' style technologies could cause the same style of event.
  - tptacek 10y ago
    That's kind of a circular argument, right? The uproar is a function of the perceived cause of the event. In the Audi case, the perceived cause is incorrect. But we don't know that about the Tesla case.
- j2bax 10y ago
  While I greatly appreciate Tesla pushing this technology out in a reasonable time frame, I do feel that they let their marketing speak trump safety by referring to this technology as Autopilot. I don't doubt that they intend to eventually push this tech to the point that Autopilot is an appropriate name, but calling it such at this stage is disenguous at best and ultimately could be hindered by people treating it as its named and getting hurt or killed. It's not reasonable to expect all drivers that may get behind the wheel of a Tesla to get proper training and definitions for its technology package.
  - stefanix 10y ago
    The term is certainly no exaggeration if you look at what an autopilot does in a plane. Keep attitude, maybe turn to next waypoint. All in all super primitive to what a Tesla does.
    - cloakandswagger 10y ago
      I'm sure Tesla has stacks of pre-prepared legal defense around the term, and they'll certainly make the same airplane argument. My bet is it will quickly fall apart. Ask your average person what "autopilot" implies and they're going to give you the sci-fi definition of a completely autonomous driving system, not many people know about the mechanics and limitations of autopilots in airplanes.
    - Xylakant 10y ago
      That's totally sufficient for airplanes since crossing semi-trailers are fairly rare on airways. Jokes aside, all obstacles for airplanes are radar tracked, so keeping attitude is all that it needs doing. If human intervention is needed, the system usually has time to warn. Still, even those super-simple systems fail.
      - ams6110 10y ago
        All commercial aircraft have TCAS (collision avoidance).
    - 10y ago
- jballanc 10y ago
  FiveThirtyEight did a really great piece (much better written and more reasonable than this article) expanding on what you identify as the problem: http://fivethirtyeight.com/features/no-technology-not-even-teslas-autopilot-can-be-completely-safe/ http://fivethirtyeight.com/features/no-technology-not-even-t... In short: as automation is better able to handle easy-to-predict scenarios, that increasingly leaves only rare, complex scenarios for the humans to deal with which, by definition, they are far worse at handling.
  - argonaut 10y ago
    Part of my criticism of Tesla, though, comes from the fact that a large truck crossing in front of you is actually not a particularly complex situation for humans (just brake), even for a sleepy driver.
- deleted 10y ago
  [deleted]
lumberjack 10y ago
>It appears to have been something of a freak accident -- white trailer riding high against the bright sky, so that the autopilot didn’t detect the truck in its path. It wasn't a freak accident. The AI failed. The human driver was distracted and failed to respond in time. Two failures Tesla has to answer for. IMO the second one is the gravest. They shouldn't have deployed the technology, primitive as it is.
- ricardobeat 10y ago
  How come Tesla has to answer for human failure?
  - benkarst 10y ago
    Tesla calls the technology "Autopilot". What does that mean to you?
    - deleted 10y ago
      [deleted]
  - diggan 10y ago
    It depends on what kind of failure. If there is a red button next to an slightly darker red button and you get the instruction from your boss to press the red one, you have a 50% of failing. But maybe the designer of said buttons should have thought about giving the buttons different colors instead of similar ones. "The Design of Everyday Things" talks about this, that sometimes it's the designer of something's fault, rather than the user of the something. Also called "Human Error" by statements. Search for "https://archive.org/stream/DesignOfEverydayThings/DesignOfEverydayThings_djvu.txt" https://archive.org/stream/DesignOfEverydayThings/DesignOfEv... here https://archive.org/stream/DesignOfEverydayThings/DesignOfEverydayThings_djvu.txt https://archive.org/stream/DesignOfEverydayThings/DesignOfEv... Point being, unless Tesla makes it safe to be used as autopilot, you probably shouldn't say it's autopilot. Otherwise it's not good enough and you would fault Tesla for the design rather than the users of the autopilot.
joshdickson 10y ago
I used to work on the radar systems in these vehicles. It's a problem that the feature is called "autopilot," yes, but the larger issue is that the car does not enforce what the company often says in press releases, which is that you need to remain alert and with your hands on the wheel. There have been similar systems in other vehicles for years - I'm most familiar with those in Mercedes vehicles that are not as advanced but perform similar actions. Those systems will automatically turn off after just a few seconds of removing your hands from the wheel. That is what the Tesla should do. The autopilot feature is a toy; it's not to be used in place of a human driver, nor would a human driver be able to step in quickly and take over for the system were it to suddenly fail had the person not been paying attention. It should not be possible to watch a Youtube video of someone driving completely hands free for minutes at a time. Yes, Tesla should not call it "autopilot," but if you want to really prevent people from using the system in a way in which it's not intended, you need to enforce it in hardware.
- te 10y ago
  Is Tesla's autopilot using radar? How is radar fooled by a white truck against a bright background? I was surprised by this point of failure, which suggests to me that they aren't even using radar.
  - agildehaus 10y ago
    https://twitter.com/elonmusk/status/748625979271045121 https://twitter.com/elonmusk/status/748625979271045121
    - protomyth 10y ago
      Maybe I'm a little clueless on how radar works, but how would something that is lower than a car's height be mistaken for something as distant as a overhead sign?
      - agildehaus 10y ago
        I would assume the forward radar isn't capable of seeing anything past a foot or two off the ground. I suppose I don't really know, but that was Elon's answer to "why didn't the radar see it?"
paulsutter 10y ago
- Tesla has collected 130 million miles of autopilot data in 8 months - Google has collected 2 million miles of data in 5 years - others have collect 0 million miles of data One might assume that each 10x of data doubles safety. So, a billion miles of data is perhaps 8x better than a million. It was bold of Tesla to ship autopilot, but it gives them a significant data lead while Google's caution means they may be squandering their once-thought-unassailable technology lead. My thinking is that first you eliminate the 1% problems, then the 0.1% problems, then 0.01% and so on. So each 10x of scenarios tested is roughly one step of improvement.
- JshWright 10y ago
  Google is operating truly autonomous vehicles. Tesla has added some driver assist features which are far from unique to Tesla... It's an apple to oranges comparison.
  - paulsutter 10y ago
    Maybe, or maybe Tesla wisely shipped an MVP and Google allowed the perfect to be the enemy of thr shippable. Project those data numbers forward, do you really think Google can maintain a technology lead with such a disparity? Maybe they can, but it's a reasonable question.
    - JshWright 10y ago
      Google could build a car tomorrow that does what Tesla does. They are far from revolutionary in what they are doing... Things like auto braking and lane maintaining are pretty much standard at the price point of the Model S, they are by no means unique to Tesla. Tesla is implementing existing technology. Google is working on solving the next problem.
      - paulsutter 10y ago
        Yes and Google hasn't shipped jack. EDIT: Tesla has.
danielmorozoff 10y ago
The tesla incident highlights the exact technical/ethical problems of using such systems. I was just at a conference speaking with the Google automated car team and the handoff problem is yet to be solved conceptually. The tesla accident is just a first example of it. Obviously the utility of an autopilot system becomes questionable if you can't guarantee safety. It's very similar to medical advances in this way. 1% errors is not a good error rate if it leads to death or serious injury even though such error rates for autonomous systems are remarkable. That said , if we look at the statistical likelihood of death in a car autonomous systems may very well be much better than humans, but whether we as a society are ethically okay with coming to terms with a piece of software that kills 1 / 100M is another question. A drunk driver that kills someone or themselves is obviously to blame. In this case who's fault is it? The drivers , tesla or the safety board for allowing autopilots to be sold?
- pipio21 10y ago
  "In this case who's fault is it? The drivers , tesla or the safety board for allowing autopilots to be sold?" Obviously the insurance. In Europe you need to have insurance to pay whenever is your fault. With autopilot the only difference will be that the event of having an accident will be less probable. This means that paying the normal driver fee the insurance company will be making money if you autodrive. But most of the time the fault will be on the other side, because the autopilot is not going to drive like a drunk or slept person. It will be deer or dogs or children on the road when they should not be there.
  - danielmorozoff 10y ago
    I wouldn't be so quick to say insurance companies would pick up the tab. I don't know which country you're from but in the us getting insurance companies to pay for a claim is very hard and until a set of laws are established I would expect dramatic pushback from them. Furthermore in most accidents insurance companies try to seek out the party at fault if it's not crystal clear to avoid paying. But an interesting situation would occur if the incentives from the insurance companies would be directed towards automated driving because of reduced risks and costs to the driver. That would be an interesting scenario
Animats 10y ago
That's part of the problem. Tesla has two basic problems with their "autopilot". One is hype, and one is bad technical design. Tesla's "autopilot" is just automatic lane keeping and automatic cruise control. Mercedes and BMW have been shipping that for years. The other car companies have hardware to check if the driver's hands are on the wheel. Tesla not only didn't do that, they hyped those basic functions as being an "autopilot", as if they had something comparable to what Google has. Tesla has significantly overhyped their technology, to a dangerous level. Here's another Tesla crash, from yesterday.[1] This is a Tesla rear-ending the corner of a stalled van on a highway. This may be an "automation in wrong mode" error, and complicated excuses from Tesla will probably be forthcoming. That's not good enough. Airline pilots get into "wrong mode" errors, and they get extensive training, including simulator time. Drivers don't get that. This driver presumably thought the automation would do something sensible in that situation. It didn't. The other big Tesla problem is a pure technical design error - the bumper-mounted radar is blind at windshield height. This is the immediate cause of the fatal accident involving under-running the semitrailer. It's also the cause of the parking accident with the truck with overhanging load sticking out the back.[2] Tesla's implementation has deadly blind spots. They need a radar at windshield-top height. It's quite likely, now that the NHTSA is looking into this crash, that Tesla will have to do a recall and retrofit one. (Along with hands-on-wheel sensors.) Tesla was relying on stereo cameras too much. Depth from stereo is always iffy; there needs to be some detail on the target to get range. You can't range a uniform surface with stereo vision. (Human flyers have the same problem flying over ice.) A big white truck under some lighting conditions has that property. Cameras are useful, but backed only by today's algorithms, not enough. Teslas come with a sensor suite inadequate for the job. From a software perspective, Tesla's software seems to be overoptimistic that there's no problem ahead. 3D vision systems know when they don't have range info for a large area. Tesla's software was willing to drive into that. It didn't note the absence of range info for the road surface; it may not even be profiling the road ahead. (We all had to do that in the DARPA Grand Challenge because we were operating off-road. That's a solved problem. Google does it with their roof-mounted LIDAR.) Tesla owners have asked for "pothole detection" as a feature. Lack of this may eventually take someone off a cliff. The press is gradually picking up on this. Until recently, Tesla's autopilot was treated as a magic black box in the press. Now some automotive writers are starting to ask the right questions. This is not a setback for automatic driving. This is just a Tesla problem. [1] https://www.youtube.com/watch?v=qQkx-4pFjus https://www.youtube.com/watch?v=qQkx-4pFjus [2] http://www.roadandtrack.com/new-cars/car-technology/news/a29133/tesla-self-driving-crash-summon-autonomous/ http://www.roadandtrack.com/new-cars/car-technology/news/a29...
soneca 10y ago
Humans "overreacting" is the reason why airplanes are so safe. This reaction (IMO "fair" not "over") is important for the development of the driverless cars tech. It will make them much, much safer and won't kill any innovation. "Underreaction"[1] is much more dangerous for society than overreaction. I would cite digital privacy as an example of this. [1] i'm not sure underreaction is a word or if i invemted it. Is underreaction so underreacted to that it is not even a word?
- the8472 10y ago
  From a utilitarian perspective one can invest too many resources in making specific things more secure. That happens when other causes of death dwarf the probability of incidence to cost of fixing ratio of a particular failure mode. If you're accounting for micromorts then we might be at the point where TSA checks destroy more human life-hours than terrorism and normal plane crashes combined. Simply reverting airport security to pre-9/11 levels could save more lives than infinite spending increases for airplane crash safety. So yes, at some point it becomes an overreaction when resources would be better spent on other ways of saving lives. I'm not saying that we're there yet with self-driving cars. We probably are with planes.
  - ams6110 10y ago
    The things we could do to improve air safety at this point are not technology related. If we want to spend money on air safety, paying pilots more, spending more on their training, and reducing their work hours would make the biggest difference.
  - rtpg 10y ago
    Didn't there used to be a huge amount of plane hijackings in the 60s? That doesn't seem good for the reliability If you look at AWS and the like, or services that tout "5 9s", you could make similar arguments. But reliability is itself a feature of a system. If airplane crashes were more frequent, less people would rely on it (if you think it's hysterical now...). Public Transit and package delivery are other things where reliability is super important. Not just public perception, but in a real way. If you have to change trains twice, you can't have both be late right? Also, death isn't just "subtract the man-hours from this person's society"....
satyajeet23 10y ago
This is how things evolve.
- deleted 10y ago
  [deleted]
- Pica_soO 10y ago
  It shall get a chapter in the orange catholic bible, the prelude to the butlerian jihad. ;)
AndrewKemendo 10y ago
This is the most horrifying part to me: Eventually, the National Highway Traffic Safety Administration got involved, and wrote up a report which found that … yup, these drivers were stepping on the gas instead of the brakes, often with horrific results. That didn’t save Audi, for which sales collapsed and which almost pulled out of the U.S. market. Despite thousands of hard working engineers, QA, Marketing, Sales, Legal and design people earnestly working to put out a safe, quality vehicle, Audi was almost undone by histrionic consumers. Really depressing when you think about it as a founder.
- carterehsmith 10y ago
  Far as I remember, that Audi had pedals that were narrower than usual and closer than usual. So it's not about the customers being histrionic, more like somebody in engineering not really understanding what he was doing.
tuna-piano 10y ago
I'm saddened, though not surprised, with how many people are so quick to come to Tesla's defense here. Autonomous cars will have issues. But we should hold the software makers accountable for those issues the same way we hold airplane makers / maintainers responsible in the rare event their planes crash. As other commenters pointed out, Tesla likely released a half-baked technical solution too early, and created misleading (and in this case dangerous) marketing to go along with it: calling it "autopilot" and not requiring hands on the steering wheel. Tesla screwed up here, and shame on them. This is not a binary world: I'm not saying that they should be bankrupted or the autonomous car industry shut down. When a plane crashes due to poor maintenance - I want that company to suffer market+legal consequences, I don't want the entire aviation industry to be shut down. White trucks on a sunny sky are not rare enough to call this an anomaly. If Tesla had installed sensors on their cars to collect data but not enabled autopilot yet - would this man still be alive today? The MVP model is great, but not for things where life is on the line. Google seems to be doing this right.
- DasIch 10y ago
  The product Tesla has built is a perfectly fine assistance mechanism. The problem is that it's not an autopilot and nowhere near autonomous driving, nevertheless Tesla is naming it and marketing it as such. The problem is that Tesla's marketing here is (borderline) fraudulent. There should be and need to be legal consequences for that.
  - cynix 10y ago
    > The problem is that it's not an autopilot… An autopilot is a system that assists, but does not replace, the human operator in operating a vehicle. Tesla's system does exactly this. > … and nowhere near autonomous driving… Nobody ever claimed it was autonomous.
    - DasIch 10y ago
      A plane can depending on the airport start, fly and land on autopilot, you just need to tell it where to go. There is pretty much no unplanned intervention necessary. This is what the term means to people, autonomous driving. I don't think most people would draw any distinction between the two. In a Tesla it would be irresponsible to take your hands of the wheel or not to pay attention to the road for even a minute. Tesla's "Autopilot" is nothing like what people imagine an autopilot to be like.
kirrent 10y ago
I don't get the argument that using the name Autopilot implies that the driver need not supervise the car. In aircraft autopilots require pilot supervision who can take over in the event of autopilot failure. In marine applications autopilots are even simpler. No-one would think that an autopilot in either of these fields where the word has been traditionally applied can function completely independently.
- comicjk 10y ago
  I disagree. Besides specialists, I think most people think that Autopilot means it doesn't need supervision. Wiktionary's definition: A mechanical, electrical, or hydraulic system used to guide a vehicle without assistance from a human being. "I set the autopilot to due south, so I could get some rest."
rdlecler1 10y ago
There were 1.07 fatalities every 100M automobile miles last year. Compare that to 1 fatality in 130M autopilot miles. This is not going to be the last fatality we see but it's not clear that autopilot is any worse than a human and statistically this still looks to be the safer option.
- unfamiliar 10y ago
  For comparison, that is 1.391 fatalities per 130M automobile miles. So 0.391 people were saved by Tesla, but that doesn't account for the fact that this is a new, eerie technology and that people are possibly using it in the safest of conditions until they get comfortable with it. Also is the average Tesla driver really representative of the overall population, i.e. would Tesla drivers have had that many crashes without Autopilot? How much of that 0.391 is due to the safety features of the car itself, and not the success of the autopilot system? At any rate, there isn't much room for error in those numbers, and considering the above this almost looks like evidence that Autopilot is less safe than normal driving.
- kartD 10y ago
  But that's 1.07/100M in all conditions, while the 1 in 130M is in ideal conditions (suited for autopilot), so I doubt this is a valid comparison.
- contravariant 10y ago
  >but it's not clear that autopilot is any worse than a human Well no, that would constitute an immediate and utter failure. More importantly though it's not significantly better than a human, which is disappointing, especially since they'd already limited the system to the simplest possible use case.
KKKKkkkk1 10y ago
The biggest elephant in my view is that Tesla has known about this accident and has been hiding it from investors since May. The only reason why this came out is the NHTSA investigation, and if it were up to Musk, this crucial information would have stayed secret until he had completed the dumping of SolarCity onto Tesla shareholders.
99_00 10y ago
As Tesla's sales mix shifts from mostly early adopters to more 'normal' consumers we are going to see a lot more complaints. Normal consumers aren't interested in technical details or complicated explanations. They just want what the marketing promised them.
rasz_pl 10y ago
Best outcome of NHTSA investigation would be financial penalty and order prohibiting use of AutoPilot brand on NON autonomous system. It was clearly picked to mislead customers into thinking they are buying more than there really is.
ychoratio 10y ago
How about $500 million fine for Mr. Elon Musk with five years supervision of public statements by or for Tesla by government appointed lawyer.
Sami_Lehtinen 10y ago
A bad joke, but this somehow reminded me about: Darwin Awards movie, Autopilot Cruise Control scene. Afaik, Tesla Autopilot is an assistant, not an fully autonomous system. Which inherently means that it requires constant supervision. Using boats or planes autopilot won't either relieve you from monitoring where you're going to end up. Afaik, some of the news articles were titled with pure lies. 'Autonomous car', hmm, nope. Tesla isn't such. It's always important to acknowledge the true capability s of a system and not blindly trust it. This applies to any more or less automated system. People start blindly trusting it even if it really shouldn't be trusted. Let's say robot or AI stock investment bots. Luckily those are't going to kill anyone. I know, quite rude. It's sad that what happened happened. But in technological sense, we all knew it was going to happen sooner or later. We're going to see more and more things like this in future. - I've blogged several times about this same phenomenon, even before this case.
ojosilva 10y ago
I think Tesla screwed up by hyping it like they have an autonomous driving system. Mercedes, BMW, etc. did not make that mistake with similar technology. Just google "tesla autopilot" and you'll see the phenomena: people sleeping, playing games, recording videos of themselves. From that search, here's an interesting article from WIRED warning about the dangers (and lawsuit proneness) of the technology: https://www.wired.com/2015/10/obviously-drivers-are-already-abusing-teslas-autopilot/ https://www.wired.com/2015/10/obviously-drivers-are-already-...

163 points by svenfaw 10y ago

This entire article is based on the premise that two very different situations are directly comparable. In the former, people sometimes accidentally stomp the gas instead of the brakes, then report the opposite. In the latter, Tesla markets a feature as "Autopilot". It pilots a car into another vehicle. The feature had apparently been demonstrated at events with key safety features disabled: attendees are able to use it without their hands on the wheel. "Autopilot" fails to account for a common real-world traffic occurrence and its driver has an inadequate mental model of the feature, which is made up of a complicated mesh of sensors and software. There may be something to the argument that self-driving car features will reduce traffic fatalities even as they become responsible for a distinctive (but hopefully small) subset of those fatalities. But this article doesn't make that point. The problem people have with features like Autopilot is that it may not be reasonable --- at least, as those features are currently designed --- to expect human drivers to model their behavior accurately enough to use them safely. This is an HCI problem that people in the computer industry should be very familiar with. I'm surprised to see it dismissed so easily.

lisper 10y ago
> without key safety features disabled Did you intend this to be a double-negative? It's pretty confusing as written, but would make sense as a single-negaitve.
- tptacek 10y ago
  Nope, thanks. I shuffled the sentences around in that paragraph but should have just rewritten it.
- deleted 10y ago
  [deleted]
joosters 10y ago
The author is comparing them in the context of the risk of uproar that they create. There's no comparison with the exact circumstances of the two problems. The article is describing how a past issue with Audi caused a huge sales problem for the company, and is suggesting that 'autopilot' style technologies could cause the same style of event.
- tptacek 10y ago
  That's kind of a circular argument, right? The uproar is a function of the perceived cause of the event. In the Audi case, the perceived cause is incorrect. But we don't know that about the Tesla case.
j2bax 10y ago
While I greatly appreciate Tesla pushing this technology out in a reasonable time frame, I do feel that they let their marketing speak trump safety by referring to this technology as Autopilot. I don't doubt that they intend to eventually push this tech to the point that Autopilot is an appropriate name, but calling it such at this stage is disenguous at best and ultimately could be hindered by people treating it as its named and getting hurt or killed. It's not reasonable to expect all drivers that may get behind the wheel of a Tesla to get proper training and definitions for its technology package.
- stefanix 10y ago
  The term is certainly no exaggeration if you look at what an autopilot does in a plane. Keep attitude, maybe turn to next waypoint. All in all super primitive to what a Tesla does.
  - cloakandswagger 10y ago
    I'm sure Tesla has stacks of pre-prepared legal defense around the term, and they'll certainly make the same airplane argument. My bet is it will quickly fall apart. Ask your average person what "autopilot" implies and they're going to give you the sci-fi definition of a completely autonomous driving system, not many people know about the mechanics and limitations of autopilots in airplanes.
  - Xylakant 10y ago
    That's totally sufficient for airplanes since crossing semi-trailers are fairly rare on airways. Jokes aside, all obstacles for airplanes are radar tracked, so keeping attitude is all that it needs doing. If human intervention is needed, the system usually has time to warn. Still, even those super-simple systems fail.
    - ams6110 10y ago
      All commercial aircraft have TCAS (collision avoidance).
  - 10y ago
jballanc 10y ago
FiveThirtyEight did a really great piece (much better written and more reasonable than this article) expanding on what you identify as the problem: http://fivethirtyeight.com/features/no-technology-not-even-teslas-autopilot-can-be-completely-safe/ http://fivethirtyeight.com/features/no-technology-not-even-t... In short: as automation is better able to handle easy-to-predict scenarios, that increasingly leaves only rare, complex scenarios for the humans to deal with which, by definition, they are far worse at handling.
- argonaut 10y ago
  Part of my criticism of Tesla, though, comes from the fact that a large truck crossing in front of you is actually not a particularly complex situation for humans (just brake), even for a sleepy driver.
deleted 10y ago
[deleted]

lumberjack 10y ago

>It appears to have been something of a freak accident -- white trailer riding high against the bright sky, so that the autopilot didn’t detect the truck in its path. It wasn't a freak accident. The AI failed. The human driver was distracted and failed to respond in time. Two failures Tesla has to answer for. IMO the second one is the gravest. They shouldn't have deployed the technology, primitive as it is.

ricardobeat 10y ago
How come Tesla has to answer for human failure?
- benkarst 10y ago
  Tesla calls the technology "Autopilot". What does that mean to you?
  - deleted 10y ago
    [deleted]
- diggan 10y ago
  It depends on what kind of failure. If there is a red button next to an slightly darker red button and you get the instruction from your boss to press the red one, you have a 50% of failing. But maybe the designer of said buttons should have thought about giving the buttons different colors instead of similar ones. "The Design of Everyday Things" talks about this, that sometimes it's the designer of something's fault, rather than the user of the something. Also called "Human Error" by statements. Search for "https://archive.org/stream/DesignOfEverydayThings/DesignOfEverydayThings_djvu.txt" https://archive.org/stream/DesignOfEverydayThings/DesignOfEv... here https://archive.org/stream/DesignOfEverydayThings/DesignOfEverydayThings_djvu.txt https://archive.org/stream/DesignOfEverydayThings/DesignOfEv... Point being, unless Tesla makes it safe to be used as autopilot, you probably shouldn't say it's autopilot. Otherwise it's not good enough and you would fault Tesla for the design rather than the users of the autopilot.

joshdickson 10y ago

I used to work on the radar systems in these vehicles. It's a problem that the feature is called "autopilot," yes, but the larger issue is that the car does not enforce what the company often says in press releases, which is that you need to remain alert and with your hands on the wheel. There have been similar systems in other vehicles for years - I'm most familiar with those in Mercedes vehicles that are not as advanced but perform similar actions. Those systems will automatically turn off after just a few seconds of removing your hands from the wheel. That is what the Tesla should do. The autopilot feature is a toy; it's not to be used in place of a human driver, nor would a human driver be able to step in quickly and take over for the system were it to suddenly fail had the person not been paying attention. It should not be possible to watch a Youtube video of someone driving completely hands free for minutes at a time. Yes, Tesla should not call it "autopilot," but if you want to really prevent people from using the system in a way in which it's not intended, you need to enforce it in hardware.

te 10y ago
Is Tesla's autopilot using radar? How is radar fooled by a white truck against a bright background? I was surprised by this point of failure, which suggests to me that they aren't even using radar.
- agildehaus 10y ago
  https://twitter.com/elonmusk/status/748625979271045121 https://twitter.com/elonmusk/status/748625979271045121
  - protomyth 10y ago
    Maybe I'm a little clueless on how radar works, but how would something that is lower than a car's height be mistaken for something as distant as a overhead sign?
    - agildehaus 10y ago
      I would assume the forward radar isn't capable of seeing anything past a foot or two off the ground. I suppose I don't really know, but that was Elon's answer to "why didn't the radar see it?"

paulsutter 10y ago

- Tesla has collected 130 million miles of autopilot data in 8 months - Google has collected 2 million miles of data in 5 years - others have collect 0 million miles of data One might assume that each 10x of data doubles safety. So, a billion miles of data is perhaps 8x better than a million. It was bold of Tesla to ship autopilot, but it gives them a significant data lead while Google's caution means they may be squandering their once-thought-unassailable technology lead. My thinking is that first you eliminate the 1% problems, then the 0.1% problems, then 0.01% and so on. So each 10x of scenarios tested is roughly one step of improvement.

JshWright 10y ago
Google is operating truly autonomous vehicles. Tesla has added some driver assist features which are far from unique to Tesla... It's an apple to oranges comparison.
- paulsutter 10y ago
  Maybe, or maybe Tesla wisely shipped an MVP and Google allowed the perfect to be the enemy of thr shippable. Project those data numbers forward, do you really think Google can maintain a technology lead with such a disparity? Maybe they can, but it's a reasonable question.
  - JshWright 10y ago
    Google could build a car tomorrow that does what Tesla does. They are far from revolutionary in what they are doing... Things like auto braking and lane maintaining are pretty much standard at the price point of the Model S, they are by no means unique to Tesla. Tesla is implementing existing technology. Google is working on solving the next problem.
    - paulsutter 10y ago
      Yes and Google hasn't shipped jack. EDIT: Tesla has.

danielmorozoff 10y ago

The tesla incident highlights the exact technical/ethical problems of using such systems. I was just at a conference speaking with the Google automated car team and the handoff problem is yet to be solved conceptually. The tesla accident is just a first example of it. Obviously the utility of an autopilot system becomes questionable if you can't guarantee safety. It's very similar to medical advances in this way. 1% errors is not a good error rate if it leads to death or serious injury even though such error rates for autonomous systems are remarkable. That said , if we look at the statistical likelihood of death in a car autonomous systems may very well be much better than humans, but whether we as a society are ethically okay with coming to terms with a piece of software that kills 1 / 100M is another question. A drunk driver that kills someone or themselves is obviously to blame. In this case who's fault is it? The drivers , tesla or the safety board for allowing autopilots to be sold?

pipio21 10y ago
"In this case who's fault is it? The drivers , tesla or the safety board for allowing autopilots to be sold?" Obviously the insurance. In Europe you need to have insurance to pay whenever is your fault. With autopilot the only difference will be that the event of having an accident will be less probable. This means that paying the normal driver fee the insurance company will be making money if you autodrive. But most of the time the fault will be on the other side, because the autopilot is not going to drive like a drunk or slept person. It will be deer or dogs or children on the road when they should not be there.
- danielmorozoff 10y ago
  I wouldn't be so quick to say insurance companies would pick up the tab. I don't know which country you're from but in the us getting insurance companies to pay for a claim is very hard and until a set of laws are established I would expect dramatic pushback from them. Furthermore in most accidents insurance companies try to seek out the party at fault if it's not crystal clear to avoid paying. But an interesting situation would occur if the incentives from the insurance companies would be directed towards automated driving because of reduced risks and costs to the driver. That would be an interesting scenario

Animats 10y ago

That's part of the problem. Tesla has two basic problems with their "autopilot". One is hype, and one is bad technical design. Tesla's "autopilot" is just automatic lane keeping and automatic cruise control. Mercedes and BMW have been shipping that for years. The other car companies have hardware to check if the driver's hands are on the wheel. Tesla not only didn't do that, they hyped those basic functions as being an "autopilot", as if they had something comparable to what Google has. Tesla has significantly overhyped their technology, to a dangerous level. Here's another Tesla crash, from yesterday.[1] This is a Tesla rear-ending the corner of a stalled van on a highway. This may be an "automation in wrong mode" error, and complicated excuses from Tesla will probably be forthcoming. That's not good enough. Airline pilots get into "wrong mode" errors, and they get extensive training, including simulator time. Drivers don't get that. This driver presumably thought the automation would do something sensible in that situation. It didn't. The other big Tesla problem is a pure technical design error - the bumper-mounted radar is blind at windshield height. This is the immediate cause of the fatal accident involving under-running the semitrailer. It's also the cause of the parking accident with the truck with overhanging load sticking out the back.[2] Tesla's implementation has deadly blind spots. They need a radar at windshield-top height. It's quite likely, now that the NHTSA is looking into this crash, that Tesla will have to do a recall and retrofit one. (Along with hands-on-wheel sensors.) Tesla was relying on stereo cameras too much. Depth from stereo is always iffy; there needs to be some detail on the target to get range. You can't range a uniform surface with stereo vision. (Human flyers have the same problem flying over ice.) A big white truck under some lighting conditions has that property. Cameras are useful, but backed only by today's algorithms, not enough. Teslas come with a sensor suite inadequate for the job. From a software perspective, Tesla's software seems to be overoptimistic that there's no problem ahead. 3D vision systems know when they don't have range info for a large area. Tesla's software was willing to drive into that. It didn't note the absence of range info for the road surface; it may not even be profiling the road ahead. (We all had to do that in the DARPA Grand Challenge because we were operating off-road. That's a solved problem. Google does it with their roof-mounted LIDAR.) Tesla owners have asked for "pothole detection" as a feature. Lack of this may eventually take someone off a cliff. The press is gradually picking up on this. Until recently, Tesla's autopilot was treated as a magic black box in the press. Now some automotive writers are starting to ask the right questions. This is not a setback for automatic driving. This is just a Tesla problem. [1] https://www.youtube.com/watch?v=qQkx-4pFjus https://www.youtube.com/watch?v=qQkx-4pFjus [2] http://www.roadandtrack.com/new-cars/car-technology/news/a29133/tesla-self-driving-crash-summon-autonomous/ http://www.roadandtrack.com/new-cars/car-technology/news/a29...

soneca 10y ago

Humans "overreacting" is the reason why airplanes are so safe. This reaction (IMO "fair" not "over") is important for the development of the driverless cars tech. It will make them much, much safer and won't kill any innovation. "Underreaction"[1] is much more dangerous for society than overreaction. I would cite digital privacy as an example of this. [1] i'm not sure underreaction is a word or if i invemted it. Is underreaction so underreacted to that it is not even a word?

the8472 10y ago
From a utilitarian perspective one can invest too many resources in making specific things more secure. That happens when other causes of death dwarf the probability of incidence to cost of fixing ratio of a particular failure mode. If you're accounting for micromorts then we might be at the point where TSA checks destroy more human life-hours than terrorism and normal plane crashes combined. Simply reverting airport security to pre-9/11 levels could save more lives than infinite spending increases for airplane crash safety. So yes, at some point it becomes an overreaction when resources would be better spent on other ways of saving lives. I'm not saying that we're there yet with self-driving cars. We probably are with planes.
- ams6110 10y ago
  The things we could do to improve air safety at this point are not technology related. If we want to spend money on air safety, paying pilots more, spending more on their training, and reducing their work hours would make the biggest difference.
- rtpg 10y ago
  Didn't there used to be a huge amount of plane hijackings in the 60s? That doesn't seem good for the reliability If you look at AWS and the like, or services that tout "5 9s", you could make similar arguments. But reliability is itself a feature of a system. If airplane crashes were more frequent, less people would rely on it (if you think it's hysterical now...). Public Transit and package delivery are other things where reliability is super important. Not just public perception, but in a real way. If you have to change trains twice, you can't have both be late right? Also, death isn't just "subtract the man-hours from this person's society"....

satyajeet23 10y ago

This is how things evolve.

deleted 10y ago
[deleted]
Pica_soO 10y ago
It shall get a chapter in the orange catholic bible, the prelude to the butlerian jihad. ;)

AndrewKemendo 10y ago

This is the most horrifying part to me: Eventually, the National Highway Traffic Safety Administration got involved, and wrote up a report which found that … yup, these drivers were stepping on the gas instead of the brakes, often with horrific results. That didn’t save Audi, for which sales collapsed and which almost pulled out of the U.S. market. Despite thousands of hard working engineers, QA, Marketing, Sales, Legal and design people earnestly working to put out a safe, quality vehicle, Audi was almost undone by histrionic consumers. Really depressing when you think about it as a founder.

carterehsmith 10y ago
Far as I remember, that Audi had pedals that were narrower than usual and closer than usual. So it's not about the customers being histrionic, more like somebody in engineering not really understanding what he was doing.

tuna-piano 10y ago

I'm saddened, though not surprised, with how many people are so quick to come to Tesla's defense here. Autonomous cars will have issues. But we should hold the software makers accountable for those issues the same way we hold airplane makers / maintainers responsible in the rare event their planes crash. As other commenters pointed out, Tesla likely released a half-baked technical solution too early, and created misleading (and in this case dangerous) marketing to go along with it: calling it "autopilot" and not requiring hands on the steering wheel. Tesla screwed up here, and shame on them. This is not a binary world: I'm not saying that they should be bankrupted or the autonomous car industry shut down. When a plane crashes due to poor maintenance - I want that company to suffer market+legal consequences, I don't want the entire aviation industry to be shut down. White trucks on a sunny sky are not rare enough to call this an anomaly. If Tesla had installed sensors on their cars to collect data but not enabled autopilot yet - would this man still be alive today? The MVP model is great, but not for things where life is on the line. Google seems to be doing this right.

DasIch 10y ago
The product Tesla has built is a perfectly fine assistance mechanism. The problem is that it's not an autopilot and nowhere near autonomous driving, nevertheless Tesla is naming it and marketing it as such. The problem is that Tesla's marketing here is (borderline) fraudulent. There should be and need to be legal consequences for that.
- cynix 10y ago
  > The problem is that it's not an autopilot… An autopilot is a system that assists, but does not replace, the human operator in operating a vehicle. Tesla's system does exactly this. > … and nowhere near autonomous driving… Nobody ever claimed it was autonomous.
  - DasIch 10y ago
    A plane can depending on the airport start, fly and land on autopilot, you just need to tell it where to go. There is pretty much no unplanned intervention necessary. This is what the term means to people, autonomous driving. I don't think most people would draw any distinction between the two. In a Tesla it would be irresponsible to take your hands of the wheel or not to pay attention to the road for even a minute. Tesla's "Autopilot" is nothing like what people imagine an autopilot to be like.

kirrent 10y ago

I don't get the argument that using the name Autopilot implies that the driver need not supervise the car. In aircraft autopilots require pilot supervision who can take over in the event of autopilot failure. In marine applications autopilots are even simpler. No-one would think that an autopilot in either of these fields where the word has been traditionally applied can function completely independently.

comicjk 10y ago
I disagree. Besides specialists, I think most people think that Autopilot means it doesn't need supervision. Wiktionary's definition: A mechanical, electrical, or hydraulic system used to guide a vehicle without assistance from a human being. "I set the autopilot to due south, so I could get some rest."

rdlecler1 10y ago

There were 1.07 fatalities every 100M automobile miles last year. Compare that to 1 fatality in 130M autopilot miles. This is not going to be the last fatality we see but it's not clear that autopilot is any worse than a human and statistically this still looks to be the safer option.

unfamiliar 10y ago
For comparison, that is 1.391 fatalities per 130M automobile miles. So 0.391 people were saved by Tesla, but that doesn't account for the fact that this is a new, eerie technology and that people are possibly using it in the safest of conditions until they get comfortable with it. Also is the average Tesla driver really representative of the overall population, i.e. would Tesla drivers have had that many crashes without Autopilot? How much of that 0.391 is due to the safety features of the car itself, and not the success of the autopilot system? At any rate, there isn't much room for error in those numbers, and considering the above this almost looks like evidence that Autopilot is less safe than normal driving.
kartD 10y ago
But that's 1.07/100M in all conditions, while the 1 in 130M is in ideal conditions (suited for autopilot), so I doubt this is a valid comparison.
contravariant 10y ago
>but it's not clear that autopilot is any worse than a human Well no, that would constitute an immediate and utter failure. More importantly though it's not significantly better than a human, which is disappointing, especially since they'd already limited the system to the simplest possible use case.

KKKKkkkk1 10y ago

The biggest elephant in my view is that Tesla has known about this accident and has been hiding it from investors since May. The only reason why this came out is the NHTSA investigation, and if it were up to Musk, this crucial information would have stayed secret until he had completed the dumping of SolarCity onto Tesla shareholders.

99_00 10y ago

As Tesla's sales mix shifts from mostly early adopters to more 'normal' consumers we are going to see a lot more complaints. Normal consumers aren't interested in technical details or complicated explanations. They just want what the marketing promised them.

rasz_pl 10y ago

Best outcome of NHTSA investigation would be financial penalty and order prohibiting use of AutoPilot brand on NON autonomous system. It was clearly picked to mislead customers into thinking they are buying more than there really is.

ychoratio 10y ago

How about $500 million fine for Mr. Elon Musk with five years supervision of public statements by or for Tesla by government appointed lawyer.

Sami_Lehtinen 10y ago

A bad joke, but this somehow reminded me about: Darwin Awards movie, Autopilot Cruise Control scene. Afaik, Tesla Autopilot is an assistant, not an fully autonomous system. Which inherently means that it requires constant supervision. Using boats or planes autopilot won't either relieve you from monitoring where you're going to end up. Afaik, some of the news articles were titled with pure lies. 'Autonomous car', hmm, nope. Tesla isn't such. It's always important to acknowledge the true capability s of a system and not blindly trust it. This applies to any more or less automated system. People start blindly trusting it even if it really shouldn't be trusted. Let's say robot or AI stock investment bots. Luckily those are't going to kill anyone. I know, quite rude. It's sad that what happened happened. But in technological sense, we all knew it was going to happen sooner or later. We're going to see more and more things like this in future. - I've blogged several times about this same phenomenon, even before this case.

ojosilva 10y ago

I think Tesla screwed up by hyping it like they have an autonomous driving system. Mercedes, BMW, etc. did not make that mistake with similar technology. Just google "tesla autopilot" and you'll see the phenomena: people sleeping, playing games, recording videos of themselves. From that search, here's an interesting article from WIRED warning about the dangers (and lawsuit proneness) of the technology: https://www.wired.com/2015/10/obviously-drivers-are-already-abusing-teslas-autopilot/ https://www.wired.com/2015/10/obviously-drivers-are-already-...