17 ms·
NTSB: Tesla’s Autopilot UX a “major role” in fatal Model S crash
- rhino369 9y agoLevel 3 autonomy is reckless. It provides next to no value when it's used as intended (i.e., a fully aware drivers instantly ready to take over) and it is dangerous under normal usage conditions (i.e., people using it to text and drive).
- mtgx 9y agoI would say that's an accurate "elevator pitch" of Level Level 3 or below autonomy. It's reckless of Tesla or anyone else to sell it as a working commercial feature just to try to get ahead of competitors. I'm a big fan of Tesla and Musk, and you can see that from one of my recent comments on HN, too, but I've pretty much always criticized the Autopilot for exactly this reason. Humans can't handle Autopilot in its current form, no matter how many asterisks Tesla puts in its Terms of Service and how many prompts and alerts it adds to warn the drivers. Either it's completely worry-free "let me take a nap until destination" self-driving - or it's not, and the driver should be expected to do manual driving at all times.
- hangonhn 9y agoI was about to make a very similar comment and have very similar positions as you. The "Autopilot" feature is in this weird no-man's land of being in between two good places. As the article points out, the ambiguity of who's doing the driving is dangerous. The whole "you must be ready to override it" caveat really ignores what many people know about human behavior: if allowed to misbehave, someone will. The first X times the drivers are allowed to ignore the caveat and not have consequences will embolden him to continue doing it until something bad happens. It seems almost reckless on Tesla's part to release the feature this way.
- ZeroBugBounce 9y agoI totally agree with you from the perspective of the driver, passengers and public safety. I wonder, though, if manufacturers are going to greatly accelerate their progress to level 4 or above from the data they get having level 3 in production vehicles. They may get much more quickly to a higher quality level 4 by having what amounts to a marketing novelty feature in practice in 1000s of vehicles in real-world conditions.
- kayoone 9y agoAudi has a new System in the A8 for traffic jams that works only up to 60kmh, but is designed to be left alone completely. The germans seem to be starting from the bottom in a conservative way working up, while Tesla is much more aggressive. I think in this case i prefer the germans in the way they are not overpromising as much as Tesla.
- pavlov 9y agoCouldn't Level 3 autonomy still be valuable for drivers who are slightly impaired, like elderly people? That seems like a substantial market.
- bluedino 9y agoThe same elderly people who drive through the front of buildings because it takes them so long to recognize they hit the gas instead of brake?
- sorenjan 9y agoHow would they be able to quickly take the wheel and brake if something happens? I think the opposite is true, if you need longer reaction time it's probably a bad idea to not be fully immersed in the driving until an accident is imminent.
- pavlov 9y agoI'm thinking of a scenario where the driver is holding the wheel, but the computer is actually doing the driving most of the time while giving just enough feedback to the driver to feel like she is in charge.
- mandevil 9y agoIn our tests a decade ago, we found that shared autonomy was actually much more frustrating for users than either full or no autonomy. When you were telling the computer to go over there and the computer wanted to go over here, it left all of the users frustrated and complaining that the computer wouldn't let them go where they wanted. Maybe you can come up with a better adjudication system than we did, but it is very tricky to design the system so that it catches when the user does something stupid but not when they are doing the right thing- if the computer could figure out the right thing you wouldn't need the human!
- pavlov 9y agoInteresting, cool to hear your experience!
- dmode 9y agoThis is just not true. It provides immense value in stop and go traffic and long distances. I have driven 18,000 miles using Tesla Autopilot and it is a life changer if you have a tough commute. One fatal crash doesn't change that.
- rhino369 9y agoWhile this is a good point, the obvious solution is to limit it to say under 20 mph and require constant hands on the wheel. If you want to have your hands off the wheel and your nose in your phone, then you are driving recklessly.
- dmode 9y agoThat will not be very useful. Stop and go traffic usually moves between 0-40 miles an hour. AP2 initial rollout was limited to 35 miles / hr. I couldn't use it even in the Bay Area as traffic would speed up to 40-45 and slow down to 0 continually. People have their nose in their phone even without autopilot. You can't change that.
- rhino369 9y agoYou could fix the speed to whatever would allow a person to jump back when warned. But I suspect that speed is pretty low. The answer is tough shit, you don't get to not pay attention while driving. Get an uber then. Having adaptive cruise control that manages stop and go will absolutely change the number of people on their phones.
- readhn 9y agoi agree. not useful for you at 20? bad luck. for most folks in traffic 20-30mph is plenty to help "auto" navigate through the traffic jams. my average current driving speed on car display is something like 25mph. thanks to constant traffic. the car is rated to 160mph top speed. how useful is that.
- dmode 9y ago
- zmarty 9y agoTesla has L2, barely. L3 in fact allows you to not pay attention. L3 requires several seconds (10-15) warning period for you to take over the wheel.
- karol 9y ago> The machine learning algorithms that underpin AEB systems have only been trained to recognize the rear of other vehicles, not profiles or other aspects. Not something I have known about Tesla's autopilot before reading this.
- mtgx 9y agoIt makes you wonder what else we don't know about how specific and non-general the Autopilot is in its object recognition capabilities, doesn't it?
- CalRobert 9y agoGeeze, this is terrifying given that my rear probably doesn't look like that of a vehicle when I'm walking, or cycling, or motorcycling, etc.
- EngineerBetter 9y agoI've seen Autopilot 2 fail to notice cyclists. But then, it's designed for use on large, fast roads, rather than city streets. I cycle through the streets of London every day, BTW, so I'm not suggesting that it's lack of recognition ability is a good thing.
- kbenson 9y agoWell, it's a glorified cruise control, so it makes some sense, even if it's not optimal. My cruise control doesn't recognize profiles or other aspects either, nor does it the read of other vehicles, because it's an old dumb system. The last time I used a lane assist system (a few months back), it had a hard time even finding the lane. These are not self driving cars, these are driver assisting cars. You should probably have a stronger sense of safety making a long trip with a 16 year old that just got their license driving. Even then, I wouldn't feel too comfortable taking my attention from the road for too long.
- dmode 9y agoAEB = Automatic Emergency Braking. It is one part of the Autopilot system and doesn't apply to the front radar which recognizes the whole car.
- dsfyu404ed 9y agoNo matter how you cut it the failure mode trying to drive a car of known dimensions though a smaller space than those dimensions is pretty inexcusable. >The machine learning algorithms that underpin AEB systems have only been trained to recognize the rear of other vehicles, not profiles or other aspects Does Tesla's software make the assumption that the lane is safely navigable unless it detects otherwise? IMO it's pushing the envelope of recklessness to have a default software of "good to go" for a non-critical system (whereas something like a shipboard fire suppression system should default to "good to go" in order to prevent corner cases from impacting functionality). While I understand that you don't want the car braking aggressively when it encounters a bag blowing in the wind or poor lane markings mid-corner not driving into something those are behaviors that are not unexpected from an excessively conservative human driver and preferable to running into the back of a van that's parked on the side of the road. edit: If you're gonna down-vote you might wanna say why.
- plaidfuji 9y agoNot sure why you're getting downvoyed because it's a good point. How in the world do you let a computer vision decision override the other sensors that are saying "massive wall is moving in front of the car"? And who tf thought training the CV only on the backs of cars was good enough? The edge cases are what cause accidents, not the typical situations.
- mandevil 9y agoBecause a lot of this is transient signals that are crap? One of our robots back in '06 would panic stop every 30 seconds or so whenever there was a dust cloud (this was an army off road formation project, so dust was a fact of life for all the followers) because occasionally there would be enough random scatter returns from the dust to convince the MMWR that there was a wall in front of it. Panic stopping from highway speeds every time your MMWR detected a flyer floating in the breeze near some dust seems like it would be pretty dangerous- especially for the cars behind you that are also traveling at highway speeds. And LADAR doesn't have the range to be useful at highway speeds (from memory, the ratio of range to safe braking distances means you can't get above 25-30kmh with LADAR being useful for you). As for whether having shared autonomy at highway speeds is a good idea, that's a really good question that I hope Tesla is asking itself right now. But if you are trying to do that, cameras pretty much have to be your primary source of truth.
- jquery 9y agoThe fact that the NTSB is investigating is comforting. We're moving crashes from the domain of "shit happens" to more systematic "autopilot system X suffered Y deficiency", more analogous to the investigation of plane crashes. I'm hopeful to see the number of car fatalities sharply drop over the next 20 years as autopilots improve.
- sokoloff 9y agoYou might be thinking of the investigation of commercial air carrier crashes. In general aviation (loosely "light aircraft") crashes, there are still a great many of them closed with "pilot lost control of aircraft for undetermined reasons" and "NTSB staff did not travel as part of the investigation but relied on other agencies who had personnel on the ground".
- Animats 9y agoYes. Here's the actual NTSB abstract.[1] The full report isn't out yet. Main recommendation: To manufacturers of vehicles equipped with Level 2 vehicle automation systems (Audi of America, BMW of North America, Infiniti USA, Mercedes-Benz USA, Tesla Inc., and VolvoCar USA): 5. Incorporate system safeguards that limit the use of automated vehicle control systems to those conditions for which they were designed. 6. Develop applications to more effectively sense the driver’s level of engagement and alert the driver when engagement is lacking while automated vehicle control systems are in use. The NTSB points out that the recorded data from automatic driving systems needs to be recoverable after a crash without manufacturer involvement. For plane crashes, the NTSB gets the "black boxes" (which are orange) and reads them out at an NTSB facility. They want that capability for self-driving cars. [1] https://www.ntsb.gov/news/events/Documents/2017-HWY16FH018-BMG-abstract.pdf https://www.ntsb.gov/news/events/Documents/2017-HWY16FH018-B...
- davedx 9y agoThat's really reassuring. Constructive and thoughtful recommendations. Even though this government agency is in a country I do not live in, I feel better about autopilot systems from reading how this is being addressed over in the USA.
- hashkb 9y agoThe report found human error was equally to blame and went out of its way to say that self driving cars are the future. This article is sensationalizing.
- ceejayoz 9y agoSure, but the human error is having too much confidence in the autopilot.
- sorenjan 9y agoI think it's weird that so many people seem to think that Tesla have made such a significant technology leap ahead of everyone else that they're comfortable with putting their lives in the hands of their Autopilot system while other manufacturers have no such thing available in production. I wonder how much of that relaxed critical thinking is down to the hype surrounding Tesla and its founder. I also think it's strange that they're allowed to sell cars equipped with Autopilot at the moment, there's too little known about its capabilities and weaknesses.
- stephengillie 9y agoMarketing is an amazing phenomenon.
- BoorishBears 9y agoThe AP 1 sensor was in cars for years. No one else was foolhardy enough to try and use it for more than LKA, not because the sensor couldn't do it, but because no one wanted to market an "AP" feature that didn't work 100% of the time it was available
- craftyguy 9y agoThey shouldn't be allowed to call it 'autopilot', and it should automatically (and very visibly/audibly) disable itself the second a driver takes his or her hands off the steering wheel. Why? Because apparently there are people out there who believe this is true autopilot and will move their attention elsewhere when flying down the road at 74mph.
- dmode 9y agoTesla has significantly reduced the nudge to put your hands on the wheel. I would hate it if they forced to keep my hands on the wheel all the time. There are many situations (stop and go traffic, miles of freeway with no one around) where it doesn't make sense to continually hold your wheel.
- 9y ago
- deleted 9y ago[deleted]
- MikusR 9y agoSeems that Elon Musk is taking quite an active role in showing dangers of AI.
- jakelarkin 9y agonaive take is he grossly overestimates AI capabilities ... tech today (1) cannot broadly drive a car (2) is not a near existential threat to humanity
- baq 9y ago1) he reads his SF 2) he understands exponential growth.
- readhn 9y agoI agree that the name of the feature - "autopilot" is misleading consumers. Honestly Tesla should be sued for this and name "autopilot" recalled until it really is an autopilot! "An autopilot - is a system used to control the trajectory of a vehicle without constant 'hands-on' control by a human operator being required." What Tesla has is not an autopilot!
- chc 9y agoWhat Tesla has seems to fit that definition. In fact, what Tesla has seems pretty similar to the autopilot in airplanes — both keep the vehicle on the right course without constant control from a human, but both require a human to be ready to take control in unforeseen circumstances. If it's deceptive, it's because drivers have an overly hopeful idea of what autopilot is, not because it's a very inaccurate name.
- undersuit 9y agoIt sounds to me like it is working as intended, intermittent human intervention of the trajectory of a vehicle. Autopilot has never meant 100% control by a robot, even airplanes still have pilots for most of the flight. Rip Riley: Uncuff me, you idiot! Holy God, if we overshot our chance to refuel... Sterling Archer: I thought you put it on autopilot! Rip Riley: It just maintains course and altitude! It doesn't know how to find THE ONLY AIRSTRIP WITHIN A THOUSAND MILES SO IT CAN LAND ITSELF WHEN IT NEEDS GAS! Sterling Archer: Then I, uh... misunderstood the concept.
- apeace 9y agoI'm a huge Musk fan and wish him and Tesla all the success in the world. But releasing any kind of machine-assisted driving without a near infallible "don't hit objects in front of the vehicle" function is simply irresponsible. This crash, as well as the one where a Tesla crashed into a concrete barrier[1] are evidence that their tech is not ready for release. How much do you want to bet the Waymo self-driving technology would have avoided both of these? Seems like one of the simplest cases to handle: something up ahead is blocking the way! Anyone who uses Tesla machine-assisted driving features today is putting themselves in grave danger (except for low-speed features like parallel parking). [1] https://www.youtube.com/watch?v=YIBTBxZ3NWw https://www.youtube.com/watch?v=YIBTBxZ3NWw
- zip1234 9y agoOn the other hand, if you know that the technology is not completely infallible, but better than humans in a general sense and will save lives by releasing it, wouldn't you be morally obligated to do so? Wouldn't holding it back even though it would save lives be worse?
- dlp211 9y agoIt can't be marginally better than humans, it has to be an order of magnitude or more better. Humans are irrational, and incidents like this erode public trust.
- zip1234 9y agoI agree, but there is also news about how Tesla's automatic braking features saved people from getting into collisions. I would think that might counteract some of the negative news. See this one from six months ago: https://www.youtube.com/watch?v=Ndeb1pMAsh4 https://www.youtube.com/watch?v=Ndeb1pMAsh4
- mantas 9y agoMost manufacturers have automatic braking. But they do not pretend it's "autopilot" nor let user to rely on it. It's just there to save their ass if shit hits the fan.
- torpfactory 9y agoThe first question should really be: "Is autopilot+human drivers safer than human drivers alone?" If it isn't, it would be arguably criminal to allow drivers access to it. As with most technological breakthroughs, there will be a learning curve (euphemism for death in a car crash in this case). As long as it is an improvement over human drivers, it is a net good, I would argue. From this perspective, even if Tesla is releasing a system with flaws, it is BETTER than the alternative: more humans dying by driving their own cars. Imagine if they waited to release a more perfect autopilot system, wouldn't that be keeping a valuable safety system off the road? This is only true, of course, if autopilot is actually safer than a human. From what I have read, there is little data to support either conclusion. Lacking proof that such a system is actually safer, Tesla should take the conservative engineering decision and reduce or eliminate drivers' access to such a system until enough data is available to make a conclusive statement on safety.
- deleted 9y ago[deleted]
- bognition 9y agoWhy does the title claim problem was in the UX when the article claims it was faulty models: >As NHTSA found, the Automatic Emergency Braking feature on the Tesla—in common with just about every other AEB fitted to other makes of cars—was not designed in such a way that it could have saved Brown's life. The machine learning algorithms that underpin AEB systems have only been trained to recognize the rear of other vehicles, not profiles or other aspects
- helloworld 9y agoBecause the user is an integral part of Tesla's "Autopilot" system. The product name implies autonomy, but in fact, the vehicle requires a human in the loop for safe operation. Unfortunately, a poorly designed user experience lulled the driver into behaving foolishly. That's why the headline mentions UX.
- josefresco 9y ago"lulled the driver" Did it? Or did the driver know what they were doing and not care? I don't think it's responsible to make such assumptions about the driver's reasoning.
- lagadu 9y agoWhat's relevant is that this wouldn't have happened with any of the other semi-autonomous systems from other brands because they force the user to be keenly aware. Had it been the same for all systems then the argument could be made that the current concept in general is flawed in this iteration but because the others demand a lot more driver involvement the problem pivots to being a Tesla problem, not an semi-autonomous driving problem.
- iamgopal 9y agoElephant in the room, why Tesla do not partner with Google for autopilot ? Google will have tons of data and Tesla will have an order of magnitude better software capabilities. Such partnership can even accelerate global automatic vehicle adoption.
- bobbygoodlatte 9y agoBecause both companies believe that owning proprietary self-driving systems will be a big competitive advantage. If one company beats the other to mainstream adoption by 12-18 months they'll have a shot at dominating the industry.
- crimsonalucard 9y agoThese self driving systems need to use machine learning models more in line with the what humans see. Nobody just recognizes the rear of a car... we look at the rear of a car and we are aware of its entire three dimensional shape and the three dimensional structure of the universe around the same car. Are there any machine learning models that can achieve this level of recognition or is it all just associating names with pictures?
- jpindar 9y agoDid you see the article recently about recognizing stop signs? However the machine supposedly recognized stop signs, it had nothing to do with them being red, hexagonal, or saying "STOP". It was possible to deface a sign such that it was clearly still red, hexagonal, etc. but the machine didn't recognize it. So IMO the machine had no 'mental model' of what a stop sign is.
- cmsonger 9y agoLots of folks are saying: too early. Let me make the opposite argument. My wife has to do regular weekend interstate travel. She also has serious RSI issues with her hands and arms. The ability to keep focus, but completely disengage from the steering wheel for a while and accelerator for the length of the trip is incredibly valuable to her. It's why we purchased the car. Tesla is clear on the limitations. The feature is turned off by default. Turning it on presents the limitations to the user.
- euyyn 9y agoI understand Tesla's autopilot is now disengaged if you fail to return the hands to the wheel on time?
- EngineerBetter 9y agoIf you use Autopilot for any length of time and believe that you don't need to be paying attention, then I'd suggest that you're responsible for whatever fate befalls you. Autopilot 2 keeps you in lane, does traffic-aware cruise control, and can do lane changes. That's it. No more, no less. It sometimes gets spooked by shadows. I've had it brake sharply when going under bridges on the motorway, because of the shadow on the road. I've had it mis-read lane markers (in understandable circumstances; the old lines had been removed but the marks they left still contrasted with the tarmac). If anything, I find Autopilot 2 to require too much hand-holding - literally! I rest my right hand gently on the bottom-right of the wheel when Autopilot is engaged, resting my wrist on my knee, which is also how I drive regular cars on the motorway. This isn't enough for Autopilot to think I've got my hands on the wheel, so it's constantly bugging me, and I have to wiggle the wheel to tell it I'm there. Apparently some Model 3s have been seen with driver-facing cameras. Eye-tracking seems like a much better solution to me. I love Autopilot. I recently had to drive ~500 miles on motorways in a thing that explodes ancient liquidised organic matter, and it was an absolute chore. Autopilot is ace when used as intended, and great in traffic jams.
- lagadu 9y agoI don't understand why you're mixing autopilot with IC engines here. The two things are completely unrelated.
- makecheck 9y agoThere is a ton of electric power available for features in a Tesla that might require an irresponsible amount of fuel in a regular car.
- mikeyouse 9y agoWhat? IC cars have been shipping with all of the features currently available in Teslas for the better part of a decade. 1 horsepower = 745 watts, so we're talking about somewhere in the range of 20HP to generate as much power as the average Tesla consumption including powering the drivetrain. I'd be surprised if all the electric features in a Tesla consume more than 5HP equivalent.
- mcguire 9y ago"The machine learning algorithms that underpin AEB systems have only been trained to recognize the rear of other vehicles, not profiles or other aspects." Holy crap. Who thought that was a good idea?
- amelius 9y agoPerhaps a stupid question, but what entity tests self-driving cars before they are admitted to the streets? Is this entity partially responsible?
- boznz 9y ago"Minor Role" I read.