12 ms·
I have now talked with two people who have autopilot in their model S's and both said the problem with autopilot is that it is "too good". Specifically it works
by ChuckMcM 9y ago
I have now talked with two people who have autopilot in their model S's and both said the problem with autopilot is that it is "too good". Specifically it works completely correctly like 80 - 90% of the time and as a result, if you use it a lot, you start to expect it to be working and not fail. If it beeps then to tell you to take over, you have to mentally get back into the situational awareness of the road and then decide what to do about it. If that lag time is longer then the time the car has given you to respond, you would likely crash.
Makes me wonder how this gets resolved without jumping through the gap to 'perfect' autopilot.
- doe88 9y agoThis. I don't know I feel ambivalent about autopilot right now, as most on HN I always crave for new technologies, but the fact that something very dangerous can be very good most of the time and very bad occasionally make me weary of using it altogether. I know that's irrational because most often then not it probably saves you from your own mistakes. Also, if Tesla intends to shield itself behind a beta status for their autopilot system until it is perfect, I think this beta status will remain even longer then the time GMail was in beta. At least, in this case they should own this problem and somewhat hardcode a meaningful warning at such intersections or do something.
- carlivar 9y agoIt's not irrational at all. I drive our family 2018 Honda Odyssey, and it's already "saving me from my own mistakes" by beeping angrily when I signal for a lane change and there's someone in my blind spot. Works really well. It also has a notification light on each side of the car to indicate blind spot object presence, so I have to miss that first. The question seems to be which is better: 1) A car very intelligently helping a human drive better. 2) Or a car mostly driving well itself but needing humans to very rarely override fatal crashes. I prefer 1) I'm not sure there are any studies comparing these things, since "human driving but lots of extra tech help" is harder to quantify and doesn't have all the data going back to the mothership like Tesla.
- JKCalhoun 9y agoFor me, you nailed it. Just more 1 above please.
- worldsayshi 9y agoI think that 1 and 2 still can entail the same problems though. If it helps you check your blind spot you might occasionally stop checking yourself and rely on it to check for you. Then you implicitly start relying on that feature to work. It's different degrees of handing over control though.
- oldgradstudent 9y agoWhy are they allowed to put beta software on public roads where third parties that never consented to their EULA are put at risk. If they want beta software on a private track, that's their (and their insurance company's) problem.
- buvanshak 9y agoBecause no one is stopping them from. Even here, in HN, which is supposed to be filled with tech people, this is rarely asked. Then it is not surprising the common people and non-tech savvy authorities are complacent...
- Piskvorrr 9y agoOh, it is asked all right. But then the business side interrupts with time-to-market and diminishing returns and the cost of settling out of court vs. probability thereof - and voila, "move fast and kill people".
- qaq 9y agoYou just described every modern car on the road.
- tlrobinson 9y agoRelevant quote from an article about the Air France 447 crash: > To put it briefly, automation has made it more and more unlikely that ordinary airline pilots will ever have to face a raw crisis in flight—but also more and more unlikely that they will be able to cope with such a crisis if one arises. https://www.vanityfair.com/news/business/2014/10/air-france-flight-447-crash https://www.vanityfair.com/news/business/2014/10/air-france-... (linked to from another HN post today: https://news.ycombinator.com/item?id=16757343 https://news.ycombinator.com/item?id=16757343 )
- icc97 9y agoAirline pilot training is designed to handle this, as well as they have a co-pilot who prevents them from being distracted and is able to take over if they are distracted. Tesla is just handing it out to anyone who can afford to buy a Model S/X (and now Model 3) with the absolute minimum of warnings that they can get away with.
- sho_hn 9y agoI would not be surprised if Musk coldly responded that this is a known problem with no solution, but autopilot still lowers the death rate on average. In other words, for the proponents it's a trade-off. And also the idea that an automatic driver is easier to improve over time (skewing the trade-off better and better) than human drivers. Which of course will snowball when reliance on automation keeps making drivers worse than better, and at some point you're past the point of no return. Whether that's right data must show.
- gambiting 9y agoI don't think that's the correct answer. Flight autopilots have lowered accident rates tenfold, but every time there is a crash while on autopilot Boeing/Airbus will ground every single plane of that type and won't allow them to fly until the problem is found and fixed. If I know that my car's autopilot is statistically less likely to kill me than I am to kill myself, I would still much much rather drive myself.
- bobsil1 9y agoYep, it degrades badly, like digital cellular calls which drop out suddenly.
- omarforgotpwd 9y agoYou have to learn how to use it properly and pay attention. I use it a lot and it can drive from San Francisco to LA pretty much without stopping. But every once in a while it does mess up and you just need to make sure you're watching and ready to take over quickly. I agree that it's good enough that people might stop paying attention, but they just need to realize that they have to hold the software's hand in these initial stages. As a matter of fact, being in the drivers seat able to take control makes me much more comfortable in a self driving Tesla than in the back seat of a much more advanced Waymo self driving car
- danmaz74 9y agoIt you have to pay attention as much as if you were driving, then, what's the point of autopilot?
- RangerScience 9y agoActually... I notice my right hip and leg hurts in a pretty unique way after long drives, particularly if there's a lot of speed variance. That might not be worth the full cost of autopilot (particularly compared with lane keepers), but it's a thing.
- danmaz74 9y agoSo, physical fatigue. I didn't consider that; it's not really what I'm waiting autopilot for, but it's indeed something useful :)
- omarforgotpwd 9y agoTrust me, it's still a huge relief. A day and night difference. Try it on a long drive, like LA to San Francisco and you'll see what I mean. It kind of feels like you're in the passenger seat rather than the drivers seat, but you can still grab the controls if you need to.
- alkonaut 9y ago
- schoen 9y agoBrad Templeton has some very relevant discussion of this in https://www.templetons.com/brad/robocars/levels.html https://www.templetons.com/brad/robocars/levels.html including > There is additional controversy, it should be noted, about the proposed level 2. In level 2, the system drives, but it is not trusted, and so a human must always be watching and may need to intervene on very short notice -- sub-second -- to correct things if it makes a mistake. [...] As such, while some believe this level is a natural first step on the path to robocars, others believe it should not be attempted, and that only a car able to operate unsupervised without needing urgent attention should be sold.
- nikofeyn 9y agoi don’t see why it’s controversial. just look at drivers now. many are already on human autopilot checking their phone, eating, etc. if you give any amount of automation but that which still requires checking in from the human, humans will almost universally check out and crashes will be prevalent. autonomous driving will simply not work without full, unsupervised driving. i don’t understand why this is even a discussion amongst people.
- worldsayshi 9y agoOne solution would be to make the autopilot worse within the realm of safe driving. Make it so that it continuously makes safe errors that the driver has to correct. Like steering off into an empty lane. Slowing down. Etc. Defining safe errors might be a challenge though.
- masklinn 9y agoThat's what most "driving assistance" does. It's limited in what it offers and it refuses to keep going unattended for long periods. Tesla has been criticised all along for both its marketing and allowing long periods of unattended driving assistance.
- cardinalfang 9y agoOr the car could regularly ask a question about e.g. the colour of the car behind you.
- jijojv 9y agoThe lag time is for it to know you're distracted is ~ every 1min when you have to jiggle the wheel.
- Smaug123 9y ago"Jiggle the wheel" and "have situational awareness of the road around you" are two very different things.
- icc97 9y agoAs far as I can tell there is no way to resolve this that will be effective. Google spoke about this problem publicly and why they directly targeted level 4. Train drivers are the closest thing and they have a lot of equipment that forces them to stay alert. Personally I think that autopilot should only take over when the driver is unwell or that it thinks there will be a crash. Even when driving full time drivers get distracted. So when the car is doing the driving humans will get bored and won't be able to respond in time. Autopilot however is always on and never distracted but doesn't work in some cases.
- agumonkey 9y agoMaybe force users back in either speeds above 30, crowded places, complex topologies.
- carlivar 9y ago> they have a lot of equipment that forces them to stay alert. As does Cadillac's Super Cruise system which is not getting the respect it should -- including by GM itself.
- philbarr 9y ago> Train drivers are the closest thing and they have a lot of equipment that forces them to stay alert. Except they sometimes fall asleep: http://www.bbc.co.uk/news/uk-england-london-39457148 http://www.bbc.co.uk/news/uk-england-london-39457148
- njarboe 9y agoAnd in the current climate of "all new things must be super extra safe to be used, even if it displaces much worse technology", I think machine driving cars will have a hard time being accepted. Imagine the autopilot is 100 times safer than human drivers. Full implementation would mean about a fatality a day in the US. It does not seem this would be acceptable by the public even though 30,000 annual human car fatalities could be avoided. Maybe we should use our education system to help heal us from the safety first (do nothing) culture we seems to gotten our selves into in the US. This is a similar situation to nuclear power. With nuclear, all your waste can be kept and stored and nuclear has almost zero deaths per year. Contrast to the crazy abomination that is coal burning.
- m_mueller 9y agoDoes it currently really reduce fatalities if you control for car cost, age group and area driven? I‘m unconvinced of the current statistics.
- stickfigure 9y agoI think you could have just stopped that sentence after does it currently really reduce fatalities?.
- m_mueller 9y agowhy? got better data than me? please share.
- njarboe 9y agoCurrently it is probably worse than the average driver. But here is a trolley problem for you. What is the ratio of people killed who volunteer to test out an auto-drive to the number of lives saved by that early testing? 1:1, 1:10, 1:100, 1:1000, 1:1,000,000? A million people die in vehicle wrecks each year.
- creato 9y ago
- throwaway84742 9y agoUncanny valley of autonomy I guess. Google has noticed this early on, and their answer was to remove the steering wheel completely from their campus-only autonomous vehicles. Either it’s very close to 100% level 5 (you can pay no attention whatsoever and trust the car 100%), or it’s explicitly level 2 (advanced ADAS essentially), there’s nothing in between that’s not inherently dangerous.
- buvanshak 9y ago> said the problem with autopilot is that it is "too good". Isn't it true that the thing cannot detect a stationary object in its path if the vehicle is traveling at above 50kph? If that is true, then I think this is an extremely dangerous situation that the owners of these vehicles are in...
- Piskvorrr 9y agoOwners are in their armored cocoon. I'd be worried about whatever is besides/behind/in/at/ that stationary object.
- jve 9y agoAre you kidding? 80-90% seems awfully low to trust it.
- cam_l 9y agoI honestly don't understand why, if the driver does not take control when the car has sensed an issue, it does not just throw on hazards and roll slow or stop. Seems like the safest way to keep people from dying.
- adanto6840 9y agoCan you think of any reasons or situations where that logic fails? I can think of tons. The very challenging, and somewhat unusual given the life/death stakes, aspect of this domain in general is that general 'easy' logic works for 99% of the scenarios. But in the remaining 1% -- and I mean every single possible last scenario, whether probable or not -- the "what to do" is so much more difficult. Enough time to do it? Can be done safely? Etc. The reason the car turns control back over to the human at all is because the vehicle presumably does NOT have any maneuvers left which are high-confidence -- whether that's because the sensing equipment failed, the software failed or failed to interpret the data satisfactorily, or there's just no "good move" left, or any other number of reasons, etc...
- cam_l 9y agoslowing down should be the first thing a car does in these scenarios. the confidence factor matters less as speed approaches zero. seriously, i slow down when i am unsure of the conditions, i surely have never evaluated every last scenario, and i have never run into a concrete barrier.
- qaq 9y agoOK so you have a speeding loaded truck behind you that would not be able to break for whatever reason still want to slow down?
- sangnoir 9y ago> OK so you have a speeding loaded truck behind you that would not be able to break for whatever reason still want to slow down? Mental experiment: which is preferable - colliding with a static barrier at an unchecked 70MPH or getting rammed from behind by a truck at a relative speed of 30MPH (assuming that's how much you've braked by)? That said - slowing down =/= emergency stop - that truck behind you should leave enough room in front of it to come to a complete stop - unless you've cut in front of it. I'm probably ranting, but I'll repeat this as I've witnessed it far too many times: do not cut in front of loaded semi-trucks. The gap in front of them is intentional and it wasn't meant for you - it's their braking distance - they have ridiculous momentum and they can squash you!
- Gustomaximus 9y ago> Makes me wonder how this gets resolved without jumping through the gap to 'perfect' autopilot. I suspect the unfortunate reality is people die on the journey to improvement. Once we decide this technology has passed a reasonable threshold, it now becomes a case of continual improvement. And as horrible as this sounds this is not new. Consider how much safer cars are than 20 years ago and how many lives modern safety features save. Or go for a drive in a 50 year of sports car and see how safe you feel. And in 50 years we'll look back at today's systems in the same way we look at the days of no seat belts.
- EliRivers 9y agoIndeed. If I have to be sober and paying attention when riding in a robot car, I might as well just drive it myself just to keep me from falling asleep. At which point, I'll just not bother with the robot car at all.
- dawnerd 9y agoBest solution is to keep the driver engaged, IE still holding the wheel and showing they are paying attention. It's how all the other cars with lane assist do it. Cant go more than 30 seconds of not touching the wheel before it complains. It's not perfect but it sure beats the driver being so used to it they start doing other things like watching movies on their laptop.
- mtgx 9y agoNo that's not the best solution. The best solution is to not have it at all until it's actually good enough to self-drive you. 30 seconds is a HUGE amount of time to react on a highway. You'll be long dead by the time another 25 seconds pass if the accident happens just 5 seconds after you took your hands off the wheel. And this is exactly what happened in the recent crash.
- dawnerd 9y agoNot going to ever be able to prevent someone from taking their eyes/hands off the road. The point is to keep reminding them to pay attention.
- antirez 9y ago"too good" -> "completely correctly like 80 - 90%". I'm unable to make the two sentences make sense in the same statement, since 85% looks extremely low to be considered too good, given that the outcome is to crash if you do not pay attention in the 15%.
- YetAnotherNick 9y agoSome people do it. Think of it as 0-100 scale, 0 useless, 100 making no mistake ever, and not as some percent of correct action.
- antirez 9y agoIMHO this is kinda of a mental trick. How good it is an hammer that allows you to place a nail 90% of times? Percentage of failures is not usefulness, or a value of perfection, like: I'm 80% handsome, that is a lot handsome. Something that has 20% of failures rate instead is terrible.
- ChuckMcM 9y agoI expect it is the nature of the problem. Autopilot fails when the road conditions are such that it cannot function correctly. Imagine that you drive your commute from point A to point B and use autopilot all the time and it has never failed you. Then you use it when driving from point A to point C and because there are road conditions on that route that it cannot handle it tries to drive you into a ditch. Your experience may be giving you more confidence in its ability than it deserves. This is made worse when it has worked fine from point A to point B until one day a painter had a bad day and one of the cans of paint they forgot to secure fell off the back of their truck and put a big diagonal splash on the lane. And on that day your autopilot drives you into the car next to you.
- chronid 9y ago> Specifically it works completely correctly like 80 - 90% of the time and as a result, if you use it a lot, you start to expect it to be working and not fail. Different environment (and stakes), but I observed the same thing happening a couple of times during IT incident response. The automation crashed/failed/got into a situation it could not automatically fix, and the occurrence was rare enough that people actually panicked and it took them quite a while to fix what in the end was a simple issue. They just didn't have the "manual skill" (in one case, the tools to actually go and solve the issue without manually manipulating databases) anymore.
- ZoFreX 9y agoIf people are using it, not paying attention, with the expectation that it will beep to tell you to take over that's a big problem. In situations like this divider issue it won't beep, it thinks everything is fine right up until it rams you into a stationary object. I think people may not be fully aware of all the potential failure modes of this tech?
- mtgx 9y ago> Makes me wonder how this gets resolved without jumping through the gap to 'perfect' autopilot. The solution is easy, but probably not what many would like to hear: ban all "self-driving" or "near-self-driving" solutions from being activated by the driver, and only allow the ones that have gone through very rigorous and extensive government testing (in a standardized test). When the government certifies the car for Level 4 or Level 5, and assuming the standardized test has strict requirements like 1 intervention at 100,000 miles or whatever is something around 10-100x better than our best drivers, then the car can be allowed on the road. Any lesser technology can still be used in other cars, bit instead of being a "self-driving mode" it should actually just assist you in the sense of auto-braking when an imminent accident is about to happen, or maybe just warn you that an imminent accident is about to happen, which could still significantly improve driver safety on the road until we figure out Level 5 cars.
- pja 9y agoThe problem is that Autopilot is simultaneously too good and not good enough.
- mkirklions 9y agoThe problem is that Tesla is releasing Enhanced driving and calling it Autopilot. I dont trust anything from Elon anymore, big talker and lots of hype.
- mozumder 9y agoGM's well reviewed SuperCruise focuses on monitoring the driver so that the driver is paying attention to the road. That's what 'too good' is supposed to be.
- deleted 9y ago[deleted]
- base698 9y agoMy first drive with auto pilot tried to follow into the back of a stopped car in a turning lane and randomly had phantom breaking issues like going under overpasses. Not to mention there are things most people do defensively while driving that autopilot doesn't: anticipate a vehicle coming into my lane by both looking at the wheel position and the lateral movement. Autopilot ignores those pieces of information and data until it's in basically in your lane. Personally I feel I have to be more on guard and attentive with it on because I know their are fatal bugs.
- blktiger 9y agoMy Subaru has a "lane assist" feature which helps keep you in your lane while driving. It only gives a little bit of pressure in the direction you should be moving to help you out and if you don't do anything the car will still move out of the current lane. This certainly helps me stay more aware as I can't ever rely on the car to fully steer, but if my concentration lacks during a turn on the freeway I get a reminder that I need to turn if I'm not paying as much attention when I come upon a curve in the road or just drift to one side a bit. So it definitely helps me remember that I'm always in control. OTOH, the car has probably saved me from at least one accident where traffic ahead of me suddenly slowed down right as my attention relaxed. The _only_ correct way of using these systems is to treat them as an extra level of safety on top of your responsibilities as a driver. An "extra set of eyes" to help you avoid an accident while leaving you as the primary driver of the vehicle.
- mrtron 9y agoI use the Subaru adaptive cruise control quite a bit. It turns off when it is unable to see well enough, which usually only happens in scenarios where I shouldn't be driving and can't see (like during a blizzard). Human+automation is very powerful! However, I don't think we are far from self-driving beating human alone.
- jsight 9y agoAs someone with a car that has a much weaker system (ProPilot), I can see where that would be a problem. Its not really tempting to let ProPilot do it on its on, as it regularly wants to take exit ramps and occasionally pushes towards the middle of the road too much. It seems like AutoPilot users take their hands off the wheel regularly, and I just don't see how that is safe with this type of system.