10 ms·
A former pilot on why autonomous vehicles are so risky
- 0zemp2c 3y ago[flagged]
- yawpitch 3y agoI’m pretty content that “the day when AI in cars can handle ALL conditions on the road, ALL OF THE TIME” (emphasis mine) won’t happen in your lifetime, or indeed the lifetime of any currently gestating fetus.
- joshribakoff 3y agoIt is also true that humans in general cannot drive [without crashing].
- croes 3y agoAnd if the have an accident they are liable. Who is if a fully autonomous driving car has an accident.
- watwut 3y agoThey provably can.
- Espressosaurus 3y agoWith suitable training they can. The insurance industry expects the average insured driver to file a claim every 18 years. That's covering all conditions, all of the time. No "hand off to someone else when I get scared" because humans can't do that. That's honestly pretty good. Wake me when autonomous driving can get within a factor of two of that.
- 0zemp2c 3y agohuman can't handle "ALL conditions ALL OF THE TIME" either this is a false goal that doesn't exist in the real world cars, planes, trains, all automation of any kind, all the code you ever wrote, nuclear power plants...all this and more operates with some fallibility that is deemed acceptable given the benefits no one is excusing AV accidents that seem to defy common sense (randomly driving into barriers etc)...but humans fail in these ways too, every day
- joshribakoff 3y agoIn case you’re living under a rock, it already happened. Are they perfect and driving everywhere in every city? No… but here in SF it is not unusual to see cars with no one in them, they’re driving all over the city, at all times, in parking lots, through school zones, etc Also you’re trying to discredit a well credentialed individual (who worked in automotive safety regulation) based upon age/gender and other irrelevant factors. I feel the need to call that out as toxic.
- lumost 3y agoThe rollout beyond sf has been … slow. Which begs the question, how autonomous are autonomous vehicles? Is this a technology or an economics problem, or both?
- 0zemp2c 3y agoshrug, I'm older than her, so the age thing doesn't resonate with me I appreciate her credentials, but I don't see why her statements are somehow unassailable and I stick by my point that her history as a fighter pilot has no bearing not sure where your "gender" point comes from? other than using the pronoun, I made no claims based on gender
- aatd86 3y agoI agree with the fact that "experts" are quite often wrong and it's too easy to be derailed by appeal to authority. On the other hand she has a point. As long as the tech won't be 99.99xxxx safe, it will introduce moral hazard (she calls it "risk-homeostasis" but I don't think that kind of neologism is particularly expressive). At a meta-level, coining some new term could make someone raise an eyebrow. Perhaps she just didn't know about it.
- AQuantized 3y agoI don't think the poster is trying to discredit her based on age, just contextualizing her comment about it not happening in her lifetime by estimating what period that's likely to refer to. The comment about her appeal to authority on the basis of being a fighter pilot is also more a counterpoint to crediting her additionally on that basis above any non-expert, since that skillset isn't actually particularly related to autonomous driving.
- Jtsummers 3y ago> This woman appears to be in her fifties...claiming autonomous driving will not happen in her lifetime She doesn't claim that. You claim she claimed that. Why make things up? In the interview she talks about existing autonomous driving applications that are operating today. How can you go from reading that part of her statements to your gross misinterpretation?
- deleted 3y ago[deleted]
- asasidh 3y agoIt is difficult to get a man or woman to understand something, when their salary depends on them not understanding it.
- EGreg 3y agoThis aphorism can be applied to pretty much the entire AI industry right now. Similar to Web3 industry before it, except AI is potentially far more dangerous. https://en.m.wikipedia.org/wiki/Existential_risk_from_artificial_general_intelligence https://en.m.wikipedia.org/wiki/Existential_risk_from_artifi... Many of the top AI experts are now coming out sounding warnings. AI Experts have predicted a chance of 10-20% human level extinction from AI, have a history of underestimating AI, but companies are rushing to the precipice, and it is just the early days. Everyone should listen to this warning: https://m.youtube.com/watch?v=xoVJKj8lcNQ https://m.youtube.com/watch?v=xoVJKj8lcNQ
- croes 3y agoHer job would still exist even with FSD. But Elon's income on the other hand depends on the success of Tesla's FSD.
- JimtheCoder 3y ago"But Elon's income on the other hand depends on the success of Tesla's FSD." Well, his income and net worth have done pretty well without delivering FSD, so maybe not?
- croes 3y agoBut not without promising FSD. What would happen if he cancels FSD?
- JimtheCoder 3y agoPeople would still buy Teslas?
- mihaaly 3y agoI have a feeling that she has the eyes of the highly trained professional driver (pilot) with good senses and ability to react well. Not something that can be told about the bulk of the drivers on the road. Although I see the point of and agree with the necessity of 'crytal clear' situation, knowing what exactly can be expected from the assitance, I believe that the average driver should have as much help as possible. The roads are pretty unsafe yet, because of the human drivers.
- tuatoru 3y agoI disagree that drivers should be assisted, and so does the interviewee in the article. She uses the phrase "risk homeostasis" to describe the phenomenon that if people feel they are getting helped they will engage in more risky behavior. I think roads and cars should be made to look less safe than they are, so the average driver will pay more attention.
- nytesky 3y agoDid any of you read the article? She acknowledged there are some cases where it’s working in well defined environments. But she clearly shows how we are currently in the most dangerous phase “car drives except handing off to human drivers in a crisis”. “ The policy should be that either the computer is driving or you are driving. And by driving I mean steering—people do fine with regular cruise control. The act of keeping your hands on the wheel and guiding the car’s lateral motion is enough to keep your brain engaged. So, no L3 [full self-driving, but the driver must be ready to take the wheel], which is too confusing, and no hands-free L2 [partial self-driving]. I am not against the passing of control per se, but there should just be two modes of operation, with crystal clear feedback about which mode you are in.” FSD has been 5 years away for… 15 years now?
- 0zemp2c 3y ago> FSD has been 5 years away for… 15 years now? this was only Elon-time hallucinations that made for great marketing you won't get to Mars on Elon-time either, but that doesn't mean you won't get there
- thedrbrian 3y ago>FSD has been 5 years away for… 15 years now? FSD is going to be enabled on HW2 anyday now. Oh look at that HW4 just rolled out.
- detaro 3y ago> FSD has been 5 years away for… 15 years now? the past few years it has been "later this year"
- MBCook 3y agoThis is one of the things that really annoys me. We have so much data and case studies of the effects of automations (of different levels of sophistication) and user interfaces and how those interact with other things including distractions through FAA investigations. To say nothing of other safety critical industries. We know what works ok. We know what’s dangerous. We know what’s lethal. There are lessons to look at. And yet Tesla and a lot of people seem content to re-learn the same lessons through blood over and over again. Like this is 100% incomparable to anything that’s ever happened before.
- ChatGTP 3y agoLobbying and conflicts of interests perhaps? Tech is so powerful now, it controls narratives, influences politics, huge revenues. I feel tech is beyond control at the moment.
- pigbearpig 3y agoSounds like a very good person to have in that position. Being a single pilot in an aircraft, she would not only have years of experience relying on automation, but also risk-reward decision making involving life or death. Self-driving cars are not a "restart the container" situation.
- jameshart 3y ago‘Former pilot’ seems like a fairly irrelevant part of the resume of a human-autonomous vehicle interaction researcher with a PhD in systems engineering and a series of professorships either side of an NHTSA advisory position.
- vba616 3y ago>‘Former pilot’ seems like a fairly irrelevant part of the resume Does it? "Tesla's Autopilot AI team drives the future of autonomy of current and new generations of vehicles"
- sigmoid10 3y agoFor most people "former navy pilot" is more likely to establish credibility by association. Not that this kind of journalism is worthy of praise, but it is pretty prevalent.
- natch 3y agoShe is pushing a competing technology. They forgot to mention that.
- lightedman 3y agoIt's actually fairly-relevant, as airplanes from commercial craft to military craft have a TON of autonomous systems.
- natch 3y agoHer being a major stockholder and board member of a LIDAR company is also important context, but was left out for some reason. She is pushing a technology approach Elon has called “a fool’s errand” which explains a lot of her incongruous hostility toward Tesla.
- vba616 3y agoHow it is important context? The article isn't about LIDAR. And if LIDAR is the wrong way to go for technical reasons, then if she pushes for better autonomous driving technology, she is working against her supposed interests.
- robotburrito 3y agoI can see it. The other day a self driving Cruise car was extremely aggressive with me, almost feeling aggressive as I tried to cross an intersection on foot. It was stop/starting quickly towards me to try and get me to walk faster it feels like. I think that these self driving cars need some kind of feedback system for pedestrians and cyclists. Even an emoji face that shows facial expressions would help. When I see a real driver at an intersection, I feel like we come to some agreement about what is going to happen next based on our facial expressions. Or I can at least guess what may happen and take an action. With the self driving car there is 0 feed back. It's just an empty car that may or may not run me over. Additionally, I also almost got run over by a dude head down driving in the Mission the other day, typing on his phone with both hands as his he just felt comfy having his Tesla do all the work. At the moment all of this feels so sketchy. I hate my life being used for beta testing software lol.
- warning26 3y ago> Even an emoji face that shows facial expressions would help. At one point I saw a proposal to put cartoon eyes on self driving cars that could turn and “look” at you to achieve this effect. Actually seems like a good idea to me!
- maxbond 3y agoMy concern would be the potential for an alignment issue. The agent directs it's eyes to look at you - because it's communicating something accurate about the plan it's formed about how to cross the intersection? Or because it's been trained to look at people when it crosses intersections, but communicating isn't it's true objective? (I'm not trying to imply the agent may fool you into crossing so it can mow you down, the case of "AI incompetence" so to speak is more concerning to me then "AI malice".)
- kirkules 3y ago100% agree with this attitude. A human whose attention is directed at you who uses their eyes for input will turn their eyes to you. The eyes are not just signalers, but the person actually needs to turn them to you to direct their attention to you, in some sense, so this is a good indicator that they have done so.
- CamperBob2 3y agoYou know what's risky? Letting humans drive. 30,000+ deaths a year in the US alone.
- vba616 3y agoIs that risky? There are something like 230 million licensed drivers. So if I was the non-existent "average driver" with no control over risk, it seems to me I should expect a fatal accident roughly once every 100 lifetimes. Or every 100 million miles. I don't perceive the state of the art in autonomous driving as anywhere near that kind of reliability.
- piloto_ciego 3y agoI’m not in favor of how Tesla has used BS marketing to make their product seem more capable than it is, nor am I excited about how all these companies Over promise and under deliver. However… driving is one of the most dangerous things we do. 1/100 people (roughly) die in a traffic related accident in their lives. That’s bonkers and unacceptably high.
- vba616 3y ago>That’s bonkers and unacceptably high. Well, that's your opinion. My estimate of my risk of death is 100/100, so 1/100 does not seem that high to me. What I think could legitimately be called "bonkers and unacceptably high" is the prevalence of medical errors as a cause of death in the US. "Analyzing medical death rate data over an eight-year period, Johns Hopkins patient safety experts have calculated that more than 250,000 deaths per year are due to medical error in the U.S." https://hub.jhu.edu/2016/05/03/medical-errors-third-leading-cause-of-death/ https://hub.jhu.edu/2016/05/03/medical-errors-third-leading-... Since I don't drive drunk, and usually miss rush hour, I do think it's reasonable for me to consider going to the doctor the most dangerous thing I do.
- piloto_ciego 3y agoWhile I agree that the rate of medical mistakes is unacceptably high and the profession is just ridiculous in terms of how they behave from a risk management standpoint… this appears some wild base rate fallacy here, in a big way. The people going to the doctor’s office because they think something is wrong are by definition the more fragile patents. I say this as someone who was misdiagnosed for some time and has dealt with the inertia and bureaucracy of the system. I spent most of my adult life flying bush planes around Alaska and working to try to make things safer in that (horrifically unsafe) industry. Trust me when I say 1/100 is unacceptable and absurd in the extreme and humans are terrible at assessing risk objectively. Just your statement: “I don’t drive drunk and I usually miss rush hour” is indicative of misunderstanding how this works. That’s ok! Everyone does that, I did too until I was trained on specific ways to think about risk and risk mitigation strategies. The risk often doesn’t come down to individual action - it’s links in the chain, or the Swiss cheese model of you prefer and when all the factors line up an accident is more likely. It’s great that you don’t drink and drive, it is also great that you avoid driving at peak hours. But do you actively do risk mitigation before driving? Probably not, more power to you if you do, but probably not. i guess this is a long way to say "something you are exposed to every day multiple times" is way different than a (albeit terrible) system you are exposed to when you are old and infirm and already prone to death. still, everyone thinks the 1/100 won't be them so they feel it isn't a problem… that is a poor way to figure out "what should we do."
- mberning 3y ago“ Cummings: There is no evidence of mitigation. At NHTSA we couldn’t answer the question that you’re less likely to get in a crash—no data.” Compared to what? I find this very hard to believe. Surely forward collision avoidance systems have stopped a number accidents over the years. To say that there is no evidence for it, no data even, seems to suggest that you might have a data problem. Hard to take them seriously when they don’t have good data, yet maintain their very negative position.
- vba616 3y agoThe reasoning you are using seems to be popular - I found something similar on a page about Waymo. If an autonomous driving system clearly prevented or would have prevented a crash that happened to a human, then that's one life saved, is the philosophy. The minimum benefit from everyone using the system is at least one life. But some people do not see it that way. So you are probably at cross purposes with them. I'm sure the skeptics understand there is evidence for what you think proves mitigation - just not for what they think would prove it. I haven't got what it takes right now to describe at length the alternative way of looking at things, plus I could have it (or phrase it) wrong. Maybe someone else will comment.
- sebzim4500 3y agoI see a lot of people arguing from their intuition (or experience in other fields) that what Tesla is doing with their driving assistance features is dangerous. There are millions of Teslas on the road, shouldn't the first step be to find out check whether they crash more often in practice? If not, then what's the big deal?
- scott00 3y agoThe data analysis is not straightforward. You would expect a driver's decision to use Autopilot to be correlated with the ability of the Autopilot to handle the conditions. So just comparing Autopilot accidents per mile to human accidents per mile won't cut it.
- sinuhe69 3y agoIf so then at least, auto pilot is not harmful, is it?
- esjeon 3y agoIt is still harmful in the sense that it gives a false sense of safety. See those lethal accidents where drivers were totally distracted.
- sokoloff 3y agoHow do you conclude that? GP’s premise amounts to “if the overall rate of human crashes per million miles is H and the autopilot rate is A, even if A is half of H, that doesn’t tell you anything about the relative safety if autopilot is engaged during a particularly safe subset of overall driving.”
- sebzim4500 3y agoMy suggestion is to compare Tesla accidents per mile to non-Tesla accidents per mile. If people are being lulled into a false sense of security by AP then this should show up in the data.
- deleted 3y ago[deleted]
- EMCymatics 3y agoI've tried 'self driving' and driving assist. Driving assist is intolerable for me because it can jerk you around even when you have a reason to be close to the line. I don't like fighting for control of the car. I am a huge proponent for good self driving but experiencing driving assist and being a passenger in Tesla cars has made me much more cautious of the current capability.
- asdfman123 3y agoDriving assist is so annoying. It has to be the exact same distance behind the car in front of it, regardless if cars are slowing down ahead or speeding up. No sense of the flow of traffic. Then it has to be exactly in the middle of the road, even in a two lane one with another car coming towards you.
- moi2388 3y agoAs with any automation; it’s not the automation that’s the problem, is that there are still some parts with people in it.
- whiplash451 3y agoFor once, someone with the courage to say « not in my lifetime »