8 ms·
Underlying study is here. [1] Full disclosure, I co-wrote the piece for The Daily Beast [2] that originally suggested that Tesla's methodology was seriously fla
by gnicholas 5y ago
Underlying study is here. [1] Full disclosure, I co-wrote the piece for The Daily Beast [2] that originally suggested that Tesla's methodology was seriously flawed. I am not a Tesla-hater though — I just thought it was odd that the company was playing quite so fast and loose with their safety claims.
1: https://engrxiv.org/preprint/view/1973/3986 https://engrxiv.org/preprint/view/1973/3986
2: https://www.thedailybeast.com/how-tesla-and-elon-musk-exaggeraged-safety-claims-about-autopilot-and-cars https://www.thedailybeast.com/how-tesla-and-elon-musk-exagge...
- Natsu 5y agoThe study says this: > An estimated 15% of injury crashes and 24% of property damage-only crashes are never reported to police (M. Davis and Company, Inc., 2015), while9% of injury crashes and 24% of property damage-only crashes are reported but not logged (Blincoe et al., 2015). Even with robust data, establishing the statistical significance of automated vehicle safety can be expensive. Kalra and Paddock (2016)demonstrated that establishing that an AV has a fatal crash rate equivalent to the national average with 95% confidence would require driving a fleet of 100 vehicles continuously for 12.5 years. I presume there's an adjustment for this in the figures you compare to, but I am having trouble finding it, though that may be my own failure so I wanted to ask if you could help me find how that's adjusted for?
- TedDoesntTalk 5y agoWhy would injury crashes go unreported?
- sokoloff 5y agoIf I had a single-car crash with minor injuries, why would I report it? It seems like there’s limited upside to me as the (obviously at-fault) driver. Go get medical care and go get my car fixed. (I drive cars cheap enough to replace and so do not insure them against collisions that are my fault.) Drivers driving without a valid license, without insurance, under the influence, wanted by the law, etc would have all kinds of reasons to not report a crash.
- deleted 5y ago[deleted]
- vanattab 5y agoI got hit a few weeks ago while walking my dog (dog was fine). I thought I had just bruised my tailbone so I waved the driver on after a short chat and never reported it. In hindsight I wish I had as it seems like my tailbone was fractured and not just bruised but it definitely happens.
- LorenPechtel 5y agoPolice responding to the actual crash scene, useful. Police report filed after the fact, only if there's some need for said report. And not all injuries are immediately apparent. 3 1/2 years ago I got a first-person view of what it would be like to be on the receiving end of a PIT maneuver courtesy of a woman who didn't look adequately. (Yes, when you looked left the road was empty--I saw that also and turned into that empty spot the block before you!) At the time the only injury I was aware of was a very minor scrape, not even worthy of a bandaid. Later I discovered a pulled muscle--it only hurt when I did certain things. I also used to know a woman who thought it was nothing more than a fender-bender, went to the doc a couple of days later because her neck was bothering her. Doc sent her straight to the hospital--incomplete cervical fracture, one wrong move and she could have dropped dead.
- tempnow987 5y agoAfter college I lived with roommates for a while. One went on a joyride of sorts and damaged a bunch of parking meters, crashed and injured themselves. They had a friend drive them home and repaired the vehicle with no reporting. In some cities currently even if you want to have police respond to a traffic accident, you'd need to wait a VERY long time - they will tell you that unless you need EMS to try and get a safe spot and call for a tow. Some folks aren't willing to wait hours for police to show up to do what, take a report?
- rootusrootus 5y agoDepending on the state, though, reporting all injury crashes and any property damage over a certain (fairly low) amount is required by law. Not necessarily to the police, but to the DMV. I assume that statistically that would count in the 'reported to police' column.
- rgmerk 5y agoKalra and Paddock's analysis is correct as far as it goes, but it's easy to misuse to claim that we couldn't feasibly demonstrate an autonomous vehicle is acceptably safe. It never got accepted for publication because the reviewers didn't think it was significant enough, but I wrote a paper presenting an alternative way to analyse autonomous vehicle reliability data (you'll have to take my word for it, but the reviewers didn't quibble about the results presented, just whether it was important): https://arxiv.org/pdf/1812.08901.pdf https://arxiv.org/pdf/1812.08901.pdf The major technical contribution was demonstrating that a class of mathematical models called software reliability growth models are a reasonable way to estimate the "anomalous behaviour" rates of an AV system as bugs are found and fixed. My second point was that using fatal accidents as a way to measure safety is not necessary; you can use more common events such as non-fatal accidents, or in the case of AV systems in development, even the number of occasions the safety driver overrode the automatic control.
- deleted 5y ago[deleted]
- oldgradstudent 5y agoI am surprised a scholar would take these reports without a massive grain of salt. > In this study, all disengagement events listed in the company reports were included in the analysis. It is likely that in some cases, disengagements occurred as a precaution by the safety driver, and no adverse event would have resulted if the AV continued in autonomous mode. Take a look at [1]. It says that: > Waymo has developed a robust process to collect, analyze and evaluate disengages for this report. We set disengagement thresholds conservatively for our public road testing. The vast majority of disengagements are not related to safety. Our test drivers routinely transition into and out of autonomous mode many times throughout the day, and the self-driving vehicle’s computer hands over control to the driver in many situations that do not involve a failure of the autonomous technology and do not require an immediate takeover of control by the driver. > To help evaluate the safety significance of disengagements, Waymo employs a powerful simulator program. In Waymo’s simulation, our team can “replay” each incident and predict the behavior of our self-driving car if the driver had not taken control of it, as well as the behavior and positions of other road users in the vicinity (such as pedestrians, cyclists, and other vehicles). Our engineers use this data to refine and improve the software to ensure the self-driving car performs safely. So effectively, Waymo has secret process, which may change all the time, to decide which disengagements to report and which not to report. There is no way to tell whether the car would have slightly bumped into another car, or killed a child if not disengaged. Their reports to the DMV do not constitute data, they are PR. Not trying to single out Waymo, they all do that. [1] https://web.archive.org/web/20190531071448/https://www.dmv.ca.gov/portal/wcm/connect/42aff875-7ab1-4115-a72a-97f6f24b23cc/Waymofull.pdf?MOD=AJPERES&CVID= https://web.archive.org/web/20190531071448/https://www.dmv.c...
- jacquesm 5y ago> I just thought it was odd that the company was playing quite so fast and loose with their safety claims. That's because it's marketing, not science.
- sandworm101 5y agoAnd because it's Tesla, not Honda. Every car company does marketing. Few make over-the-top safety claims about their products. This is the company that decided to have its cars not stop at stop signs. Normal rules don't apply to Tesla. https://www.bbc.com/news/technology-60230072 https://www.bbc.com/news/technology-60230072 "In recall documents, the electric vehicle maker notes that if certain conditions are met, the "rolling stop" feature is designed to allow the vehicle to travel through all-way-stop intersections at up to 5.6mph without coming to a complete stop."
- omgwtfbyobbq 5y agoIt's not like Honda doesn't engage in puffery. Most retailers make over-the-top claims regarding their products, including safety. https://www.latimes.com/business/autos/la-fi-honda-fine-20150109-story.html https://www.latimes.com/business/autos/la-fi-honda-fine-2015...
- NikolaNovak 5y agoI mean... I am "Tesla Hater" in that I don't like the direction they're proceeding in; but every human being I ever met in real actual life does rolling stops under some conditions. Last time I can confidently state that "I never rolled through a stop at very slow speed" is when I was 17, and only NOW do I realize how much of an annoying bum I was :P. So if a car does a rolling stop through a stop sign when there are no other cars obstacles or people, my thought is 1. yes that's a breach of law as written 2. It's what everybody else safely does. Kind of like in many jurisdictions, police will gently talk to you if you drive 60kph on the freeway, or even if you're doing 90kph in the left lane. Yes it's legal, no it isn't safe.
- sandworm101 5y ago
- andrepd 5y ago>I am not a Tesla-hater though — I just thought it was odd that the company was playing quite so fast and loose with their safety claims. If you're still surprised by this you've not been paying attention to how Telsa operates.
- mdoms 5y agoHe never said he was surprised.
- maxdo 5y ago
- simplyaccont 5y ago>Facts: Tesla was the first to introduce such wide range of safety features in every car they sold. They don't make safety a premium feature. They invest in own crash test facilities. Only Volvo is kind of on par. Volvo built their crash test facilities before Tesla came to existence.
- FireBeyond 5y ago> Facts: Tesla was the first to introduce such wide range of safety features in every car they sold. Please itemize these "wide range of safety features" for us that other manufacturers do not have or did not have until Tesla innovated them. > They invest in own crash test facilities. Only Volvo is kind of on par. Audi owns one, for just one.
- deleted 5y ago[deleted]
- maxdo 5y agoSure, https://www.tesla.com/safety https://www.tesla.com/safety https://www.youtube.com/watch?v=9KR2N_Q8ep8 https://www.youtube.com/watch?v=9KR2N_Q8ep8 Tesla roll out cars in generations. So just take a $40k Model 3/Y platform was introduced in 2017 and compare it with any car been introduced that year. My friend bought a BMW in 2019 for similar money, it doesn't have half of that. Maybe you can have them but you have to buy extra packages for extra money. Tesla don't sell safety features for extra price. The only extra feature is Autopilot. But hardware, camera's , lane departure avoidance, blind spot collision warning etc in every car. You can even take such a simple feature as forward collision warning system. When you drive it is super accurate it uses the same data and models from Autopilot to predict when there is a crash. If your slowing down dynamic is not good or the car in front/back change it dynamics it will warn you immediately. Only this feature saved me several times from collision. I'm not a good driver. As I said in 2022 there are many car brands that catching up in some cases they doing better. And this is good. Human is a bad, constantly distracted driver. Technology is the only way to make our life safer.
- contravariant 5y agoSo to summarize: - Freeways account for 93% of distance travelled with Autopilot - Freeways account for 28% of the distance travelled by regular drivers. - "vehicles on non-freeways crashed 2.01 times more often per mile" a quick back of the envelop calculation suggests merely driving on the freeway lowers accidents per mile by ~40%, which accounts for most of the difference between Autopilot and regular drivers.
- avs733 5y agoso in sum, Thomas Bayes just kicked Tesla square in the ass?
- nwiswell 5y agoSquare in the frunk.
- adolph 5y agoThere are additional confounding variables noted in the paper: - Age of driver - Specific highways used (congestion, road conditions) - What is classified as a highway Overall this is pretty good news for "autopilot" systems. It gives some humans an ego-boost as they clutch to steering wheels like John Henry. It gives others interested in a more relaxed driving experience the confidence that the computer isn't going to kill them any worse than they already do.
- AtlasBarfed 5y agoAnd you know what? All I want out of the first fifteen years of self-driving is a really good highway autopilot. As in one I can hopefully sleep at the wheel on a late night trip between midwestern US cities. John Henryism can navigate the suburbs, stroads, and drivethrus. If they achieve a backup safety system that monitors the humans, well, ok great. But right now, I want long distance automation, and I can handle the messy last mile. Highways are much easier to get uniform infrastructure convergence: some sensors, PSA warnings, better signs + markings + construction notices, etc. I think Uber making as big a splash at it did sidetracked this low hanging fruit. IMO that was where the bootstrap bucks were, because you would have gotten freight/shipping to pay the cost of the next gen software, and would have provided the impetus for the convergent infrastructure. Alas, they are obsessed with robotaxis.
- deleted 5y ago[deleted]
- RyEgswuCsn 5y agoSo it sounds like it's Simpson's paradox [1] at work. I had always wondered why many people are slow to accept self-driving cars despite the claims of them being statistically safer. I think that explains it --- to make consumers confident about the self-driving technologies, it is not enough for them to be able to handle the "easy" kind of driving (e.g. on highway and under relatively good visibility conditions) better than humans; they need to demonstrate that they can drive better than humans in the most challenging driving environments too. I guess this is the classic scenario where human intuition is defeated by carefully presented statistics. [1]: https://en.wikipedia.org/wiki/Simpson%27s_paradox https://en.wikipedia.org/wiki/Simpson%27s_paradox
- wilde 5y agoI think traditional auto makers have done a much better job of communicating to consumers what the tech can actually do. We don’t have self driving cars (which implies that they can drive in those harder conditions). We have cars that are good at lane keeping on freeways. I wish Tesla would celebrate that victory rather than double down on lies.
- deleted 5y ago[deleted]
- oblio 5y agoThe thing is, you don't even need Tesla "self driving" for that. Adaptive cruise control, tech we've had for what, 20+ years? is more than enough for most highways, it takes out ~80% of the stress/boredom of driving, especially since lane changes are minimal and curves are gentle. The hard parts of driving (aka not highways) are the real problem, as expected. We solved the easy parts in 2000 and who knows if we'll solve the hard parts by 2050.
- CSSer 5y agoI have a car with adaptive cruise control and lane assist, and most of my morning commute (on days when I go into the office) is driven on the highway. Most of the time I don't turn it on because I'm in a self-perceived hurry to get to work, so I forget. You know what would be really nice? Something that detects when it's safe and reasonable to suggest turning on adaptive cruise control so I use it more. Maybe something like detecting I've driven a few miles at a consistent rate of speed above a certain threshold. The hard parts are definitely the problem, but there are lots of easy wins left on the table too.
- julianz 5y agoSo that Daily Beast article has been up for more than 5 years with nobody fixing the horrible typo in the headline? Wow.
- rosndo 5y agoDaily Beast, not the Economist.
- ricardobeat 5y ago> naturalistic driving study of 28 vehicles The study seems to extrapolate from this. Tesla has something like 10 billion miles total driven on Autopilot, on a fleet of a million cars.
- dsl 5y agoThis is the dataset: https://insight.shrp2nds.us/documents/shrp2_background.pdf https://insight.shrp2nds.us/documents/shrp2_background.pdf It covers 5.4 million trips in 3,400 vehicles with digitally collected records and vehicle data and has been independently validated by the Virginia Tech Transportation Institute and Department of Transportation.
- ricardobeat 5y agoThe 93% highway driving figure, which is used to normalize Tesla numbers, and then compare agains the NHS dataset, is coming from that 28 Tesla cars study from 2018, or did I read that wrong?
- Robotbeat 5y agoEven after adjusting, it does look like autopilot is slightly safer (although not a strong enough effect to say for sure). It doesn't appear to be worse, so good enough to allow it, IMHO. I actually wonder if there's another effect: Drivers only eligible for beta FSD if they meet a super high safety score (which isn't a perfect metric, but probably better than nothing). That incentivizes good driving even when FSD not engaged. Of course, that's a recent thing and hasn't shown up in data.
- ubercore 5y agoAfter correcting for age, it becomes less safe