8 ms·

In 2017, the feds said Tesla Autopilot cut crashes 40%–that was bogus (2019)

57 points by mcguire 2y ago

mcguire 2y ago
"The National Highway Traffic Safety Administration has egg on its face after a small research and consulting firm called Quality Control Systems produced a devastating critique of a 2017 agency report finding that Tesla's Autopilot reduced crashes by 40 percent ... activation of Autosteer actually increased crash rates by 59 percent."
- jccooper 2y ago
  "So does that mean that Autosteer actually makes crashes 59 percent more likely? Probably not. Those 5,714 vehicles represent only a small portion of Tesla's fleet, and there's no way to know if they're representative. And that's the point: it's reckless to try to draw conclusions from such flawed data. NHTSA should have either asked Tesla for more data or left that calculation out of its report entirely."
  - dhosek 2y ago
    Anecdata: I have a 2022 Prius with radar cruise control and lane guidance. During my first road trip, I discovered that the lane guidance actually did steering through curves (I don’t know how tight it handles, but certainly, it handles the kind of curves that are typical of interstate-level highways.¹ Similarly, the cruise control will slow the car down to a full stop if necessary. Of course, while magical by the standards of my previous cars, it’s still rudimentary. It doesn’t do a good job of catching people merging in front of me and slowing down appropriately and while I often use it on surface streets as well (giving me more consistent acceleration and better gas milage), it will happily drive right through a stop sign or red light if there’s no car in front of me coming to a stop. This puts the automation into the danger valley between manual operation and pushing a button in an elevator. Really, anything short of elevator-level automation, especially without high-level training (consider the accidents that happen with jet autopilot even with that training) is ultimately going to be more dangerous on balance because it can lull the operator into a false sense of complacency. ⸻ 1. It also gets very upset if it detects your hands not on the wheel as I discovered when I decided to see if the car really was steering itself through the curves.
    - Avi-D-coder 2y ago
      My 2023 corolla will steer straight into a curb if a lane gradually exits. until i learned it's habits driving with the assistance was very stressful.
MBCook 2y ago
(2019)
readyman 2y ago
Long live trains and many more of them
deleted 2y ago
[deleted]
bmitc 2y ago
It's alarming just how in bed with the government and regulators Tesla has been. I don't get it. They are and have been a middling startup car company. I don't understand how they've had access to so much government and regulatory corruption.
- kjkjadksj 2y ago
  You mean like all the other middling tech startups in bed with a lazy regulator? Looking at you uber/lyft/lime/bird/doordash et al.
  - bmitc 2y ago
    Yea, it's all alarming. It's so ridiculous. None of these companies provide any real value, but people salivate over performing corruption for them and throwing money at them. I haven't ever understood.
    - recursive 2y ago
      None of them? The auto industry seemed to be convinced that EVs were impossible until Tesla showed them the blueprint.
      - bmitc 2y ago
        I'm not sure that Tesla showed anyone the blueprint of anything. If the air from the EV hype pops tomorrow, Kia and Hyundai could drop their entire EV lines with barely a hiccup. Tesla, not so much. And it was easy for Kia and Hyundai and others to wait for a market to develop. If anything, they used Tesla for market research, whereas Tesla needed to lean on venture capitalist money, government grants and subsidies, and cutting every corner possible to squeak out a handful of models that they can barely keep up with.
        recursive 2y ago
        Personally, I think there's very little hype, and no air to be released. There are plenty of EV manufacturers now. But precious few that do it sustainably at a profit. Is it still only Tesla? I don't know who else is breaking even. Perhaps they still have something to learn.
deleted 2y ago
[deleted]
enragedcacti 2y ago
Related: 2024 NHTSA report on Autopilot: https://static.nhtsa.gov/odi/inv/2022/INCR-EA22002-14496.pdf https://static.nhtsa.gov/odi/inv/2022/INCR-EA22002-14496.pdf "This analysis, conducted before Recall 23V838, indicated that drivers involved in the crashes were not sufficiently engaged in the driving task and that the warnings provided by Autopilot when Autosteer was engaged did not adequately ensure that drivers maintained their attention on the driving task. The drivers were involved in crashes while using Autopilot despite fulfilling Tesla’s pre-recall driver engagement monitoring criteria. Crashes with no or late evasive action attempted by the driver were found across all Tesla hardware versions and crash circumstances."
rossjudson 2y ago
Let's see...a 2019 Ars article, sourcing analysis from a company (QCS) that no longer provides a web site, is claiming that the 2016 data (8 years ago) can be interpreted to show a higher crash rate on autopilot than NHTSA computed at that time. This is, of course, very old news. There's far more data available today. Indeed, this old article says "Tesla could settle the debate by releasing more data". Great! What does current data say? There are two pretty good sources of data. One is Tesla's quarterly vehicle safety reports, and the other is NHTSA's mandatory crash reporting data. Tesla's quarterly data tells a pretty convincing story, from 18Q4 through to 23Q4. The US average accident rate has stayed almost unchanged that whole time, providing a useful baseline. What is completely clear is that miles per crash goes from 3.35 million (18Q4) to 5.39 million (23Q4). Occam's razor provides the simplest explanation: The autopilot software just keeps getting better at avoiding crashes. The currently popular way of poking holes in Tesla's safety reports is to say "of course autopilot miles per crash are better; they're all on the highway and it's unfair to compare". First, anybody who drives a Tesla will tell you that autopilot gets activated all the time off freeways, by all kinds of drivers. Current criticism of the quarterly safety data requires adopting assumptions that bias the data away from the metrics that being presented. They roughly amount to "well, here is a way that this data could be inaccurate. We don't know if it is or isn't inaccurate, but it could be". But that doesn't really matter. Unless the methodology has fundamentally changed (which it hasn't), you only need to compare autopilot data against itself to see that there are huge improvements, and you can also see that at the same time, the US fleet does not improve. For the US fleet, more and more cars have active emergency braking and forward collision warning systems...but this does not seem to be improving the US seasonally adjusted averages much. It might also be interesting to compare Tesla data with cars of similar capabilities...which are mostly from the realms of super- and hyper-cars. How's the miles driven per crash there? I think it's a pretty good trick to deliver supercar performance and have dramatically superior crash stats. I also think these discussions are loaded with all kinds of people who think they are Lake Woebegon, above-average drivers.
- mcguire 2y ago
  From the article: "But now NHTSA's raw data set is available, and, if anything, it appears to contradict Musk's claims. The majority of the vehicles in the Tesla data set suffered from missing data or other problems that made it impossible to say whether the activation of Autosteer increased or decreased the crash rate. But when QCS focused on 5,714 vehicles whose data didn't suffer from these problems, it found that the activation of Autosteer actually increased crash rates by 59 percent." Why are you assuming current data is better than 2016 data?

57 points by mcguire 2y ago

"The National Highway Traffic Safety Administration has egg on its face after a small research and consulting firm called Quality Control Systems produced a devastating critique of a 2017 agency report finding that Tesla's Autopilot reduced crashes by 40 percent ... activation of Autosteer actually increased crash rates by 59 percent."

jccooper 2y ago
"So does that mean that Autosteer actually makes crashes 59 percent more likely? Probably not. Those 5,714 vehicles represent only a small portion of Tesla's fleet, and there's no way to know if they're representative. And that's the point: it's reckless to try to draw conclusions from such flawed data. NHTSA should have either asked Tesla for more data or left that calculation out of its report entirely."
- dhosek 2y ago
  Anecdata: I have a 2022 Prius with radar cruise control and lane guidance. During my first road trip, I discovered that the lane guidance actually did steering through curves (I don’t know how tight it handles, but certainly, it handles the kind of curves that are typical of interstate-level highways.¹ Similarly, the cruise control will slow the car down to a full stop if necessary. Of course, while magical by the standards of my previous cars, it’s still rudimentary. It doesn’t do a good job of catching people merging in front of me and slowing down appropriately and while I often use it on surface streets as well (giving me more consistent acceleration and better gas milage), it will happily drive right through a stop sign or red light if there’s no car in front of me coming to a stop. This puts the automation into the danger valley between manual operation and pushing a button in an elevator. Really, anything short of elevator-level automation, especially without high-level training (consider the accidents that happen with jet autopilot even with that training) is ultimately going to be more dangerous on balance because it can lull the operator into a false sense of complacency. ⸻ 1. It also gets very upset if it detects your hands not on the wheel as I discovered when I decided to see if the car really was steering itself through the curves.
  - Avi-D-coder 2y ago
    My 2023 corolla will steer straight into a curb if a lane gradually exits. until i learned it's habits driving with the assistance was very stressful.

MBCook 2y ago

(2019)

readyman 2y ago

Long live trains and many more of them

deleted 2y ago

[deleted]

bmitc 2y ago

It's alarming just how in bed with the government and regulators Tesla has been. I don't get it. They are and have been a middling startup car company. I don't understand how they've had access to so much government and regulatory corruption.

kjkjadksj 2y ago
You mean like all the other middling tech startups in bed with a lazy regulator? Looking at you uber/lyft/lime/bird/doordash et al.
- bmitc 2y ago
  Yea, it's all alarming. It's so ridiculous. None of these companies provide any real value, but people salivate over performing corruption for them and throwing money at them. I haven't ever understood.
  - recursive 2y ago
    None of them? The auto industry seemed to be convinced that EVs were impossible until Tesla showed them the blueprint.
    - bmitc 2y ago
      I'm not sure that Tesla showed anyone the blueprint of anything. If the air from the EV hype pops tomorrow, Kia and Hyundai could drop their entire EV lines with barely a hiccup. Tesla, not so much. And it was easy for Kia and Hyundai and others to wait for a market to develop. If anything, they used Tesla for market research, whereas Tesla needed to lean on venture capitalist money, government grants and subsidies, and cutting every corner possible to squeak out a handful of models that they can barely keep up with.
      - recursive 2y ago
        Personally, I think there's very little hype, and no air to be released. There are plenty of EV manufacturers now. But precious few that do it sustainably at a profit. Is it still only Tesla? I don't know who else is breaking even. Perhaps they still have something to learn.

deleted 2y ago

[deleted]

enragedcacti 2y ago

Related: 2024 NHTSA report on Autopilot: https://static.nhtsa.gov/odi/inv/2022/INCR-EA22002-14496.pdf https://static.nhtsa.gov/odi/inv/2022/INCR-EA22002-14496.pdf "This analysis, conducted before Recall 23V838, indicated that drivers involved in the crashes were not sufficiently engaged in the driving task and that the warnings provided by Autopilot when Autosteer was engaged did not adequately ensure that drivers maintained their attention on the driving task. The drivers were involved in crashes while using Autopilot despite fulfilling Tesla’s pre-recall driver engagement monitoring criteria. Crashes with no or late evasive action attempted by the driver were found across all Tesla hardware versions and crash circumstances."

rossjudson 2y ago

Let's see...a 2019 Ars article, sourcing analysis from a company (QCS) that no longer provides a web site, is claiming that the 2016 data (8 years ago) can be interpreted to show a higher crash rate on autopilot than NHTSA computed at that time. This is, of course, very old news. There's far more data available today. Indeed, this old article says "Tesla could settle the debate by releasing more data". Great! What does current data say? There are two pretty good sources of data. One is Tesla's quarterly vehicle safety reports, and the other is NHTSA's mandatory crash reporting data. Tesla's quarterly data tells a pretty convincing story, from 18Q4 through to 23Q4. The US average accident rate has stayed almost unchanged that whole time, providing a useful baseline. What is completely clear is that miles per crash goes from 3.35 million (18Q4) to 5.39 million (23Q4). Occam's razor provides the simplest explanation: The autopilot software just keeps getting better at avoiding crashes. The currently popular way of poking holes in Tesla's safety reports is to say "of course autopilot miles per crash are better; they're all on the highway and it's unfair to compare". First, anybody who drives a Tesla will tell you that autopilot gets activated all the time off freeways, by all kinds of drivers. Current criticism of the quarterly safety data requires adopting assumptions that bias the data away from the metrics that being presented. They roughly amount to "well, here is a way that this data could be inaccurate. We don't know if it is or isn't inaccurate, but it could be". But that doesn't really matter. Unless the methodology has fundamentally changed (which it hasn't), you only need to compare autopilot data against itself to see that there are huge improvements, and you can also see that at the same time, the US fleet does not improve. For the US fleet, more and more cars have active emergency braking and forward collision warning systems...but this does not seem to be improving the US seasonally adjusted averages much. It might also be interesting to compare Tesla data with cars of similar capabilities...which are mostly from the realms of super- and hyper-cars. How's the miles driven per crash there? I think it's a pretty good trick to deliver supercar performance and have dramatically superior crash stats. I also think these discussions are loaded with all kinds of people who think they are Lake Woebegon, above-average drivers.

mcguire 2y ago
From the article: "But now NHTSA's raw data set is available, and, if anything, it appears to contradict Musk's claims. The majority of the vehicles in the Tesla data set suffered from missing data or other problems that made it impossible to say whether the activation of Autosteer increased or decreased the crash rate. But when QCS focused on 5,714 vehicles whose data didn't suffer from these problems, it found that the activation of Autosteer actually increased crash rates by 59 percent." Why are you assuming current data is better than 2016 data?