38 ms·

I do believe that Teslas are quite safe despite the high-profile failures, but I'm very unconvinced by a stat that compares accidents per mile using autopilot t

by Imnimo 4y ago

I do believe that Teslas are quite safe despite the high-profile failures, but I'm very unconvinced by a stat that compares accidents per mile using autopilot to accidents per mile not using autopilot. Those are not comparable miles - people don't uniformly use autopilot across all driving conditions. Imagine you had a subsystem that did absolutely nothing, but people turned it on when driving on empty highways. It's accidents per mile number would look great!

FireBeyond 4y ago
This needs to be emphasized so much more. Short of "not driving", human drivers don't have the option of "well, this is less than optimal, I can't do well, so I won't try". FSD absolutely does. Disengages, or is not even engaged in the first place. I wonder Tesla's FSD safety / disengagement stats for December to February in Pittsburgh, for example.
- gumby 4y ago
  > Short of "not driving", human drivers don't have the option of "well, this is less than optimal, I can't do well, so I won't try". They sure do, and will stop when things are too complex for them (e.g. bad weather).
- ok_dad 4y ago
  Yes, how about a stat for, "accidents/incidents which were proceeded by AutoPilot disengagement within the past ~5 seconds."
  - chroma 4y ago
    That's what Tesla is counting. From the methodology section of the safety report: > To ensure our statistics are conservative, we count any crash in which Autopilot was deactivated within 5 seconds before impact, and we count all crashes in which the incident alert indicated an airbag or other active restraint deployed.
    - ok_dad 4y ago
      That's what I get for not reading the source material! Thanks for correcting me.
    - FireBeyond 4y ago
      Yeah, the "no active restraint deployed" thing is horseshit, and Tesla knows it, too. I go to manufacturer training sessions on these systems maybe once a year as a firefighter-paramedic, and there are multiple criteria for deployments these days - you can have a significant collision and if the system determines its better for the occupants not to deploy, then it doesn't. According to Tesla, that's not an accident. For FSD evaluation purposes? It sure as hell is. But it's no longer counted.
      - chroma 4y ago
        I was responding to the point about counting crashes after autopilot disengagement, not the criteria for what counts as a crash.
  - dbt00 4y ago
    Right, like when they tried to blame Walter Huang for FSD veering directly into a barrier US-101/CA-85. "Autopilot was disengaged when it crashed." Yes, because the driver grabbed the wheel when autopilot veered directly into a gore point.
    - FireBeyond 4y ago
      That wasn't the first time they'd done that, either. Prior to that, another fatal collision, released a statement, "The vehicle had warned driver prior to collision to pay attention to driving". They, strangely, didn't mention that this warning was issued for a single steering wheel inactivity failure, and that that happened _eighteen_ _minutes_ prior to the collision.
seeekr 4y ago
I don't disagree that it's not the best of comparisons (and I wonder if a better one could be imagined + implemented...). But still, it's not like we turn on Autopilot/FSD only "on empty highways", far from it! Certainly it's a tool where the user needs to learn its strengths and weaknesses and use it accordingly, but it is useful in so many more situations than not, that it's also not a terrible or meaningless comparison to make! Anecdata: Almost all (95%?) of my highway driving (Europe) is on Autopilot. I don't even enjoy doing the driving myself any more in those situations where I know that Autopilot is doing a pretty good job. In particular, Autopilot does a better job than I can in conditions of heavy snow / rain / otherwise poor visibility conditions. I feel a lot safer being the operator than the driver in those instances! (The alternative would often be to slow down by a significant amount, and/or use up more of my focus/attention, leading to either less safe driving or forced breaks.)
- enragedcacti 4y ago
  > and I wonder if a better one could be imagined + implemented... There is a ton of data using industry defined conditions and a ton of research has gone into determining the types of conditions that can affect accident rates. The only reason we can't compare is because Tesla only releases what we see in the link above. [1] Best guess estimates of normalizing autopilot data against the average mix of highway/city driving finds that AP's safety advantage effectively disappears. Of course this is rough and Tesla likely has much better data that they are choosing not to share. [2] [1] https://www.iihs.org/topics/fatality-statistics/detail/urban-rural-comparison#where-crashes-occur https://www.iihs.org/topics/fatality-statistics/detail/urban... [2] https://twitter.com/NoahGoodall/status/1489291552845357058 https://twitter.com/NoahGoodall/status/1489291552845357058
- femto113 4y ago
  It's not at all difficult to imagine many other measurements: minutes driven, type of road driven on, speed at time of crash, whether another vehicle was involved, etc. As a data analyst what I really want though is more granular data so we can figure out whether the published metrics are being cherry-picked.
- alphabettsy 4y ago
AtlasBarfed 4y ago
And I assume that autopilot is used primarily on highways, in either stop and go or steady flow situations. Really they should know this, because the autopilot system should be identifying it based on speed limits and detected type of road. Of course that assumes some ability to look into a black box, but also location tracking should provide it too.
oneoff786 4y ago
This is true, but there’s a counterpoint here too. If the autopilot takes all of the “easy miles” then you should expect to see a spike or accidents concentrated on the non autopilot “hard miles”. Since we don’t see that spike, I’m inclined to believe that the effect size of the highway driving preference is relatively small.
FireBeyond 4y ago
What's also egregious about this is that Tesla has now "decided" that if active restraints or airbags are not deployed, that incident "does not count going forward". And as an automobile manufacturer, I cannot believe in good conscience that Tesla is unaware that active restraint and airbag systems of today are far more nuanced and weigh multiple criteria when deciding to deploy, versus "speed exceeds X mph, deploy" of old. You can have a very significant incident (two that I've witnessed recently involve vehicles into stationary objects at ~30mph) without airbag deployment. But if that was a Tesla in FSD that hit something at 30mph and didn't deploy airbags, well, that's not an accident according to Tesla. That also doesn't account for "incident was so catastrophic that restraint systems could not deploy", also "not counted" by Tesla. Or just as egregious, "systems failed to deploy for any reason up to and including poor assembly line quality control", also not an accident and "not counted".

by Imnimo 4y ago

FireBeyond 4y ago

This needs to be emphasized so much more. Short of "not driving", human drivers don't have the option of "well, this is less than optimal, I can't do well, so I won't try". FSD absolutely does. Disengages, or is not even engaged in the first place. I wonder Tesla's FSD safety / disengagement stats for December to February in Pittsburgh, for example.

gumby 4y ago
> Short of "not driving", human drivers don't have the option of "well, this is less than optimal, I can't do well, so I won't try". They sure do, and will stop when things are too complex for them (e.g. bad weather).
ok_dad 4y ago
Yes, how about a stat for, "accidents/incidents which were proceeded by AutoPilot disengagement within the past ~5 seconds."
- chroma 4y ago
  That's what Tesla is counting. From the methodology section of the safety report: > To ensure our statistics are conservative, we count any crash in which Autopilot was deactivated within 5 seconds before impact, and we count all crashes in which the incident alert indicated an airbag or other active restraint deployed.
  - ok_dad 4y ago
    That's what I get for not reading the source material! Thanks for correcting me.
  - FireBeyond 4y ago
    Yeah, the "no active restraint deployed" thing is horseshit, and Tesla knows it, too. I go to manufacturer training sessions on these systems maybe once a year as a firefighter-paramedic, and there are multiple criteria for deployments these days - you can have a significant collision and if the system determines its better for the occupants not to deploy, then it doesn't. According to Tesla, that's not an accident. For FSD evaluation purposes? It sure as hell is. But it's no longer counted.
    - chroma 4y ago
      I was responding to the point about counting crashes after autopilot disengagement, not the criteria for what counts as a crash.
- dbt00 4y ago
  Right, like when they tried to blame Walter Huang for FSD veering directly into a barrier US-101/CA-85. "Autopilot was disengaged when it crashed." Yes, because the driver grabbed the wheel when autopilot veered directly into a gore point.
  - FireBeyond 4y ago
    That wasn't the first time they'd done that, either. Prior to that, another fatal collision, released a statement, "The vehicle had warned driver prior to collision to pay attention to driving". They, strangely, didn't mention that this warning was issued for a single steering wheel inactivity failure, and that that happened _eighteen_ _minutes_ prior to the collision.

seeekr 4y ago

I don't disagree that it's not the best of comparisons (and I wonder if a better one could be imagined + implemented...). But still, it's not like we turn on Autopilot/FSD only "on empty highways", far from it! Certainly it's a tool where the user needs to learn its strengths and weaknesses and use it accordingly, but it is useful in so many more situations than not, that it's also not a terrible or meaningless comparison to make! Anecdata: Almost all (95%?) of my highway driving (Europe) is on Autopilot. I don't even enjoy doing the driving myself any more in those situations where I know that Autopilot is doing a pretty good job. In particular, Autopilot does a better job than I can in conditions of heavy snow / rain / otherwise poor visibility conditions. I feel a lot safer being the operator than the driver in those instances! (The alternative would often be to slow down by a significant amount, and/or use up more of my focus/attention, leading to either less safe driving or forced breaks.)

enragedcacti 4y ago
> and I wonder if a better one could be imagined + implemented... There is a ton of data using industry defined conditions and a ton of research has gone into determining the types of conditions that can affect accident rates. The only reason we can't compare is because Tesla only releases what we see in the link above. [1] Best guess estimates of normalizing autopilot data against the average mix of highway/city driving finds that AP's safety advantage effectively disappears. Of course this is rough and Tesla likely has much better data that they are choosing not to share. [2] [1] https://www.iihs.org/topics/fatality-statistics/detail/urban-rural-comparison#where-crashes-occur https://www.iihs.org/topics/fatality-statistics/detail/urban... [2] https://twitter.com/NoahGoodall/status/1489291552845357058 https://twitter.com/NoahGoodall/status/1489291552845357058
femto113 4y ago
It's not at all difficult to imagine many other measurements: minutes driven, type of road driven on, speed at time of crash, whether another vehicle was involved, etc. As a data analyst what I really want though is more granular data so we can figure out whether the published metrics are being cherry-picked.
alphabettsy 4y ago

AtlasBarfed 4y ago

And I assume that autopilot is used primarily on highways, in either stop and go or steady flow situations. Really they should know this, because the autopilot system should be identifying it based on speed limits and detected type of road. Of course that assumes some ability to look into a black box, but also location tracking should provide it too.

oneoff786 4y ago

This is true, but there’s a counterpoint here too. If the autopilot takes all of the “easy miles” then you should expect to see a spike or accidents concentrated on the non autopilot “hard miles”. Since we don’t see that spike, I’m inclined to believe that the effect size of the highway driving preference is relatively small.

FireBeyond 4y ago

What's also egregious about this is that Tesla has now "decided" that if active restraints or airbags are not deployed, that incident "does not count going forward". And as an automobile manufacturer, I cannot believe in good conscience that Tesla is unaware that active restraint and airbag systems of today are far more nuanced and weigh multiple criteria when deciding to deploy, versus "speed exceeds X mph, deploy" of old. You can have a very significant incident (two that I've witnessed recently involve vehicles into stationary objects at ~30mph) without airbag deployment. But if that was a Tesla in FSD that hit something at 30mph and didn't deploy airbags, well, that's not an accident according to Tesla. That also doesn't account for "incident was so catastrophic that restraint systems could not deploy", also "not counted" by Tesla. Or just as egregious, "systems failed to deploy for any reason up to and including poor assembly line quality control", also not an accident and "not counted".