4 ms·

Wow, adjusted for driver age the difference vanishes. That's quite an interesting observation, and it shows the importance of proper statistics. It also raises

Wow, adjusted for driver age the difference vanishes. That's quite an interesting observation, and it shows the importance of proper statistics. It also raises questions about companies producing their own stats, it's often only when someone takes the time to dig into it that we discover this kind of thing. I am not holding my breath waiting for the common consumer to understand statistical critical thinking though. I sometimes catch myself forgetting the due diligence even after many years of forcing myself to think about stats. Perhaps some sort of ombudsman could do this, pointing out when stats are lying to people.

djanogo 5y ago
They should adjust for car-age and price.
ProAm 5y ago
If you have taken a course in statistics you'll know its always possible to make the numbers looks favorable to your cause. It would take several unrelated blind third parties to do this correctly and no public company will ever let that happen, especially Tesla.
- kadoban 5y ago
  They should be forced to let that happen. It's crucial for public safety.
  - ProAm 5y ago
    Public knowledge of safety affects stock price and brand recognition. Will never happen.
    - gretch 5y ago
      This is an arrogant amount of speculation based on nothing more than personal intuition. In fact, there are plenty of examples where public knowledge of safety is published by mandate of the government. Here is an example of FAA flight data and you can see details about incidents across airlines: https://www.faa.gov/data_research/accident_incident/ https://www.faa.gov/data_research/accident_incident/ I'm not going to spend time digging it up, but I'm pretty sure there are just as many sources of data on other important things such as pharmaceuticals e.g. the covid vaccine. So no, just because 'stock price go down' doesn't mean a thing will never happen
  - IanDrake 5y ago
tantalor 5y ago
You could easily have a Simpsons paradox, e.g., it's safer overall, but for any single age group its neutral or even less safe.
kfor 5y ago
I highly recommend How to Lie with Statistics (1954) to learn more about these sorts of misleading stats: https://en.wikipedia.org/wiki/How_to_Lie_with_Statistics https://en.wikipedia.org/wiki/How_to_Lie_with_Statistics
rurp 5y ago
> It also raises questions about companies producing their own stats Companies saying "trust us, we have the data to prove it" while keeping that data secret should be ignored. Especially if their CEO has a long history of blatant dishonesty! I'm most annoyed that people gave any weight to Tesla's safety statistics claims in the first place.
- dahfizz 5y ago
  > Companies saying "trust us, we have the data to prove it" while keeping that data secret should be ignored. While true, I do not think this is a great heuristic. How would the average person know whether the "data" is available? Most people don't even know what "data" really is. They have never heard of CSVs or SQL. We need a better signal for consumers to know what they can and can't trust.
  - rurp 5y ago
    A good journalist will include that context in a story. I know I read at least one article that included these safety claims while also noting that the data was private and unverifiable. I think it's a good default to assume any self-beneficial corporate message is false, absent evidence to the contrary.
  - shihab 5y ago
    An average person doesn't need to directly access the data. As long as journalists or researchers can, an avg person could rely on their analysis/opinion instead.
    - dahfizz 5y ago
      I would certainly not rely on journalists' to actually have a look at the data, let alone to have the technical skills to analyze and report on it. As it is, reporters don't bother mentioning / linking to the data, or mentioning the lack of publicly available data.
      - ceejayoz 5y ago
ajross 5y ago
This isn't "proper statistics", this is P-hacking. Tesla's data is incomplete and bad. But going out and finding confounding variables to confirm your priors isn't the treatment.
contravariant 5y ago
Driver age isn't the factor that explains the difference, accident rate adjusted for age shifts up a similar amount for both Autopilot and regular drivers. What truly makes a difference is the kind of road they're driving on. The accompanying paper provides a figure showing Autopilot is essentially only used on freeways which are inherently safer. Adjusting for that seems to explain most of the difference. Though personally I'd prefer to have just compared the accident rate on freeways for both types rather than this weird weighting method they try to use.

djanogo 5y ago

They should adjust for car-age and price.

ProAm 5y ago

If you have taken a course in statistics you'll know its always possible to make the numbers looks favorable to your cause. It would take several unrelated blind third parties to do this correctly and no public company will ever let that happen, especially Tesla.

kadoban 5y ago
They should be forced to let that happen. It's crucial for public safety.
- ProAm 5y ago
  Public knowledge of safety affects stock price and brand recognition. Will never happen.
  - gretch 5y ago
    This is an arrogant amount of speculation based on nothing more than personal intuition. In fact, there are plenty of examples where public knowledge of safety is published by mandate of the government. Here is an example of FAA flight data and you can see details about incidents across airlines: https://www.faa.gov/data_research/accident_incident/ https://www.faa.gov/data_research/accident_incident/ I'm not going to spend time digging it up, but I'm pretty sure there are just as many sources of data on other important things such as pharmaceuticals e.g. the covid vaccine. So no, just because 'stock price go down' doesn't mean a thing will never happen
- IanDrake 5y ago

tantalor 5y ago

You could easily have a Simpsons paradox, e.g., it's safer overall, but for any single age group its neutral or even less safe.

kfor 5y ago

I highly recommend How to Lie with Statistics (1954) to learn more about these sorts of misleading stats: https://en.wikipedia.org/wiki/How_to_Lie_with_Statistics https://en.wikipedia.org/wiki/How_to_Lie_with_Statistics

rurp 5y ago

> It also raises questions about companies producing their own stats Companies saying "trust us, we have the data to prove it" while keeping that data secret should be ignored. Especially if their CEO has a long history of blatant dishonesty! I'm most annoyed that people gave any weight to Tesla's safety statistics claims in the first place.

dahfizz 5y ago
> Companies saying "trust us, we have the data to prove it" while keeping that data secret should be ignored. While true, I do not think this is a great heuristic. How would the average person know whether the "data" is available? Most people don't even know what "data" really is. They have never heard of CSVs or SQL. We need a better signal for consumers to know what they can and can't trust.
- rurp 5y ago
  A good journalist will include that context in a story. I know I read at least one article that included these safety claims while also noting that the data was private and unverifiable. I think it's a good default to assume any self-beneficial corporate message is false, absent evidence to the contrary.
- shihab 5y ago
  An average person doesn't need to directly access the data. As long as journalists or researchers can, an avg person could rely on their analysis/opinion instead.
  - dahfizz 5y ago
    I would certainly not rely on journalists' to actually have a look at the data, let alone to have the technical skills to analyze and report on it. As it is, reporters don't bother mentioning / linking to the data, or mentioning the lack of publicly available data.
    - ceejayoz 5y ago

ajross 5y ago

This isn't "proper statistics", this is P-hacking. Tesla's data is incomplete and bad. But going out and finding confounding variables to confirm your priors isn't the treatment.

contravariant 5y ago

Driver age isn't the factor that explains the difference, accident rate adjusted for age shifts up a similar amount for both Autopilot and regular drivers. What truly makes a difference is the kind of road they're driving on. The accompanying paper provides a figure showing Autopilot is essentially only used on freeways which are inherently safer. Adjusting for that seems to explain most of the difference. Though personally I'd prefer to have just compared the accident rate on freeways for both types rather than this weird weighting method they try to use.