4 ms·

Is this really what passes for statistical analysis in the investment community? From a biomedical statistics background, this analysis so atrocious that I don'

by et2o 8y ago

Is this really what passes for statistical analysis in the investment community? From a biomedical statistics background, this analysis so atrocious that I don't know where to start.

sdrothrock 8y ago
> From a biomedical statistics background, this analysis so atrocious that I don't know where to start. This sounds like a good basis of a line-by-line refutation or counterargument on Medium. :)
wjnc 8y ago
So please explain. It's quite a long story but with some good finds (data quality problems, obfuscated reporting, probability of much higher casualty rate). I wonder what your statistical rigour makes of this.
- google_censors 8y ago
  Well, including data from China for only Tesla is a joke. That would be like saying "we couldn't find any deaths from Ebola in the United States, so we took the ones in Africa and added them in. Wow, look at how many deaths per year from Ebola occurred in the US compared to deaths from the plague in the US!". It makes absolutely no sense.
  - wjnc 8y ago
    Well it might be. They also include a casualty from NL at face value, while casualties are MUCH rarer here. I agree with the need for discounting of a Chinese casualty, but it's quite common practice to try to get more data points. That is good practice, not bad, and we only seem to disagree with the execution (is a China casualty 1 or .8?) In this case it wouldn't matter the base take-away much.
    - AstralStorm 8y ago
      You have to at least account for differences in base rate of accidents. The typical normalisation would take at least published accident total to synthesise data for the other cars. (By using sales numbers of cars.) What he did there is statistically unacceptable. The medical equivalent would be to compare side effect profile of current best of line treatment done in general populace to results to your new treatment's phase 1 trial results which is done in healthy people only. Then advertise that. Regardless, can we have the raw data please? Are the rates even meaningful when the incident counts are low? Vehicle years are not the right unit as this counts cars that are not being driven. Actual range driven is the right unit.
      - wjnc 8y ago
        I totally agree. But keep in mind the effect "doing the right thing" has on the effect to be estimated. I venture it will not change the gist of the story. Heck, I should contact the author and build a shiny app to demonstrate. Even discounting a Chinese death 50% would put Tesla in the ballpark double the risk of luxury cars. My thought would be that Tesla is more a supersport car (McLaren, etc) in risk (for some drivers, say, male, young or midlife, etc). That one local death in NL, the guy did double the allowed speed on a stretch where I can hardly hit 20% above the allowed speed accounting for traffic lights and local situation. The one Tesla death with the young kid, the same. Don't put maximum traction hundreds of HP in male hands. We tend to use it (sometimes).
        AstralStorm 8y ago
        I suspect the correct use of Chinese data would make error bars overlap, exposing the BS. Moreover, they should use the data where miles drive is available, say, from taxi drivers. Oops, that's 0 out of 0 total? :) For such low incidences, even studentising will not produce right confidence intervals based on normal distribution. Poisson distribution cannot even be used, direct binomial instead, because it is likely that the error due to approximation will be important. (calculate from rule of rare events)
- jpatokal 8y ago
  The biggest problem is that they go through all sorts of hoops to add unreported Tesla fatalities to the data, then compare those against unadjusted other fatalities and report that there are "more than Triple" apples (adjusted) vs orange (unadjusted). The least they could do is run the same stats for both Tesla and another luxury brand (say, Audi), and then compare like to like.
f00_ 8y ago
maybe some censored data problems here? I'm interested in seeing a jupyter notebook analysis on driver fatality rates Does anyone have experience doing survival analysis? I've read a lot of the documentation of CamDavidsonPilon's Lifetimes. maybe i'll give it a shot, seems like the type of thing people would already do to price car insurance Tesla’s mortality rate (41 deaths per million vehicle years) is so much higher than the average luxury car (13 deaths per million vehicle years) that when comparing the two, the difference is hugely statistically significant. The difference is 28 additional deaths per million vehicle years, with a confidence interval of 11 to 63, and a p-value of 0.0001. Binomial confidence intervals for individual cars except Tesla provided by IIHS Binomial confidence intervals for Tesla and groups using Conflint.xls from John Pezzullo so he is doing some kind of hypothesis testing, using the binomial distribution but I wish he would post the data and spreadsheets used to generate everything. probably some p-hacking going on? I'm not really that statistically literate, just a computer engineering dropout if someone could explain that'd be cool
- rgbrenner 8y ago
  China's fatality rate is 8x higher than the US. The author only included it for Tesla. The author then went on google and tried to find more articles. Included whatever he could find only for Tesla. If you're going to go through extra steps to pump Tesla's numbers, then of course they're going to be higher. Duh.

by et2o 8y ago

Is this really what passes for statistical analysis in the investment community? From a biomedical statistics background, this analysis so atrocious that I don't know where to start.

sdrothrock 8y ago

> From a biomedical statistics background, this analysis so atrocious that I don't know where to start. This sounds like a good basis of a line-by-line refutation or counterargument on Medium. :)

wjnc 8y ago

So please explain. It's quite a long story but with some good finds (data quality problems, obfuscated reporting, probability of much higher casualty rate). I wonder what your statistical rigour makes of this.

google_censors 8y ago
Well, including data from China for only Tesla is a joke. That would be like saying "we couldn't find any deaths from Ebola in the United States, so we took the ones in Africa and added them in. Wow, look at how many deaths per year from Ebola occurred in the US compared to deaths from the plague in the US!". It makes absolutely no sense.
- wjnc 8y ago
  Well it might be. They also include a casualty from NL at face value, while casualties are MUCH rarer here. I agree with the need for discounting of a Chinese casualty, but it's quite common practice to try to get more data points. That is good practice, not bad, and we only seem to disagree with the execution (is a China casualty 1 or .8?) In this case it wouldn't matter the base take-away much.
  - AstralStorm 8y ago
    You have to at least account for differences in base rate of accidents. The typical normalisation would take at least published accident total to synthesise data for the other cars. (By using sales numbers of cars.) What he did there is statistically unacceptable. The medical equivalent would be to compare side effect profile of current best of line treatment done in general populace to results to your new treatment's phase 1 trial results which is done in healthy people only. Then advertise that. Regardless, can we have the raw data please? Are the rates even meaningful when the incident counts are low? Vehicle years are not the right unit as this counts cars that are not being driven. Actual range driven is the right unit.
    - wjnc 8y ago
      I totally agree. But keep in mind the effect "doing the right thing" has on the effect to be estimated. I venture it will not change the gist of the story. Heck, I should contact the author and build a shiny app to demonstrate. Even discounting a Chinese death 50% would put Tesla in the ballpark double the risk of luxury cars. My thought would be that Tesla is more a supersport car (McLaren, etc) in risk (for some drivers, say, male, young or midlife, etc). That one local death in NL, the guy did double the allowed speed on a stretch where I can hardly hit 20% above the allowed speed accounting for traffic lights and local situation. The one Tesla death with the young kid, the same. Don't put maximum traction hundreds of HP in male hands. We tend to use it (sometimes).
      - AstralStorm 8y ago
        I suspect the correct use of Chinese data would make error bars overlap, exposing the BS. Moreover, they should use the data where miles drive is available, say, from taxi drivers. Oops, that's 0 out of 0 total? :) For such low incidences, even studentising will not produce right confidence intervals based on normal distribution. Poisson distribution cannot even be used, direct binomial instead, because it is likely that the error due to approximation will be important. (calculate from rule of rare events)
jpatokal 8y ago
The biggest problem is that they go through all sorts of hoops to add unreported Tesla fatalities to the data, then compare those against unadjusted other fatalities and report that there are "more than Triple" apples (adjusted) vs orange (unadjusted). The least they could do is run the same stats for both Tesla and another luxury brand (say, Audi), and then compare like to like.

f00_ 8y ago

maybe some censored data problems here? I'm interested in seeing a jupyter notebook analysis on driver fatality rates Does anyone have experience doing survival analysis? I've read a lot of the documentation of CamDavidsonPilon's Lifetimes. maybe i'll give it a shot, seems like the type of thing people would already do to price car insurance Tesla’s mortality rate (41 deaths per million vehicle years) is so much higher than the average luxury car (13 deaths per million vehicle years) that when comparing the two, the difference is hugely statistically significant. The difference is 28 additional deaths per million vehicle years, with a confidence interval of 11 to 63, and a p-value of 0.0001. Binomial confidence intervals for individual cars except Tesla provided by IIHS Binomial confidence intervals for Tesla and groups using Conflint.xls from John Pezzullo so he is doing some kind of hypothesis testing, using the binomial distribution but I wish he would post the data and spreadsheets used to generate everything. probably some p-hacking going on? I'm not really that statistically literate, just a computer engineering dropout if someone could explain that'd be cool

rgbrenner 8y ago
China's fatality rate is 8x higher than the US. The author only included it for Tesla. The author then went on google and tried to find more articles. Included whatever he could find only for Tesla. If you're going to go through extra steps to pump Tesla's numbers, then of course they're going to be higher. Duh.