6 ms·

It says as much in the official Tesla press release: NHTSA does not publish a star rating above 5, however safety levels better than 5 stars are captured in th

It says as much in the official Tesla press release: NHTSA does not publish a star rating above 5, however safety levels better than 5 stars are captured in the overall Vehicle Safety Score (VSS) provided to manufacturers, where the Model S achieved a new combined record of 5.4 stars. You can't blame Tesla for taking that number and running with it, given that (1) modeling car safety from two seaters to SUVs on a range from 1 to 5 stars is ridiculous and (2) the star systems suffer from the same problems as contemporary game and movie reviews: 99% of all evaluated products land between 3 and 5 stars, where 3 then becomes the new horrible, compressing the already low entropy of a star value further.

apalmer 13y ago
Nothing wrong with the 5 star rating system as far as it goes... the marketing 'usefulness' of a measure is not the be all and end all, in that regards it is very different from video game and movie reviews.
jlgreco 13y ago
This makes me wonder, do they adjust the rating system over time, and if so, are they doing it frequently enough? You can buy five star cars today, and I assume you could buy 5-star cars 15 years ago, but 5 star cars from 15 years ago wouldn't get rated as highly today, right? Maybe they need to be more aggressive with their adjustment so that the minimum expected really is 1 star.
- gamblor956 13y ago
  Safety ratings are adjusted annually, so a car that was rated 5-stars a decade ago would probably be lucky to get 2 stars today.
  - chiph 13y ago
    Assuming it could even be sold, if it were lacking now-mandatory equipment (tire-pressure sensors, etc).
deleted 13y ago
[deleted]
smackfu 13y ago
The VSS is not a star score though. It looks like Tesla got a 0.43 VSS. Tesla was the one who converted that to a 5.4, by making up a methodology, and their wording on that is definitely a bit unclear.
CatMtKing 13y ago
I demand p-values! sum(p(accident type) * p(estimated of surviving corresponding safety test))
- throwaway_yy2Di 13y ago
  Nitpick: Those things are not p-values.
- Strilanc 13y ago
  P-values aren't probabilities you should multiply or add. At best you're computing odds according to the null hypothesis (probably not what you indented). At worst it's as meaningless as adding meters to seconds.
- ars 13y ago
  They have those. Maybe not exactly in the format you specify, but there are statistics for various star ratings and accident survivability.
lmkg 13y ago
> (2) the star systems suffer from the same problems as contemporary game and movie reviews: 99% of all evaluated products land between 3 and 5 stars, where 3 then becomes the new horrible, compressing the already low entropy of a star value further. That's not a bad thing. We wanted safer cars. We started measuring safety, and publishing the measurements. As a result, safety improved. Working As Intended.
gamblor956 13y ago
The threshold for each star is increased each year, based on the available safety features and safety performance of cars. A car that received 5 stars under the 2000 rating system would receive only 2 stars today because it lacks some of the now-required safety features. 3 is the new horrible simply because that is the lowest that a commercially viable car can go--Chevy and Subaru have released sub-3 cars in the past and suffered for it with poor sales. They and others learned that consumers wouldn't buy cars that were rated below-average safety-wise and have vastly increased R&D spending on safety features. That's a good thing for consumers.
- ars 13y ago
  > rated below-average It better not be a real average if everyone is always above average! You mean to say poorly rated, not below average, since it's not an average.
stcredzero 13y ago
Here is how star rating actually works: For a truly excellent product, the distribution should approximate a power law graph: XXXXXXXXXXXXXXXX XXXXXX XX X X If you see something like these: XXXXXXX X X X X X X X X XXXXX XXXXXXXXXXXXXXXX XXXXXX XXX X XXXXXXXX Then ask yourself if the sample size makes this plausible, in which case the star rating isn't worth much, because there isn't enough data. Ask yourself if it's possible that someone has been gaming the system for or against. The uppercase gamma-shaped graph is either someone gaming the system or it's a power law graph hidden by aliasing. (In which case, the thing is of godlike quality.) We should do away with star ratings, and instead just present an aggregate "rating." Genuine customer ratings are valuable, but are almost immediately gamed by cheaters. If I were Apple or Amazon, I would build stealth captchas and "honeypot" traps for ratings manipulation, and figure those into the rating. Be sure to separate detection from consequences by a random time interval of at least a few days. This makes the "development cost" for the cheaters much higher. The problem with star ratings is obviously what PG refers to as "schlepping". Is anyone tackling this already? EDIT: I would surmise that AirBNB is somewhat tackling this problem already.

apalmer 13y ago

Nothing wrong with the 5 star rating system as far as it goes... the marketing 'usefulness' of a measure is not the be all and end all, in that regards it is very different from video game and movie reviews.

jlgreco 13y ago

This makes me wonder, do they adjust the rating system over time, and if so, are they doing it frequently enough? You can buy five star cars today, and I assume you could buy 5-star cars 15 years ago, but 5 star cars from 15 years ago wouldn't get rated as highly today, right? Maybe they need to be more aggressive with their adjustment so that the minimum expected really is 1 star.

gamblor956 13y ago
Safety ratings are adjusted annually, so a car that was rated 5-stars a decade ago would probably be lucky to get 2 stars today.
- chiph 13y ago
  Assuming it could even be sold, if it were lacking now-mandatory equipment (tire-pressure sensors, etc).

deleted 13y ago

[deleted]

smackfu 13y ago

The VSS is not a star score though. It looks like Tesla got a 0.43 VSS. Tesla was the one who converted that to a 5.4, by making up a methodology, and their wording on that is definitely a bit unclear.

CatMtKing 13y ago

I demand p-values! sum(p(accident type) * p(estimated of surviving corresponding safety test))

throwaway_yy2Di 13y ago
Nitpick: Those things are not p-values.
Strilanc 13y ago
P-values aren't probabilities you should multiply or add. At best you're computing odds according to the null hypothesis (probably not what you indented). At worst it's as meaningless as adding meters to seconds.
ars 13y ago
They have those. Maybe not exactly in the format you specify, but there are statistics for various star ratings and accident survivability.

lmkg 13y ago

> (2) the star systems suffer from the same problems as contemporary game and movie reviews: 99% of all evaluated products land between 3 and 5 stars, where 3 then becomes the new horrible, compressing the already low entropy of a star value further. That's not a bad thing. We wanted safer cars. We started measuring safety, and publishing the measurements. As a result, safety improved. Working As Intended.

gamblor956 13y ago

The threshold for each star is increased each year, based on the available safety features and safety performance of cars. A car that received 5 stars under the 2000 rating system would receive only 2 stars today because it lacks some of the now-required safety features. 3 is the new horrible simply because that is the lowest that a commercially viable car can go--Chevy and Subaru have released sub-3 cars in the past and suffered for it with poor sales. They and others learned that consumers wouldn't buy cars that were rated below-average safety-wise and have vastly increased R&D spending on safety features. That's a good thing for consumers.

ars 13y ago
> rated below-average It better not be a real average if everyone is always above average! You mean to say poorly rated, not below average, since it's not an average.

stcredzero 13y ago

Here is how star rating actually works: For a truly excellent product, the distribution should approximate a power law graph: XXXXXXXXXXXXXXXX XXXXXX XX X X If you see something like these: XXXXXXX X X X X X X X X XXXXX XXXXXXXXXXXXXXXX XXXXXX XXX X XXXXXXXX Then ask yourself if the sample size makes this plausible, in which case the star rating isn't worth much, because there isn't enough data. Ask yourself if it's possible that someone has been gaming the system for or against. The uppercase gamma-shaped graph is either someone gaming the system or it's a power law graph hidden by aliasing. (In which case, the thing is of godlike quality.) We should do away with star ratings, and instead just present an aggregate "rating." Genuine customer ratings are valuable, but are almost immediately gamed by cheaters. If I were Apple or Amazon, I would build stealth captchas and "honeypot" traps for ratings manipulation, and figure those into the rating. Be sure to separate detection from consequences by a random time interval of at least a few days. This makes the "development cost" for the cheaters much higher. The problem with star ratings is obviously what PG refers to as "schlepping". Is anyone tackling this already? EDIT: I would surmise that AirBNB is somewhat tackling this problem already.