11 ms·

Amazon's overcomplicated new product star ratings are no bright idea

59 points by shscs911 3y ago

jasonjamerson 3y ago
I was so confused by this yesterday. I want to see the number of ratings compared to the average rating. That's how I know if a large number of people have tested this product and reviewed it properly. This system seems designed to hide that. You have to open every product to see that info.
- karmelapple 3y ago
  It reminded me how important information design is. Amazon has an average number of stars there, but it's no longer visually shown - instead, just a number. I have relied on the stars being "filled up" for so long that it now felt like every product was basically the same: a single, lone star. It was so harder to get an at-a-glance feeling of which products were good and which not-so-good. And unforutnately, that's probably Amazon's goal.
juujian 3y ago
At this point, aren't sellers gaming the system enough that the overall rating has become effectively meaningless? Individual reviews, maybe useful. But its not like the olden days where you could scour amazon for good products, ideally you would have a good idea what you want before even visiting the site.
- fullshark 3y ago
  I still find them useful. I use rating volume + ensuring rating is above 4 + checking whether the brand has a legit amazon store/landing page mostly to try and avoid duds. Individual reviews are useful for specific questions. As much as it sucks these days it's still better than going to your local durable goods retailer, having only one option to choose from with no ratings at all.
  - CryptoBanker 3y ago
    For me rating volume can be a red flag. Any product that has over 1,000 reviews and isn’t something has been on the market for years, I automatically deduct a star or two in my mind. People just don’t leave legitimate reviews at that kind of rate
    - rjh29 3y ago
      It usually means their product is cheap, not terrible enough to get <4 stars, and they paid for a high search position.
  - schnable 3y ago
    I wish this were true but my purchasing experiences tell me otherwise. There's a lot of awful products with a high volume of good reviews. To the point where for common household products, the decision of the Walmart or Target purchaser is usually better than me picking between a dozen generic Chinese products with 4.5 stars. And it's easier!
    - CryptoBanker 3y ago
      One hundred percent. Not to mention the possibility of receiving counterfeit goods even when purchasing name brand on Amazon.
  - specialp 3y ago
    There are plenty of items that have been replaced with a different one but maintain the previous product's high review volume. Also there are those with a huge number of fake reviews. I look at the 1-2 star reviews. If they are dumb ones like box was damaged or shipping slow, that is good. If there are 4000 reviews saying "Great product" and 30 saying it went up in smoke with pictures, I am not buying it.
- deleted 3y ago
  [deleted]
- alistairSH 3y ago
  That's my take. I do my research elsewhere (which has it's own problems), and then check Amazon for specific products (and these days, they often aren't any less expensive for name-brand items).
soneca 3y ago
Interestingly, I am seeing (on Amazon.com.br) a slightly different test of ”percentage of 4-star ratings or above” I don’t have an opinion if this is good or bad for customers yet.
malfist 3y ago
Amazon is nothing but a giant flea market these days. Everything is 5 stars through fake reviews and everything lies about it's capabilities. Just today I was looking for a hairdryer and the top one advertised an RPM of 150,000. That'd put its 2 inch turbine spinning at almost 900mph. How much you want to bet the sound barrier isn't broken every time you turn that hairdryer on?
- Pxtl 3y ago
  AliExpress with faster shipping.
- coldpie 3y ago
  Honestly I've mostly stopped shopping online entirely. B&M stores have limited space, which means they have to choose what to stock, so they do the work of selecting decent products for me. When I do shop online, I don't use any of the flea markets (Amazon, Newegg, Walmart, etc).
  - specialp 3y ago
    I have shopped online almost exclusively for over 20 years and have started doing the same thing. If I buy a fan at Home Depot, I can be reasonably assured it won't go on fire. They have product buyers that assess the quality and prices of products. Amazon purposely does not have this as it would open them to liability. So "Independent reseller" GOODAIR fan is not at all assessed, and they only provide a platform for users to judge the quality of items and post reviews.
    - malfist 3y ago
      But the GOODAIR fan says it's UL certified and can produce 12047492037 CFM of air! It has 3,000 5 star reviews, surely it's high quality
    - delfinom 3y ago
      Legally speaking, B&M retailers specifically buy things such as UL rated products just because they would incur legal liability for not doing due diligence before stocking and selling goods. Amazon avoids all of that liability by claiming they are essentially, a flea market. Hence why Amazon begrudgingly reacted to the FTC demanding the takedown of completely fake smoke alarm listings.
Pxtl 3y ago
I mean five star ratings have always been flawed. Any system where getting 4/5 is bad means the raters are confused, and then it's GIGO. This weird % five-star thing is a workaround for that GIGO because they're treating 4 stars as the same as 1 star. Personally I'd switch to a "good/flawed/terrible" rating system and just give viewers a pie-chart (or a stacked-bar) of ratings.
tgv 3y ago
It doesn't look overcomplicated. When the average goes up, the number of 5 stars ratings (usually) goes up too, but the distinction between a 4.4 and a 4.6 is harder to spot than that between 40% and 60% 5 star rating (in this example, all ratings are either 4 or 5). So you could argue it offers a better discrimination. But, it also really looks like they want to hide information that would stop you from buying a product. I know the "don't attribute to malice" maxim, but this is Amazon: they know what they're doing. You don't get to be this size by just fumbling around. They are obsessed with the look of their pages, keeping changes minimal, so they've thought about this long and hard. This change does look like a dark pattern.
- Supermancho 3y ago
  > From a user experience standpoint, the new rating display system isn’t as glanceable as the five yellow stars. Additionally, the mention of “5-star” and the use of just one yellow star in the new arrangement could mislead new users, at least briefly. It's overcomplicated enough to have 2 different star indicators. It's an idea that's a few decades old. The late 90s had Bizrate.com managing millions of star images (via java apps) for retail ratings.
- hgsgm 3y ago
  Amazon has always been the leading company in the word for rapidly shipping and testing changes. Fumbling around incrementally is their strategy. They made A/B Testing and Multi-Armed Bandit household terms.
- AJ007 3y ago
  Well there is always increased returns and a drop in overall purchases because you believe the rating is bullshit. Maybe they were already so bad on both, that this lifts overall sales long term.
fidotron 3y ago
I should invent a statistical distribution for scores in 5 star rating systems. There should be peaks at 5 and 1 with a spread in between. Deviations from this distribution would be a fraud indicator, and the 5:1 ratio the rating. With moves like this Amazon are trending towards the same discoverability quality in their marketplace that Google has achieved with the web as a whole, which may hint at another unknown fundamental law of internet economics.
- daft_pink 3y ago
  i agree. i researched reviews for products and services i like and product and services that look fake. i think the top restaurants in my area have a staircase like distribution where 1 star reviews have the lowest aggregate quantity. products and services that suck have more one star reviews than 2, 3 and 4 star reviews and are consistently more polarized with one and five star reviews.
mvuijlst 3y ago
I think a sparkline bar chart would be more useful.
- hgsgm 3y ago
  That's the old way but it's only on the single item page.
taeric 3y ago
User contributed ratings have, it seems, mostly outlived their usefulness. It isn't that things can't be reviewed, but ratings are like anything else in life, in that they have deaths. You want me to trust ratings more, let me order them by date. Oddly, Steam does a good job with this. Two ratings, recent and all time. Google, amusingly, only shows the all time on the search page, but the recent reviews are very relevant. Does this have to apply to all things? Maybe not, but manufacturing changes. needs of the time change. General expectations change. A 4 star rating from a decade is of much less reliability than one today. Even if it is critically a good rating.
- stronglikedan 3y ago
  My system is to just focus on the 1 & 2 star ratings, and look for trends, specifically. It takes a bit of work, since most of those reviews are just nonsense, but eventually true flaws in the product will be exposed through repetition. Due to the amount of work, I typically only reserve this method for big ticket items that would be difficult to return, such as appliances.
  - taeric 3y ago
    Oddly, appliances are the area I would trust user reviews the least. It will be dominated by luck of experience on who will post. And with how variable manufacturing is for most, the exact same model sold on the other side of the nation is as likely to be a different beast as not. Not even considering last year's version of it. Still, I don't discard the idea here. If there are features of it that every low star mentions, I think I agree that I would trust the feature is bad. I can't think of often that has happened. Will try to pay more attention for it.
- deleted 3y ago
  [deleted]
- rjh29 3y ago
  Steam does that well, but its recommended/not recommended system is a dark pattern IMO. People tend to choose recommended even for games they don't like on the basis that someone else might like it, or it might improve in the future. Virtually every popular game has 'Overwhelmingly Positive' or similar. To get useful data you need to read the not recommended reviews in detail. A more honest system would be 'Bad', 'Average' and 'Fantastic' - then I bet you'd get a ton more 'average' reviews.
marban 3y ago
I find the indicator for how often an item has been purchased in the last month (on the European site) far more helpful. On a side note, galaxus.ch even shows you the return rate.
daft_pink 3y ago
Most reliable metric: ignore 5 star reviews and only compare the number of 4 star reviews to the number of 1 star reviews. this works across all review sites. most sellers only fake 5 star reviews. however, if you review products and services that you know are quite excellent, you will that they often have way more 4 star reviews than one star reviews and more 4 than 3, more three than 2, and more 2 than one. this is because they are consistently excellent. but products with polar review metrics like all 5 star reviews and all 1 star reviews generally i find have the most fraudulent reviews and are most likely to have a bad experience. I think you should weigh 1 star reviews much higher than 5 star reviews. the new amazon metric or the advertisements talking about their aggregate quantity of five star reviews are wrong and misguided.
- rjh29 3y ago
  1 star reviews can also be paid for. I tend to look at 1/2/3-star reviews that seem passionate (was written like a real person with actual emotions - bonus points for photos) and see if the things that bothered them would also bother me. I like your idea though, and will use it as an extra data point.
- manuelabeledo 3y ago
  I tend to ignore the 1 star reviews because, in my experience, most are either from disgruntled people who wouldn't even try to open the manual first, or absolute binary thinkers. When it comes to more subjective categories, like books, 1 star reviews are even worse.
  - foobarian 3y ago
    However they are likely not fake, or less likely than 5-stars.
    - TylerE 3y ago
      Review bombing is absolutely "a thing".
  - AdmiralAsshat 3y ago
tguvot 3y ago
Did Q&A also disappeared for everyone at same time ? Its not shown to me for past week or so
trickpattyyo 3y ago
Script to add back the review counts: https://gist.github.com/trickpattyFH20/8d02b18aaf63b7d405456c13602b3f16 https://gist.github.com/trickpattyFH20/8d02b18aaf63b7d405456...
manuelabeledo 3y ago
It baffles me that Amazon doesn't use their shopper's data more effectively. They could have gone with a weighted average, accounting for number of reviews, consistency of scores, etc. of the shopper in question. Instead, they want to highlight the most deceptive side of the scores, i.e. 5-star ones. Kinda feels like they don't want to end fake reviews, but prop them.
hulamedulla 3y ago
[dead]
deleted 3y ago
[deleted]
noja 3y ago
Normalise each person's reviews to make 3/5 their average. Take notice when people do not leave a review who normally do (hello AirBNB! this is the metric you want!), it often indicates something has gone quite wrong.
ericjack427 3y ago
[dead]

59 points by shscs911 3y ago

I was so confused by this yesterday. I want to see the number of ratings compared to the average rating. That's how I know if a large number of people have tested this product and reviewed it properly. This system seems designed to hide that. You have to open every product to see that info.

karmelapple 3y ago
It reminded me how important information design is. Amazon has an average number of stars there, but it's no longer visually shown - instead, just a number. I have relied on the stars being "filled up" for so long that it now felt like every product was basically the same: a single, lone star. It was so harder to get an at-a-glance feeling of which products were good and which not-so-good. And unforutnately, that's probably Amazon's goal.

juujian 3y ago

At this point, aren't sellers gaming the system enough that the overall rating has become effectively meaningless? Individual reviews, maybe useful. But its not like the olden days where you could scour amazon for good products, ideally you would have a good idea what you want before even visiting the site.

fullshark 3y ago
I still find them useful. I use rating volume + ensuring rating is above 4 + checking whether the brand has a legit amazon store/landing page mostly to try and avoid duds. Individual reviews are useful for specific questions. As much as it sucks these days it's still better than going to your local durable goods retailer, having only one option to choose from with no ratings at all.
- CryptoBanker 3y ago
  For me rating volume can be a red flag. Any product that has over 1,000 reviews and isn’t something has been on the market for years, I automatically deduct a star or two in my mind. People just don’t leave legitimate reviews at that kind of rate
  - rjh29 3y ago
    It usually means their product is cheap, not terrible enough to get <4 stars, and they paid for a high search position.
- schnable 3y ago
  I wish this were true but my purchasing experiences tell me otherwise. There's a lot of awful products with a high volume of good reviews. To the point where for common household products, the decision of the Walmart or Target purchaser is usually better than me picking between a dozen generic Chinese products with 4.5 stars. And it's easier!
  - CryptoBanker 3y ago
    One hundred percent. Not to mention the possibility of receiving counterfeit goods even when purchasing name brand on Amazon.
- specialp 3y ago
  There are plenty of items that have been replaced with a different one but maintain the previous product's high review volume. Also there are those with a huge number of fake reviews. I look at the 1-2 star reviews. If they are dumb ones like box was damaged or shipping slow, that is good. If there are 4000 reviews saying "Great product" and 30 saying it went up in smoke with pictures, I am not buying it.
deleted 3y ago
[deleted]
alistairSH 3y ago
That's my take. I do my research elsewhere (which has it's own problems), and then check Amazon for specific products (and these days, they often aren't any less expensive for name-brand items).

soneca 3y ago

Interestingly, I am seeing (on Amazon.com.br) a slightly different test of ”percentage of 4-star ratings or above” I don’t have an opinion if this is good or bad for customers yet.

malfist 3y ago

Amazon is nothing but a giant flea market these days. Everything is 5 stars through fake reviews and everything lies about it's capabilities. Just today I was looking for a hairdryer and the top one advertised an RPM of 150,000. That'd put its 2 inch turbine spinning at almost 900mph. How much you want to bet the sound barrier isn't broken every time you turn that hairdryer on?

Pxtl 3y ago
AliExpress with faster shipping.
coldpie 3y ago
Honestly I've mostly stopped shopping online entirely. B&M stores have limited space, which means they have to choose what to stock, so they do the work of selecting decent products for me. When I do shop online, I don't use any of the flea markets (Amazon, Newegg, Walmart, etc).
- specialp 3y ago
  I have shopped online almost exclusively for over 20 years and have started doing the same thing. If I buy a fan at Home Depot, I can be reasonably assured it won't go on fire. They have product buyers that assess the quality and prices of products. Amazon purposely does not have this as it would open them to liability. So "Independent reseller" GOODAIR fan is not at all assessed, and they only provide a platform for users to judge the quality of items and post reviews.
  - malfist 3y ago
    But the GOODAIR fan says it's UL certified and can produce 12047492037 CFM of air! It has 3,000 5 star reviews, surely it's high quality
  - delfinom 3y ago
    Legally speaking, B&M retailers specifically buy things such as UL rated products just because they would incur legal liability for not doing due diligence before stocking and selling goods. Amazon avoids all of that liability by claiming they are essentially, a flea market. Hence why Amazon begrudgingly reacted to the FTC demanding the takedown of completely fake smoke alarm listings.

Pxtl 3y ago

I mean five star ratings have always been flawed. Any system where getting 4/5 is bad means the raters are confused, and then it's GIGO. This weird % five-star thing is a workaround for that GIGO because they're treating 4 stars as the same as 1 star. Personally I'd switch to a "good/flawed/terrible" rating system and just give viewers a pie-chart (or a stacked-bar) of ratings.

tgv 3y ago

It doesn't look overcomplicated. When the average goes up, the number of 5 stars ratings (usually) goes up too, but the distinction between a 4.4 and a 4.6 is harder to spot than that between 40% and 60% 5 star rating (in this example, all ratings are either 4 or 5). So you could argue it offers a better discrimination. But, it also really looks like they want to hide information that would stop you from buying a product. I know the "don't attribute to malice" maxim, but this is Amazon: they know what they're doing. You don't get to be this size by just fumbling around. They are obsessed with the look of their pages, keeping changes minimal, so they've thought about this long and hard. This change does look like a dark pattern.

Supermancho 3y ago
> From a user experience standpoint, the new rating display system isn’t as glanceable as the five yellow stars. Additionally, the mention of “5-star” and the use of just one yellow star in the new arrangement could mislead new users, at least briefly. It's overcomplicated enough to have 2 different star indicators. It's an idea that's a few decades old. The late 90s had Bizrate.com managing millions of star images (via java apps) for retail ratings.
hgsgm 3y ago
Amazon has always been the leading company in the word for rapidly shipping and testing changes. Fumbling around incrementally is their strategy. They made A/B Testing and Multi-Armed Bandit household terms.
AJ007 3y ago
Well there is always increased returns and a drop in overall purchases because you believe the rating is bullshit. Maybe they were already so bad on both, that this lifts overall sales long term.

fidotron 3y ago

I should invent a statistical distribution for scores in 5 star rating systems. There should be peaks at 5 and 1 with a spread in between. Deviations from this distribution would be a fraud indicator, and the 5:1 ratio the rating. With moves like this Amazon are trending towards the same discoverability quality in their marketplace that Google has achieved with the web as a whole, which may hint at another unknown fundamental law of internet economics.

daft_pink 3y ago
i agree. i researched reviews for products and services i like and product and services that look fake. i think the top restaurants in my area have a staircase like distribution where 1 star reviews have the lowest aggregate quantity. products and services that suck have more one star reviews than 2, 3 and 4 star reviews and are consistently more polarized with one and five star reviews.

mvuijlst 3y ago

I think a sparkline bar chart would be more useful.

hgsgm 3y ago
That's the old way but it's only on the single item page.

taeric 3y ago

User contributed ratings have, it seems, mostly outlived their usefulness. It isn't that things can't be reviewed, but ratings are like anything else in life, in that they have deaths. You want me to trust ratings more, let me order them by date. Oddly, Steam does a good job with this. Two ratings, recent and all time. Google, amusingly, only shows the all time on the search page, but the recent reviews are very relevant. Does this have to apply to all things? Maybe not, but manufacturing changes. needs of the time change. General expectations change. A 4 star rating from a decade is of much less reliability than one today. Even if it is critically a good rating.

stronglikedan 3y ago
My system is to just focus on the 1 & 2 star ratings, and look for trends, specifically. It takes a bit of work, since most of those reviews are just nonsense, but eventually true flaws in the product will be exposed through repetition. Due to the amount of work, I typically only reserve this method for big ticket items that would be difficult to return, such as appliances.
- taeric 3y ago
  Oddly, appliances are the area I would trust user reviews the least. It will be dominated by luck of experience on who will post. And with how variable manufacturing is for most, the exact same model sold on the other side of the nation is as likely to be a different beast as not. Not even considering last year's version of it. Still, I don't discard the idea here. If there are features of it that every low star mentions, I think I agree that I would trust the feature is bad. I can't think of often that has happened. Will try to pay more attention for it.
deleted 3y ago
[deleted]
rjh29 3y ago
Steam does that well, but its recommended/not recommended system is a dark pattern IMO. People tend to choose recommended even for games they don't like on the basis that someone else might like it, or it might improve in the future. Virtually every popular game has 'Overwhelmingly Positive' or similar. To get useful data you need to read the not recommended reviews in detail. A more honest system would be 'Bad', 'Average' and 'Fantastic' - then I bet you'd get a ton more 'average' reviews.

marban 3y ago

I find the indicator for how often an item has been purchased in the last month (on the European site) far more helpful. On a side note, galaxus.ch even shows you the return rate.

daft_pink 3y ago

Most reliable metric: ignore 5 star reviews and only compare the number of 4 star reviews to the number of 1 star reviews. this works across all review sites. most sellers only fake 5 star reviews. however, if you review products and services that you know are quite excellent, you will that they often have way more 4 star reviews than one star reviews and more 4 than 3, more three than 2, and more 2 than one. this is because they are consistently excellent. but products with polar review metrics like all 5 star reviews and all 1 star reviews generally i find have the most fraudulent reviews and are most likely to have a bad experience. I think you should weigh 1 star reviews much higher than 5 star reviews. the new amazon metric or the advertisements talking about their aggregate quantity of five star reviews are wrong and misguided.

rjh29 3y ago
1 star reviews can also be paid for. I tend to look at 1/2/3-star reviews that seem passionate (was written like a real person with actual emotions - bonus points for photos) and see if the things that bothered them would also bother me. I like your idea though, and will use it as an extra data point.
manuelabeledo 3y ago
I tend to ignore the 1 star reviews because, in my experience, most are either from disgruntled people who wouldn't even try to open the manual first, or absolute binary thinkers. When it comes to more subjective categories, like books, 1 star reviews are even worse.
- foobarian 3y ago
  However they are likely not fake, or less likely than 5-stars.
  - TylerE 3y ago
    Review bombing is absolutely "a thing".
- AdmiralAsshat 3y ago

tguvot 3y ago

Did Q&A also disappeared for everyone at same time ? Its not shown to me for past week or so

trickpattyyo 3y ago

Script to add back the review counts: https://gist.github.com/trickpattyFH20/8d02b18aaf63b7d405456c13602b3f16 https://gist.github.com/trickpattyFH20/8d02b18aaf63b7d405456...

manuelabeledo 3y ago

It baffles me that Amazon doesn't use their shopper's data more effectively. They could have gone with a weighted average, accounting for number of reviews, consistency of scores, etc. of the shopper in question. Instead, they want to highlight the most deceptive side of the scores, i.e. 5-star ones. Kinda feels like they don't want to end fake reviews, but prop them.

hulamedulla 3y ago

[dead]

deleted 3y ago

[deleted]

noja 3y ago

Normalise each person's reviews to make 3/5 their average. Take notice when people do not leave a review who normally do (hello AirBNB! this is the metric you want!), it often indicates something has gone quite wrong.

ericjack427 3y ago

[dead]