5 ms·

I don't think I'll take this seriously until Amazon stops gaming their own review system. For instance, if you take a look at Comcast's rating on Amazon the ha

by TrevorJ 10y ago

I don't think I'll take this seriously until Amazon stops gaming their own review system. For instance, if you take a look at Comcast's rating on Amazon the have 4 stars, but if you click in and look at the breakdown 85% of the ratings are one star. The fact that this magically translates into a 4-star rating in Amazon's world is a huge issue.

Someone1234 10y ago
If others want to see what they're talking about, search for "Xfinity Internet" on Amazon. It has a four star rating, but 90% of the reviews are three stars or less. It doesn't make any mathematical sense.
throwaway13337 10y ago
I was in sort of disbelief so I had to look it up. https://www.amazon.com/Xfinity-Internet-Mbps-12-month-term/dp/B01B6ZHV7C/ https://www.amazon.com/Xfinity-Internet-Mbps-12-month-term/d... Yep. 4 stars with 85% one star ratings. Amazingly awful. To be fair, there are certain categories that amazon weights differently based on the average rating of those categories. Some types of things are just always rated poorly. Maybe that has something to do with it?
- mikequinlan 10y ago
  Quoting... "Amazon calculates a product’s star ratings using a machine learned model instead of a raw data average. The machine learned model takes into account factors including: the age of a review, helpfulness votes by customers and whether the reviews are from verified purchases."
  - timv 10y ago
    But what is the machine learning to do? It sounds like it could be optimising for sales. I actually mean that quite seriously - a machine learning system needs to have some goal to optimise for, sone way to tell whether the rating it produces is "good". In Amazon's case that is quite likely to be some measure of clicks or sales. [Edited to remove typo]
    - tajen 10y ago
      I only buy from Amazon when the product isn't available anywhere else: Between counterfeit products, external vendors that I don't know how to distinguish from Amazon itself, stuff "Available in 2 days" that gets delivered 2 months later, bought reviews, strange ratings and pictures which don't natch the description — I have a 70% chance of paying the return shipment.
  - r00fus 10y ago
    Sounds like secret sauce and not transparent. I really liked IMDB's rating scheme developed back in the late 90s - openly transparent but not a simple mean of rating values. Of course, back then they weren't part of Amazon yet.
  - deleted 10y ago
    [deleted]
  - heakins 10y ago
    Throwing a little bit more data into the conversation. Filtering to only the verified reviews shows 16 reviews: 8x 5 star reviews 1x 4 star reviews 1x 3 star reviews 4x 2 star reviews 2x 1 star reviews Mean star rating of verified reviews: 3.5625 That's a lot closer to the overall star rating. Disclosure: I work for Amazon Lab126 (not related to the retail website).
  - dragonwriter 10y ago
    > Amazon calculates a product’s star ratings using a machine learned model instead of a raw data average. A machine learned model optimizing what output measure? Ideally, a machine learned model for ratings should optimize for what I, as a potential consumer, would most likely rate the product were I to purchase it.
    - Spivak 10y ago
      That's noble but it's going to be difficult to overcome the temptation to train their machine to produce the rating that gets the most sales. Up to the point where it starts affecting their reputation or it concerns their branded products, Amazon doesn't care too much what they're selling, just in what volume.
      - daveFNbuck 10y ago
        The fact that they don't care what they sell is why you can mostly trust them to be trying for accurate ratings. If you can trust the ratings, you'll buy more of your stuff on Amazon. Boosting the ratings only changes what you buy (which we agree they don't care about) and leaves you with a sour experience.
- newesthn 10y ago
  There also appear to be satirical reviews. Consider the top one: 169 people found this helpful 5.0 out of 5 stars "Hitler highly recommends Comcast" Adolf Hitler was quoted saying this about Comcast: "Beeindruckend, das ist böse" Rough translation: "Wow, that's evil"
Cyph0n 10y ago
I heard that it's because Amazon gives higher weight to reviews backed by verified purchases. If you look closely, you'll notice that most reviews are from people who didn't purchase the product. I agree that it doesn't make sense in this case - Comcast is just evil - but the system seems fair in general.
- r00fus 10y ago
  Which is a bit disingenuous. So if I'm a Comcast customer my rating doesn't count unless I bought through Amazon. I honestly don't know anyone who's done that. Amazon used to be the place to validate if something wasn't crap (and maybe buy it). Now it looks like they're not interested in that anymore. Is there any product site that's not infested with secret-sauce ratings schemes and/or outright fake reviews?
  - Cyph0n 10y ago
    I think it's more Comcast's fault because they're the ones who got listed on Amazon. The way I see it is Comcast is using Amazon's algorithm to mislead potential customers who just look at the Amazon rating. It's not Amazon's fault at all. This is a glaring loophole that Amazon should address of course, but the system they have in place is fair imo and prevents trolls from affecting a product's ranking for no valid reason.
Shivetya 10y ago
Its Comcast, I doubt they could get a good review simply from the piling on that many people do just to be part of the in crowd. There are many products I do read from the 1 star up direction, but any company as polarizing as Comcast is not one to take most of them seriously. In all cases just use your head, you cannot do much to help people who want to believe what they read. The only reason the system bothers me is that when it comes to sorts
InclinedPlane 10y ago
Yup, the system is pretty broken, 4.5 stars is "pretty good", anything else is questionable. This is because the average is a mean, and the scale is 1 to 5 stars. The average of 1 and 5 is 3, so 3, not 2.5, is a "mixed reactions" score. With two 1s and one 5 the average is 2. So if you ever see something with a two star average, run. With a single 5 star review, it takes 16 one star reviews to bring the average down to over star, to reflect the majority opinion.
- imtringued 10y ago
  I still don't know why we use scales to rate anything. It gives us the illusion of being able to compare two scores with each other. By dumping everything into a scale the only thing we know is: on average it might be good or bad. It doesn't tell us whether it's actually good or bad. It makes far more sense to just ask a few yes or no questions with some context. Did it deliver on time? Did the product arrive without damage? Is the product fit for purpose? etc Different people value different things. With a linear score a good "attribute" that you don't care about might cancel out a bad "attribute" that you do care about since both end up in the same pot.
guelo 10y ago
That actually looks like fraud of the kind that the FTC should look into. Just saying "the algorithm made me lie" isn't a good enough excuse.
Obi_Juan_Kenobi 10y ago
Could it be that there was 'review brigading' against Comcast? I could see e.g. a popular Reddit thread directing people to leave poor reviews wherever they could, which might lead to some spikes of one star reviews that are then given very little weight for the overall rating. Just trying to apply Hanlon's Razor.
WhitneyLand 10y ago
Maybe someone should create a browser extension to calculate "TrueReview" scores. So when you're on a Amazon product page click the TrueReview button to show alternate models such as average, average last 30 days, average after removing reviewers who disclose receiving a free product, etc

by TrevorJ 10y ago

Someone1234 10y ago

If others want to see what they're talking about, search for "Xfinity Internet" on Amazon. It has a four star rating, but 90% of the reviews are three stars or less. It doesn't make any mathematical sense.

throwaway13337 10y ago

I was in sort of disbelief so I had to look it up. https://www.amazon.com/Xfinity-Internet-Mbps-12-month-term/dp/B01B6ZHV7C/ https://www.amazon.com/Xfinity-Internet-Mbps-12-month-term/d... Yep. 4 stars with 85% one star ratings. Amazingly awful. To be fair, there are certain categories that amazon weights differently based on the average rating of those categories. Some types of things are just always rated poorly. Maybe that has something to do with it?

mikequinlan 10y ago
Quoting... "Amazon calculates a product’s star ratings using a machine learned model instead of a raw data average. The machine learned model takes into account factors including: the age of a review, helpfulness votes by customers and whether the reviews are from verified purchases."
- timv 10y ago
  But what is the machine learning to do? It sounds like it could be optimising for sales. I actually mean that quite seriously - a machine learning system needs to have some goal to optimise for, sone way to tell whether the rating it produces is "good". In Amazon's case that is quite likely to be some measure of clicks or sales. [Edited to remove typo]
  - tajen 10y ago
    I only buy from Amazon when the product isn't available anywhere else: Between counterfeit products, external vendors that I don't know how to distinguish from Amazon itself, stuff "Available in 2 days" that gets delivered 2 months later, bought reviews, strange ratings and pictures which don't natch the description — I have a 70% chance of paying the return shipment.
- r00fus 10y ago
  Sounds like secret sauce and not transparent. I really liked IMDB's rating scheme developed back in the late 90s - openly transparent but not a simple mean of rating values. Of course, back then they weren't part of Amazon yet.
- deleted 10y ago
  [deleted]
- heakins 10y ago
  Throwing a little bit more data into the conversation. Filtering to only the verified reviews shows 16 reviews: 8x 5 star reviews 1x 4 star reviews 1x 3 star reviews 4x 2 star reviews 2x 1 star reviews Mean star rating of verified reviews: 3.5625 That's a lot closer to the overall star rating. Disclosure: I work for Amazon Lab126 (not related to the retail website).
- dragonwriter 10y ago
  > Amazon calculates a product’s star ratings using a machine learned model instead of a raw data average. A machine learned model optimizing what output measure? Ideally, a machine learned model for ratings should optimize for what I, as a potential consumer, would most likely rate the product were I to purchase it.
  - Spivak 10y ago
    That's noble but it's going to be difficult to overcome the temptation to train their machine to produce the rating that gets the most sales. Up to the point where it starts affecting their reputation or it concerns their branded products, Amazon doesn't care too much what they're selling, just in what volume.
    - daveFNbuck 10y ago
      The fact that they don't care what they sell is why you can mostly trust them to be trying for accurate ratings. If you can trust the ratings, you'll buy more of your stuff on Amazon. Boosting the ratings only changes what you buy (which we agree they don't care about) and leaves you with a sour experience.
newesthn 10y ago
There also appear to be satirical reviews. Consider the top one: 169 people found this helpful 5.0 out of 5 stars "Hitler highly recommends Comcast" Adolf Hitler was quoted saying this about Comcast: "Beeindruckend, das ist böse" Rough translation: "Wow, that's evil"

Cyph0n 10y ago

I heard that it's because Amazon gives higher weight to reviews backed by verified purchases. If you look closely, you'll notice that most reviews are from people who didn't purchase the product. I agree that it doesn't make sense in this case - Comcast is just evil - but the system seems fair in general.

r00fus 10y ago
Which is a bit disingenuous. So if I'm a Comcast customer my rating doesn't count unless I bought through Amazon. I honestly don't know anyone who's done that. Amazon used to be the place to validate if something wasn't crap (and maybe buy it). Now it looks like they're not interested in that anymore. Is there any product site that's not infested with secret-sauce ratings schemes and/or outright fake reviews?
- Cyph0n 10y ago
  I think it's more Comcast's fault because they're the ones who got listed on Amazon. The way I see it is Comcast is using Amazon's algorithm to mislead potential customers who just look at the Amazon rating. It's not Amazon's fault at all. This is a glaring loophole that Amazon should address of course, but the system they have in place is fair imo and prevents trolls from affecting a product's ranking for no valid reason.

Shivetya 10y ago

Its Comcast, I doubt they could get a good review simply from the piling on that many people do just to be part of the in crowd. There are many products I do read from the 1 star up direction, but any company as polarizing as Comcast is not one to take most of them seriously. In all cases just use your head, you cannot do much to help people who want to believe what they read. The only reason the system bothers me is that when it comes to sorts

InclinedPlane 10y ago

Yup, the system is pretty broken, 4.5 stars is "pretty good", anything else is questionable. This is because the average is a mean, and the scale is 1 to 5 stars. The average of 1 and 5 is 3, so 3, not 2.5, is a "mixed reactions" score. With two 1s and one 5 the average is 2. So if you ever see something with a two star average, run. With a single 5 star review, it takes 16 one star reviews to bring the average down to over star, to reflect the majority opinion.

imtringued 10y ago
I still don't know why we use scales to rate anything. It gives us the illusion of being able to compare two scores with each other. By dumping everything into a scale the only thing we know is: on average it might be good or bad. It doesn't tell us whether it's actually good or bad. It makes far more sense to just ask a few yes or no questions with some context. Did it deliver on time? Did the product arrive without damage? Is the product fit for purpose? etc Different people value different things. With a linear score a good "attribute" that you don't care about might cancel out a bad "attribute" that you do care about since both end up in the same pot.

guelo 10y ago

That actually looks like fraud of the kind that the FTC should look into. Just saying "the algorithm made me lie" isn't a good enough excuse.

Obi_Juan_Kenobi 10y ago

Could it be that there was 'review brigading' against Comcast? I could see e.g. a popular Reddit thread directing people to leave poor reviews wherever they could, which might lead to some spikes of one star reviews that are then given very little weight for the overall rating. Just trying to apply Hanlon's Razor.

WhitneyLand 10y ago

Maybe someone should create a browser extension to calculate "TrueReview" scores. So when you're on a Amazon product page click the TrueReview button to show alternate models such as average, average last 30 days, average after removing reviewers who disclose receiving a free product, etc