4 ms·

curious if all the fake reviews will end up creating a market for certified reviewers or something like consumer reports. Sure we can't trust cnet or wired or p

curious if all the fake reviews will end up creating a market for certified reviewers or something like consumer reports. Sure we can't trust cnet or wired or pcmag but maybe if the incentive is high enough there will be a market for honest reviews including an incentive for auditing and verification? could start a reviewer guild and use crypto signatures to verify their guild membership is up to date and still valid. hoping the guild has the incentive to stay honest etc.

supertrope 8y ago
There was always a problem with misaligned incentives. If you're not paying for the review, they have no loyalty to the reader. e-commerce websites pay lip service to deleting fake reviews but higher product review scores result in more sales. Even if this doesn't result in intentionally ignoring fake reviews in pursuit of short term sales growth, note how Amazon.com like all other five star rating systems suffers from massive score inflation. Most have a 4.5/5.0 rating. 4.0 indicate some potential problems (or less sophisticated customers), and 3.5 means it's sub-par. A lot of online stores simply have no customer review section because they have rationally determined that for them decreased conversions due to bad reviews and moderation costs exceed increase in conversions due to good reviews.
- bostik 8y ago
  > five star rating systems suffers from massive score inflation Indeed. I liked how Goodreads had their original rating scale done: 3 was good, 4 was very good, and 5 was truly exceptional. You weren't supposed to give 5 to more than maybe 2% of the books you'd read. Then the got bought out (by Amazon) and the scale got soon enough devalued. I still think that one potential solution to the scale inflation would be to consider the grade distribution a person uses. If all they give out is ones or fives, their reviews should have near zero weight. If they give out a more balanced (on a ~Gaussian curve) reviews, then their rare extremes should be weighted much higher.
  - dswalter 8y ago
    Your idea is along the lines of 'rater reliability' in the field of educational measurement/psychometrics.
  - tomjen3 8y ago
    Honestly just make it thumbs up or thumps down. Five star ratings are quite pointless.
    - supertrope 8y ago
      I like Rotten Tomatoes's system. Individual critics give a thumbs up or down. A movie is "certified fresh" if at least 80% think it was worth watching. It doesn't mean the average rating was 80% or that that number means it was amazing or excellent, it just means the vast majority thought it was worth your time and money.
- alextheparrot 8y ago
  I recently used a site called “Tabelog” while traveling in Japan - the median review is roughly a 3.2, a pleasant surprise that left me with a strong insight that a restuarant with a 4 or above was quite a quality establishment.
rogerbinns 8y ago
> creating a market for certified reviewers or something like consumer reports. I'd argue that is exactly what The Wirecutter is. https://thewirecutter.com/ https://thewirecutter.com/ https://en.wikipedia.org/wiki/Wirecutter_(website) https://en.wikipedia.org/wiki/Wirecutter_(website) They solve the reviewer problem by having in house staff write the recommendations (reviews). Those staff then find and bring in experts. Virtually every page has a "Why you should trust us" section listing experience of those who contributed to the article. It does mean you get credible opinions, but also that you don't get the "wisdom of the masses" such as at Amazon.
- enobrev 8y ago
  They do sometimes reference the "wisdom of the masses" in their reviews as well, where they'll occasionally reference the overall ratings from various sources within the detailed portions of their reviews. I'm generally a fan of the Wirecutter while remaining somewhat skeptical of the motives behind their reviews. My skepticism hasn't changed since NYTimes took over, but I do still visit the site when looking for a specific product. Nonetheless, they've made a good deal of affiliate $$ from me, and I haven't been severely disappointed yet.
  - rogerbinns 8y ago
    Once they've done a page there is also often a vigourous comments section, and they do roll that feedback into future updates. My opinion of the site quality is much the same as yours. But at least they show their working and do update content regularly.
plankers 8y ago
The guild could issue ReviewCoin and use this to pay reviewers, thus incentivizing them to work in good faith lest the public lose faith in the project, tanking the value of the coin. ReviewCoin could be redeemed at participating online retailers for goods, thus allowing reviewers to buy more products to review. This is a silly idea but it makes me smile.

supertrope 8y ago

There was always a problem with misaligned incentives. If you're not paying for the review, they have no loyalty to the reader. e-commerce websites pay lip service to deleting fake reviews but higher product review scores result in more sales. Even if this doesn't result in intentionally ignoring fake reviews in pursuit of short term sales growth, note how Amazon.com like all other five star rating systems suffers from massive score inflation. Most have a 4.5/5.0 rating. 4.0 indicate some potential problems (or less sophisticated customers), and 3.5 means it's sub-par. A lot of online stores simply have no customer review section because they have rationally determined that for them decreased conversions due to bad reviews and moderation costs exceed increase in conversions due to good reviews.

bostik 8y ago
> five star rating systems suffers from massive score inflation Indeed. I liked how Goodreads had their original rating scale done: 3 was good, 4 was very good, and 5 was truly exceptional. You weren't supposed to give 5 to more than maybe 2% of the books you'd read. Then the got bought out (by Amazon) and the scale got soon enough devalued. I still think that one potential solution to the scale inflation would be to consider the grade distribution a person uses. If all they give out is ones or fives, their reviews should have near zero weight. If they give out a more balanced (on a ~Gaussian curve) reviews, then their rare extremes should be weighted much higher.
- dswalter 8y ago
  Your idea is along the lines of 'rater reliability' in the field of educational measurement/psychometrics.
- tomjen3 8y ago
  Honestly just make it thumbs up or thumps down. Five star ratings are quite pointless.
  - supertrope 8y ago
    I like Rotten Tomatoes's system. Individual critics give a thumbs up or down. A movie is "certified fresh" if at least 80% think it was worth watching. It doesn't mean the average rating was 80% or that that number means it was amazing or excellent, it just means the vast majority thought it was worth your time and money.
alextheparrot 8y ago
I recently used a site called “Tabelog” while traveling in Japan - the median review is roughly a 3.2, a pleasant surprise that left me with a strong insight that a restuarant with a 4 or above was quite a quality establishment.

rogerbinns 8y ago

> creating a market for certified reviewers or something like consumer reports. I'd argue that is exactly what The Wirecutter is. https://thewirecutter.com/ https://thewirecutter.com/ https://en.wikipedia.org/wiki/Wirecutter_(website) https://en.wikipedia.org/wiki/Wirecutter_(website) They solve the reviewer problem by having in house staff write the recommendations (reviews). Those staff then find and bring in experts. Virtually every page has a "Why you should trust us" section listing experience of those who contributed to the article. It does mean you get credible opinions, but also that you don't get the "wisdom of the masses" such as at Amazon.

enobrev 8y ago
They do sometimes reference the "wisdom of the masses" in their reviews as well, where they'll occasionally reference the overall ratings from various sources within the detailed portions of their reviews. I'm generally a fan of the Wirecutter while remaining somewhat skeptical of the motives behind their reviews. My skepticism hasn't changed since NYTimes took over, but I do still visit the site when looking for a specific product. Nonetheless, they've made a good deal of affiliate $$ from me, and I haven't been severely disappointed yet.
- rogerbinns 8y ago
  Once they've done a page there is also often a vigourous comments section, and they do roll that feedback into future updates. My opinion of the site quality is much the same as yours. But at least they show their working and do update content regularly.

plankers 8y ago

The guild could issue ReviewCoin and use this to pay reviewers, thus incentivizing them to work in good faith lest the public lose faith in the project, tanking the value of the coin. ReviewCoin could be redeemed at participating online retailers for goods, thus allowing reviewers to buy more products to review. This is a silly idea but it makes me smile.