5 ms·

Today I was thinking about this kind of thing, for example Epic not allowing reviews at all, Valve "off topic" review hiding, metacritic changes to their system

Today I was thinking about this kind of thing, for example Epic not allowing reviews at all, Valve "off topic" review hiding, metacritic changes to their system, or Netflix removing their review system... I concluded that review systems often are on the losing side when their income comes from what they are reviewing, reviews you can trust are those that are unrelated. For example a famous bastion of review is the Michelin Guide, although controversies happened to it, and often restaurant owners get mad at it for not "supporting the restaurant industry" when the guide doesn't do what they want, I realized the probable reason the guide works, is because it wants to sell car tyres, not food. Michelin Guide purpose is make people drive to places and use their tyres (there was even a controversy caused by confused restauranteurs that thought it was "unfair" restaurant you can't drive to, aren't in the guide), thus the only bad incentive they might have is to choose restaurants that might cause bigger tyre usage than normal. But Netflix for example can't piss off the movie companies or they risk losing their catalog. Metacritic depends on ads, can't let users piss off the advertisers, and so on.

bryanlarsen 6y ago
Consumer's Reports is the canonical example of a review site where the incentives line up correctly. The only way they make money is by selling memberships, so the consumer and the customer are the same people.
- Goronmon 6y ago
  Manufacturers either on purpose, or by accident, have made Consumer Reports reviews not super useful, at least from my experience. The ability to match a product reviewed by Consumer Reports to something I can actual purchase has become pretty difficult over the years. This would be less a problem if brands followed some kind of trend in overall reliability, but it seems like individual models vary widely in quality even within a specific brand. So when it comes down to trying to find the specific model of something being reviewed, it seems like that model was discontinued, replaced, or not available at the retailers I use.
  - reaperducer 6y ago
    Different retailers get different model numbers, especially for white goods and other large purchases. The big retailers ask for this so that you can't use their fake "price match guarantees" because the model number is different between stores. For example, the TV I bought at Fry's was the same as a model at Wal-Mart, but with one fewer HDMI input. Another TV I bought (again at Fry's) was the same as a model from another store, but with the built-in DVR disabled. That leaves the burden on you to figure out which model was in Consumer Reports by matching up the features. It's annoying, but possible.
  - curryst 6y ago
    > This would be less a problem if brands followed some kind of trend in overall reliability, but it seems like individual models vary widely in quality even within a specific brand. This is, frustratingly, incredibly true. Even otherwise good products will tend to have some single component with a significantly lower lifespan than the rest of the device. This is compounded by the fact that everything is now manufactured to such exact specifications that it becomes effectively impossible to replace that piece with a higher quality one they should have used in the first place. Then you have the insane number of SKUs a single base device can generate, and it becomes difficult to tell whether the thing you're holding in your hand has the shitty version of that component, or the better one. Sometimes it's even the same SKU and there's a range of bad serial numbers (typically because the manufacturer got a bad batch of something from one of their suppliers). My guess is that the sheer number of parts in many modern products has drastically increased the chance that a device fails. My office has a water cooler with a purifier, and a spout for hot water. It broke and a guy came in to fix it. There are 2, 2 circuit boards in that thing! For a water cooler. How many parts are there that could fail and brick it? How many would a more primitive water cooler have? I just can't believe the simplest water cooler we could invent has 2 separate circuit boards in it
- devrand 6y ago
  It's a bit of a different model though since Consumer Reports are often producing the reviews themselves (which may or may not include some survey data from users) and have first-hand experience to editorialize. With things like Yelp [1] and particularly with Glassdoor this is a difficult model to emulate. You generally expect users themselves to contribute quality reviews which you then publish directly (plus maybe do a bit of curation to gather sentiment). It's difficult to both ask users to pay you for the service and also contribute the very content you charge them for. I think the only way to make a user review site work are: (a) Be a non-profit and have a donation based model. Would people actually donate though? I'm not sure... but at least you don't seem like a greedy middleman. (b) Charge a membership fee, but grant free durations of membership for quality reviews. The downside here is that you're likely severely limiting the already small pool of people willing to leave a review. [1]: The nearest comparison to CR<->Yelp would be Zagat (well, pre-Google Zagat) and Michelin. However, they both only cover a narrow slice.
  - nerdponx 6y ago
    What about Angie's List?
    - function_seven 6y ago
      They're just a clone of Home Advisor now. IAC bought them a few years back. They no longer charge their users.
  - bryanlarsen 6y ago
    Excellent point. A glassdoor competitor cannot use the Consumer Reports model since you can't review the working conditions of a company without being inside it. It shouldn't take much money to set up and run a review site so monetization should be a soluble problem. On a 6 figure budget you won't be as slick as competitors but if you're competing on trustworthiness... The biggest problem is going to be combating gamification, that'll be very hard on a 6 figure budget. And even your non-profit suggestion is a problem because what if a company donates $1M with strings attached? I really think the only solution is that all money has to come from consumers. Job hunting is important enough that people would be willing to shell out for it, but only once you have a sufficient pool of reviews. So a real chicken and egg problem here.
jdofaz 6y ago
For whatever reason the disc rental side of Netflix still has user reviews and the 5 star system
InitialLastName 6y ago
> But Netflix for example can't piss off the movie companies or they risk losing their catalog More than that, Netflix needs your eyes on their service, as opposed to the (dozen?) other competing, equally shoddy services. What you watch makes no difference to their bottom line... the only thing that matters is that you don't see a bad review on Netflix and switch to another streaming service (because if you do that too often, you might decide to remove Netflix from your list of $10/month services).
city41 6y ago
Reviews for entertainment are of very little value, and always have been. The only time this isn't true is if you get to know the reviewer and understand that your tastes and theirs line up. I suppose if something is universally panned that could be useful information, but that isn't very common.

bryanlarsen 6y ago

Consumer's Reports is the canonical example of a review site where the incentives line up correctly. The only way they make money is by selling memberships, so the consumer and the customer are the same people.

Goronmon 6y ago
Manufacturers either on purpose, or by accident, have made Consumer Reports reviews not super useful, at least from my experience. The ability to match a product reviewed by Consumer Reports to something I can actual purchase has become pretty difficult over the years. This would be less a problem if brands followed some kind of trend in overall reliability, but it seems like individual models vary widely in quality even within a specific brand. So when it comes down to trying to find the specific model of something being reviewed, it seems like that model was discontinued, replaced, or not available at the retailers I use.
- reaperducer 6y ago
  Different retailers get different model numbers, especially for white goods and other large purchases. The big retailers ask for this so that you can't use their fake "price match guarantees" because the model number is different between stores. For example, the TV I bought at Fry's was the same as a model at Wal-Mart, but with one fewer HDMI input. Another TV I bought (again at Fry's) was the same as a model from another store, but with the built-in DVR disabled. That leaves the burden on you to figure out which model was in Consumer Reports by matching up the features. It's annoying, but possible.
- curryst 6y ago
  > This would be less a problem if brands followed some kind of trend in overall reliability, but it seems like individual models vary widely in quality even within a specific brand. This is, frustratingly, incredibly true. Even otherwise good products will tend to have some single component with a significantly lower lifespan than the rest of the device. This is compounded by the fact that everything is now manufactured to such exact specifications that it becomes effectively impossible to replace that piece with a higher quality one they should have used in the first place. Then you have the insane number of SKUs a single base device can generate, and it becomes difficult to tell whether the thing you're holding in your hand has the shitty version of that component, or the better one. Sometimes it's even the same SKU and there's a range of bad serial numbers (typically because the manufacturer got a bad batch of something from one of their suppliers). My guess is that the sheer number of parts in many modern products has drastically increased the chance that a device fails. My office has a water cooler with a purifier, and a spout for hot water. It broke and a guy came in to fix it. There are 2, 2 circuit boards in that thing! For a water cooler. How many parts are there that could fail and brick it? How many would a more primitive water cooler have? I just can't believe the simplest water cooler we could invent has 2 separate circuit boards in it
devrand 6y ago
It's a bit of a different model though since Consumer Reports are often producing the reviews themselves (which may or may not include some survey data from users) and have first-hand experience to editorialize. With things like Yelp [1] and particularly with Glassdoor this is a difficult model to emulate. You generally expect users themselves to contribute quality reviews which you then publish directly (plus maybe do a bit of curation to gather sentiment). It's difficult to both ask users to pay you for the service and also contribute the very content you charge them for. I think the only way to make a user review site work are: (a) Be a non-profit and have a donation based model. Would people actually donate though? I'm not sure... but at least you don't seem like a greedy middleman. (b) Charge a membership fee, but grant free durations of membership for quality reviews. The downside here is that you're likely severely limiting the already small pool of people willing to leave a review. [1]: The nearest comparison to CR<->Yelp would be Zagat (well, pre-Google Zagat) and Michelin. However, they both only cover a narrow slice.
- nerdponx 6y ago
  What about Angie's List?
  - function_seven 6y ago
    They're just a clone of Home Advisor now. IAC bought them a few years back. They no longer charge their users.
- bryanlarsen 6y ago
  Excellent point. A glassdoor competitor cannot use the Consumer Reports model since you can't review the working conditions of a company without being inside it. It shouldn't take much money to set up and run a review site so monetization should be a soluble problem. On a 6 figure budget you won't be as slick as competitors but if you're competing on trustworthiness... The biggest problem is going to be combating gamification, that'll be very hard on a 6 figure budget. And even your non-profit suggestion is a problem because what if a company donates $1M with strings attached? I really think the only solution is that all money has to come from consumers. Job hunting is important enough that people would be willing to shell out for it, but only once you have a sufficient pool of reviews. So a real chicken and egg problem here.

jdofaz 6y ago

For whatever reason the disc rental side of Netflix still has user reviews and the 5 star system

InitialLastName 6y ago

> But Netflix for example can't piss off the movie companies or they risk losing their catalog More than that, Netflix needs your eyes on their service, as opposed to the (dozen?) other competing, equally shoddy services. What you watch makes no difference to their bottom line... the only thing that matters is that you don't see a bad review on Netflix and switch to another streaming service (because if you do that too often, you might decide to remove Netflix from your list of $10/month services).

city41 6y ago

Reviews for entertainment are of very little value, and always have been. The only time this isn't true is if you get to know the reviewer and understand that your tastes and theirs line up. I suppose if something is universally panned that could be useful information, but that isn't very common.