5 ms·

Negative reviews can generally be tracked easily by registered user. It seems like a good enough idea that the force of negative reviews should fall off exponen

Negative reviews can generally be tracked easily by registered user. It seems like a good enough idea that the force of negative reviews should fall off exponentially with respect to frequency per user that I'm surprised its either not done more, or more apparent. If someone complains all the time about everything, the people around them quickly learn to ignore it. Our systems should too.

juhzy 8y ago
Why? I only bother to write negative reviews, does that mean my opinion is worthless, or that I think all restaurants/venues/products are bad?
- byproxy 8y ago
  No, but it shows that you only care to share in the bad, not the good. There is a tint to your worldview and it'd be useful for others to see that when weighing what you have to say. For the record, I rarely leave reviews. When I do, though, it's only for the exceptionally bad or the exceptionally good.
  - juhzy 8y ago
    Why would I write a good review? If I go to a place and pay I expect to be treated well; being treated the way I expect is not noteworthy (or "reviewworthy")
    - byproxy 8y ago
      I'm not trying to change your process, but just letting you know that that process is not the same for everyone, hence it'd be useful for others to know your "tint." It's interesting that your main criteria is how you feel you were treated, though. Discussed in the article: "Restaurant reviews in which people sound traumatized by perceived injustice don’t tend to comment much on the food — it’s usually the perception of being treated rudely or uncaringly that seems to have pushed people into processing by writing out their feelings in a public forum."
      - juhzy 8y ago
        I think it's normal. Poor food may mean many things and most of them are not malicious. But being treated rudely or uncaringly? You deserve your bad review.
        byproxy 8y ago
        >Poor food may mean many things and most of them are not malicious. I honestly feel the same about poor service. I'm usually accommodating and understanding, but I'm no monk. Of course there are times when I feel either the poor food or poor service merit some mention. Most of the time, however, I think "they're human, going through human things. No big deal."
    - cirgue 8y ago
      Because the world is not divided into good and bad, and I care about the 'why' of a rating much more than I care about whether it's positive or negative.
    - intopieces 8y ago
      Your response sounds flippant, but I am with you, except I don’t see the value in writing reviews at all, good or bad. I have no stake in whether another person is persuaded to patronize or avoid a place. Moreover, I doubt any online review I write would have any impact on the business itself. If I have a specific complaint for a place close to my heart, like a coffee shop or restaurant or local shop, I’ll talk to the manager, privately and calmly, and be on my way.
    - munchbunny 8y ago
      Most people look at reviews to see whether the thing actually is what it claims to be, so reviews saying that it is in fact what it claims to be are useful. Doesn't mean you have to, just means that it'd be more helpful to others if you write "it actually is what it claims to be" reviews.
    - gowld 8y ago
      What's the difference between an entity with a long, postitive reputation, and a new unknown entity? Do you consider them equally attractive?
  - jjoonathan 8y ago
    I'd argue that the bias is a natural consequence of two interacting effects, neither of which necessarily represents a "tinted worldview": a skew in quality distribution and a minimum threshold for information value. It's hard for a product or service to exceed its expected value by more than a fraction or small multiple, while it's easy to cause misery well in excess of a dozen or even a hundred times the expected value. I believe that's more a reflection of it being easier to destroy vs create rather than a reflection of psychology. Combine this distribution with a heuristic to only report deviations from expectation in excess of a minimum threshold and the "average review score" will reliably undershoot the actual quality. That's only a problem if you want to interpret the average review score as an absolute measure of quality, though, and I don't think anyone really does. Most of us are more interested in communicating and informing our decision processes than in passing moral judgement, and if our goal is to optimize communication then we should expect negative reviews to dominate the discussion because they're inherently capable of more meaningful excursions from the mean. I agree with your overall point: it would be fantastically useful to be able to contextualize reviews against reviewer psychology. That way I could ignore both shouting from negative-nellys and forced positivity from those who feel compelled to balance the universe :-)
    - byproxy 8y ago
      >if our goal is to optimize communication then we should expect negative reviews to dominate the discussion because they're inherently capable of more meaningful excursions from the mean. In reality, however, it seems that positive reviews tend to dominate. Using Google Maps reviews as my barometer, I hardly ever see any place rated less than 4.5 stars. So, I tend to think to myself "4.5-5 stars: might be good. 4 stars: probably okay. Less than 4: maybe steer clear." Though, in practice I disregard reviews, take a plunge, and then decide on my own. Often I find myself in conflict with the average majority opinion.
- noonespecial 8y ago
  You are exactly the intended use case. If you write a bad review every time the system will soon ignore you. If you seldom write a review but bad is the only kind you ever write, each will carry some weight after a significant "cool-down period". In other words, we successfully encode the difference between "wow, juhzy only complains when its really bad" vs "ignore juhzy, they complain about everything." This vs. the more naive and destructive "5 bad reviews and you're out" we seem to do too much of now. I suppose it should work the other way too. Someone who always "5-stars" everything should be ignored as well. The point is that the system right now seems to have a negative bias and is able to do so because of a glut of providers which it burns through like an expendable, renewable resource. (Which morally sucks because they are real people with real lives.)
  - gowld 8y ago
    What is the difference between "every time" and "seldom" ? Yelp has no idea how often I eat out at restaurants.* * just kidding; of course they do; they subscribe to a feed of my location data published by some game on my phone.
    - noonespecial 8y ago
      Your * is very insightful. Implicit in my suggestion is the fact that the service is able to count how often you use it vs your +ve and -ve reviews. When I say "seldom" I of course mean "uses the service often, but seldom complains". The unit of time in this exponential weighting is not chronological, but uses of the service. This of course is harder with yelp than uber.
- wpietri 8y ago
  It's certainly worthless to me. Some people can complain about anything. (When I'm in a grumpy mood, I certainly will. Puppies? Too wiggly. Sunshine? Too bright!) When I read a review, my goal is to find out what I would think of the place or thing. The reviewers I look for are balanced and thoughtful, able to point the good points about something they hate and the bad points of the things they love. I have a very smart friend who apparently has never liked a movie. If one comes up, he will have a complaint about it. If you see one at a theater with him, before the credits have finished rolling, you'll be getting a list of his issues. It's exhausting. And it means I'll never listen to him on the topic of whether a movie is good. It's the classic stopped clock: it's right twice a day, but you can't know which times, so why look at it?
- munchbunny 8y ago
  If the ratings scale is 1-10 and all of your ratings are 1-5, does that mean you only give reviews to things in your 1-5 range, or does that mean everyone else's 10 is your 5? From a perspective that is external to your inner monologue, I have no way of knowing. Even if your criticisms are valid, I know too many people who seem to never talk about the positive parts to know whether you're just perpetually unimpressed. Or, more likely, you're the type that never talks about the nice parts. I'm guessing this because of your comment elsewhere in the tree that specifically asks why one should compliment when something is doing what it's supposed to do. (The answer is that people usually look at reviews for confirmation that X is as advertised.) Or there's actually nothing redeeming about the subject of your review. So yes, taken in context I would end up throwing your opinion away. In practice, I don't check every reviewer's review history, but maybe that would be a useful signal to see.
  - gowld 8y ago
    The honest opinion of a stranger is useless. A summary of the opinions of many strangers has some signal. The facts of a stranger have some utility.
Bartweiss 8y ago
What surprises me is that sensible review systems seem to be used least by the sites that ought to know best. Amazon, as we all know, basically expresses quality as a raw average of stars plus a histogram, with no correction except for removing reviews. Meanwhile, BeerAdvocate is essentially just a labor-of-love beer tracking site, but it offers multiple secondary stats about ratings. Each beer gets a percentage deviation stat (pDev) to show how varied their rating are, each specific review lists its deviation (rDev) from the product's average review, and each reviewer's profile offers their frequency of reviewing above, below, and inside the average window (|rDev| - pDev > 0, then rDev - pDev). I don't think anything is actively done with those stats, but even that's enough to spawn forum threads where reviewers discuss which beers are most controversial, which beers they gave outlier reviews to, and whether they're typically harsh or generous reviewers. The site also recommends rating sub-categories to get people thinking about different aspects of a problem, while Amazon is laden with reviews that are either myopic (e.g. that XKCD about a tornado tracker) or completely off-topic (e.g. about the seller instead of the product). A site that wanted to go a little further could use the same stats I mentioned for spam detection (there are several papers on doing that effectively) and for score correction. (Basically, take a user's average deviation over all reviews and use that to scale or adjust their impact on averages.) Given how easy all that is, why do Yelp and Amazon consistently have some of the least useful reviews and averages of any site I know? I suspect this article nails it - casual readers appreciate the simplicity more than depth.
- wtracy 8y ago
  Now I'm imagining a system that never displays a user's absolute rating. Instead, ratings are only displayed as a number of standard deviations above or below that user's mean rating. The main drawback I see (assuming that you are seeking an honest picture of that provider's quality, and not a tool for punishment) is that you don't meaningfully capture people who only write reviews to flag serious problems.
  - Bartweiss 8y ago
    > you don't meaningfully capture people who only write reviews to flag serious problems Yes - reading the comments here I'm realizing a major problem with any reviewer-centered system is that people decide whether to review on hugely varied conditions. An always-five-stars reviewer might just be easily impressed (or a fraud), but they equally might subscribe to "if you don't have anything nice to say...". And quite a lot of people write exclusively bad reviews, but it's not obvious how to discern grumpy reviewers from people who only speak up about major issues. A partial fix might be available by analyzing how far a given review is off user's average difference from product average, which could discern a five-star bot from a person who only reviews great products. But even that doesn't solve the XKCD problem where a product has median-case appeal but a high rate of critical failures. In true "what can't meta-analysis fix?" style, this could be improved by looking at a user's average distance outside 1SD of the mean review, or perhaps by special-casing products with multimodal reviews. Of course, it's deeply unclear how to convert this to an output. Scaling reviews based on reviews sounds like a nightmare, reviews shouldn't be differential equations and no one wants to see 4+ layers of statistics to buy a new lamp. Perhaps all of the indirect work could be done off raw ratings behind the scenes to produce a general "adjusted average" for display? (More realistically, the serious-problem case only seems solvable by reading text reviews, and just devaluing outright fraud and always-angry cranks would be a massive improvement over existing systems.)
tivert 8y ago
> Negative reviews can generally be tracked easily by registered user. It seems like a good enough idea that the force of negative reviews should fall off exponentially with respect to frequency per user that I'm surprised its either not done more, or more apparent. > If someone complains all the time about everything, the people around them quickly learn to ignore it. Our systems should too. That assumes that "too much" negative criticism is false and merely the result of a disagreeable personality. I think that assumption is false. I think that's false, people who frequently post negative reviews may just be the kind of people who have legitimately higher standards vs. people who are happy with even objectively crappy things. I don't think the frequency of negative reviews, by itself, gives you any real information about the quality of the reviewer.
jdc 8y ago
That's true. And vice-versa. Reminds me of the infographic where they compare American and Eastern-European responses to quality (https://i.imgur.com/h4ybnAN.jpg https://i.imgur.com/h4ybnAN.jpg).

juhzy 8y ago

Why? I only bother to write negative reviews, does that mean my opinion is worthless, or that I think all restaurants/venues/products are bad?

byproxy 8y ago
No, but it shows that you only care to share in the bad, not the good. There is a tint to your worldview and it'd be useful for others to see that when weighing what you have to say. For the record, I rarely leave reviews. When I do, though, it's only for the exceptionally bad or the exceptionally good.
- juhzy 8y ago
  Why would I write a good review? If I go to a place and pay I expect to be treated well; being treated the way I expect is not noteworthy (or "reviewworthy")
  - byproxy 8y ago
    I'm not trying to change your process, but just letting you know that that process is not the same for everyone, hence it'd be useful for others to know your "tint." It's interesting that your main criteria is how you feel you were treated, though. Discussed in the article: "Restaurant reviews in which people sound traumatized by perceived injustice don’t tend to comment much on the food — it’s usually the perception of being treated rudely or uncaringly that seems to have pushed people into processing by writing out their feelings in a public forum."
    - juhzy 8y ago
      I think it's normal. Poor food may mean many things and most of them are not malicious. But being treated rudely or uncaringly? You deserve your bad review.
      - byproxy 8y ago
        >Poor food may mean many things and most of them are not malicious. I honestly feel the same about poor service. I'm usually accommodating and understanding, but I'm no monk. Of course there are times when I feel either the poor food or poor service merit some mention. Most of the time, however, I think "they're human, going through human things. No big deal."
  - cirgue 8y ago
    Because the world is not divided into good and bad, and I care about the 'why' of a rating much more than I care about whether it's positive or negative.
  - intopieces 8y ago
    Your response sounds flippant, but I am with you, except I don’t see the value in writing reviews at all, good or bad. I have no stake in whether another person is persuaded to patronize or avoid a place. Moreover, I doubt any online review I write would have any impact on the business itself. If I have a specific complaint for a place close to my heart, like a coffee shop or restaurant or local shop, I’ll talk to the manager, privately and calmly, and be on my way.
  - munchbunny 8y ago
    Most people look at reviews to see whether the thing actually is what it claims to be, so reviews saying that it is in fact what it claims to be are useful. Doesn't mean you have to, just means that it'd be more helpful to others if you write "it actually is what it claims to be" reviews.
  - gowld 8y ago
    What's the difference between an entity with a long, postitive reputation, and a new unknown entity? Do you consider them equally attractive?
- jjoonathan 8y ago
  I'd argue that the bias is a natural consequence of two interacting effects, neither of which necessarily represents a "tinted worldview": a skew in quality distribution and a minimum threshold for information value. It's hard for a product or service to exceed its expected value by more than a fraction or small multiple, while it's easy to cause misery well in excess of a dozen or even a hundred times the expected value. I believe that's more a reflection of it being easier to destroy vs create rather than a reflection of psychology. Combine this distribution with a heuristic to only report deviations from expectation in excess of a minimum threshold and the "average review score" will reliably undershoot the actual quality. That's only a problem if you want to interpret the average review score as an absolute measure of quality, though, and I don't think anyone really does. Most of us are more interested in communicating and informing our decision processes than in passing moral judgement, and if our goal is to optimize communication then we should expect negative reviews to dominate the discussion because they're inherently capable of more meaningful excursions from the mean. I agree with your overall point: it would be fantastically useful to be able to contextualize reviews against reviewer psychology. That way I could ignore both shouting from negative-nellys and forced positivity from those who feel compelled to balance the universe :-)
  - byproxy 8y ago
    >if our goal is to optimize communication then we should expect negative reviews to dominate the discussion because they're inherently capable of more meaningful excursions from the mean. In reality, however, it seems that positive reviews tend to dominate. Using Google Maps reviews as my barometer, I hardly ever see any place rated less than 4.5 stars. So, I tend to think to myself "4.5-5 stars: might be good. 4 stars: probably okay. Less than 4: maybe steer clear." Though, in practice I disregard reviews, take a plunge, and then decide on my own. Often I find myself in conflict with the average majority opinion.
noonespecial 8y ago
You are exactly the intended use case. If you write a bad review every time the system will soon ignore you. If you seldom write a review but bad is the only kind you ever write, each will carry some weight after a significant "cool-down period". In other words, we successfully encode the difference between "wow, juhzy only complains when its really bad" vs "ignore juhzy, they complain about everything." This vs. the more naive and destructive "5 bad reviews and you're out" we seem to do too much of now. I suppose it should work the other way too. Someone who always "5-stars" everything should be ignored as well. The point is that the system right now seems to have a negative bias and is able to do so because of a glut of providers which it burns through like an expendable, renewable resource. (Which morally sucks because they are real people with real lives.)
- gowld 8y ago
  What is the difference between "every time" and "seldom" ? Yelp has no idea how often I eat out at restaurants.* * just kidding; of course they do; they subscribe to a feed of my location data published by some game on my phone.
  - noonespecial 8y ago
    Your * is very insightful. Implicit in my suggestion is the fact that the service is able to count how often you use it vs your +ve and -ve reviews. When I say "seldom" I of course mean "uses the service often, but seldom complains". The unit of time in this exponential weighting is not chronological, but uses of the service. This of course is harder with yelp than uber.
wpietri 8y ago
It's certainly worthless to me. Some people can complain about anything. (When I'm in a grumpy mood, I certainly will. Puppies? Too wiggly. Sunshine? Too bright!) When I read a review, my goal is to find out what I would think of the place or thing. The reviewers I look for are balanced and thoughtful, able to point the good points about something they hate and the bad points of the things they love. I have a very smart friend who apparently has never liked a movie. If one comes up, he will have a complaint about it. If you see one at a theater with him, before the credits have finished rolling, you'll be getting a list of his issues. It's exhausting. And it means I'll never listen to him on the topic of whether a movie is good. It's the classic stopped clock: it's right twice a day, but you can't know which times, so why look at it?
munchbunny 8y ago
If the ratings scale is 1-10 and all of your ratings are 1-5, does that mean you only give reviews to things in your 1-5 range, or does that mean everyone else's 10 is your 5? From a perspective that is external to your inner monologue, I have no way of knowing. Even if your criticisms are valid, I know too many people who seem to never talk about the positive parts to know whether you're just perpetually unimpressed. Or, more likely, you're the type that never talks about the nice parts. I'm guessing this because of your comment elsewhere in the tree that specifically asks why one should compliment when something is doing what it's supposed to do. (The answer is that people usually look at reviews for confirmation that X is as advertised.) Or there's actually nothing redeeming about the subject of your review. So yes, taken in context I would end up throwing your opinion away. In practice, I don't check every reviewer's review history, but maybe that would be a useful signal to see.
- gowld 8y ago
  The honest opinion of a stranger is useless. A summary of the opinions of many strangers has some signal. The facts of a stranger have some utility.

Bartweiss 8y ago

What surprises me is that sensible review systems seem to be used least by the sites that ought to know best. Amazon, as we all know, basically expresses quality as a raw average of stars plus a histogram, with no correction except for removing reviews. Meanwhile, BeerAdvocate is essentially just a labor-of-love beer tracking site, but it offers multiple secondary stats about ratings. Each beer gets a percentage deviation stat (pDev) to show how varied their rating are, each specific review lists its deviation (rDev) from the product's average review, and each reviewer's profile offers their frequency of reviewing above, below, and inside the average window (|rDev| - pDev > 0, then rDev - pDev). I don't think anything is actively done with those stats, but even that's enough to spawn forum threads where reviewers discuss which beers are most controversial, which beers they gave outlier reviews to, and whether they're typically harsh or generous reviewers. The site also recommends rating sub-categories to get people thinking about different aspects of a problem, while Amazon is laden with reviews that are either myopic (e.g. that XKCD about a tornado tracker) or completely off-topic (e.g. about the seller instead of the product). A site that wanted to go a little further could use the same stats I mentioned for spam detection (there are several papers on doing that effectively) and for score correction. (Basically, take a user's average deviation over all reviews and use that to scale or adjust their impact on averages.) Given how easy all that is, why do Yelp and Amazon consistently have some of the least useful reviews and averages of any site I know? I suspect this article nails it - casual readers appreciate the simplicity more than depth.

wtracy 8y ago
Now I'm imagining a system that never displays a user's absolute rating. Instead, ratings are only displayed as a number of standard deviations above or below that user's mean rating. The main drawback I see (assuming that you are seeking an honest picture of that provider's quality, and not a tool for punishment) is that you don't meaningfully capture people who only write reviews to flag serious problems.
- Bartweiss 8y ago
  > you don't meaningfully capture people who only write reviews to flag serious problems Yes - reading the comments here I'm realizing a major problem with any reviewer-centered system is that people decide whether to review on hugely varied conditions. An always-five-stars reviewer might just be easily impressed (or a fraud), but they equally might subscribe to "if you don't have anything nice to say...". And quite a lot of people write exclusively bad reviews, but it's not obvious how to discern grumpy reviewers from people who only speak up about major issues. A partial fix might be available by analyzing how far a given review is off user's average difference from product average, which could discern a five-star bot from a person who only reviews great products. But even that doesn't solve the XKCD problem where a product has median-case appeal but a high rate of critical failures. In true "what can't meta-analysis fix?" style, this could be improved by looking at a user's average distance outside 1SD of the mean review, or perhaps by special-casing products with multimodal reviews. Of course, it's deeply unclear how to convert this to an output. Scaling reviews based on reviews sounds like a nightmare, reviews shouldn't be differential equations and no one wants to see 4+ layers of statistics to buy a new lamp. Perhaps all of the indirect work could be done off raw ratings behind the scenes to produce a general "adjusted average" for display? (More realistically, the serious-problem case only seems solvable by reading text reviews, and just devaluing outright fraud and always-angry cranks would be a massive improvement over existing systems.)

tivert 8y ago

> Negative reviews can generally be tracked easily by registered user. It seems like a good enough idea that the force of negative reviews should fall off exponentially with respect to frequency per user that I'm surprised its either not done more, or more apparent. > If someone complains all the time about everything, the people around them quickly learn to ignore it. Our systems should too. That assumes that "too much" negative criticism is false and merely the result of a disagreeable personality. I think that assumption is false. I think that's false, people who frequently post negative reviews may just be the kind of people who have legitimately higher standards vs. people who are happy with even objectively crappy things. I don't think the frequency of negative reviews, by itself, gives you any real information about the quality of the reviewer.

jdc 8y ago

That's true. And vice-versa. Reminds me of the infographic where they compare American and Eastern-European responses to quality (https://i.imgur.com/h4ybnAN.jpg https://i.imgur.com/h4ybnAN.jpg).