8 ms·

Eh, Goodreads has always suffered from the same problem that plagues every other review system which uses "score out of X" ranking. Humans just aren't very goo

Eh, Goodreads has always suffered from the same problem that plagues every other review system which uses "score out of X" ranking. Humans just aren't very good at ranking things on a normal distribution, so you invariably end up with every item (books in this case) being ranked somewhere in the 3.5-4.5 range (since Goodreads is out of 5). For IMDB the rankings all hover around 8ish. When in reality the average book should have a 2.5. If you don't rate like this then you just end up with garbage. Just allowing a boolean rating (ala Rotten Tomatoes when aggregated) is much better, assuming you can get enough reviews for that system to actually work (probably > 30 is required for most applications). I think "aggregated personal Elo" would be a fun way to rank things: I just give you two books that you've read and you tell me which is better. Do this loads of times and eventually you have a solid ranking of every book you've ever read. Aggregate everyone's rankings and you have a much more robust system then "please rate this book out of 5 stars".

fourmajor 3y ago
I disagree that stars should be evenly distributed between 1 to 5 stars. I think it's quite possible that most books that people choose to read end up being a 3 (good with some flaws) or 4 (good but not all-time great). It's kind of like the same thing with pizza. I'd give most pizza a 3 or 4. Very few 1s and 5s to be sure. 1 doesn't have to mean bottom 20% of pizzas. It can mean "awful, couldn't finish," where very few pizzas would fall into that category. 5 doesn't need to mean "best 20% of pizzas," it can mean "telling strangers about it the next day," again where very few pizzas would fall into that category.
- deleted 3y ago
  [deleted]
- LVB 3y ago
  I had a debate on the Criticker site about this topic since they try hard to turn your raw ratings into a normalized span. The fact is that because of ratings sites, I'm very rarely watching 1-star movies. That's good! So the low end was dominated by a 2.5/3 type ratings, which per my scale meant "just OK," but they were mapping it to "bad," somehow inferring that my rating range is 3-5.
- kortilla 3y ago
  “Normal distribution” doesn’t mean evenly distributed. Most being in the middle is a normal distribution.
nextos 3y ago
There's also TrueSkill, by Microsoft Research, which turns Elo into a proper generative model: https://en.wikipedia.org/wiki/TrueSkill https://en.wikipedia.org/wiki/TrueSkill It'd be definitely interesting to see a book review site like this. In the meantime, for math books, I trust curated rankings by MAA: https://www.maa.org/press/maa-reviews https://www.maa.org/press/maa-reviews
- pizza_pleb 3y ago
  We use this at https://languageroadmap.com https://languageroadmap.com, but for difficulty rankings between titles in the context of language learning using media. It does seem to work pretty well and we disclaim rankings with the confidence score. As for the grandparent comment: recency bias, as pointed out by another commenter is a thing, as is the tediousness of doing a bunch of pairwise decisions. I think a happier medium is to have everyone fill in tier charts (with variable number of tiers) and build the pairwise rankings from that.
pessimizer 3y ago
> I just give you two books that you've read and you tell me which is better. I've always been so for this in situations where the users also have a database of their own history on the site, like Goodreads or Boardgamegeek.com, for example. Just throw up modals every once and a while comparing two books that you know the user knows, and make it easy to opt out of. I probably wouldn't opt out, because it would appeal to my ego and feel like a game. > Humans just aren't very good at ranking things on a normal distribution, so you invariably end up with every item (books in this case) being ranked somewhere in the 3.5-4.5 range (since Goodreads is out of 5). For IMDB the rankings all hover around 8ish. When in reality the average book should have a 2.5. If you don't rate like this then you just end up with garbage. Just normalize the users against themselves. I wouldn't be giving a 4-star review, I'd be giving a pessimizer(4-star review).
deleted 3y ago
[deleted]
dreamcompiler 3y ago
And in many such systems the peer pressure to give a 5 is tremendous, which further devalues the rating system. Uber drivers for example have told me that if more than a few customers give them a 4 they will be effectively fired. Restaurants, tour companies etc have asked me to please give them the maximum number of stars on Yelp, Google, Tripadvisor, etc lest there be dire consequences for them. This is of course Goodheart's Law in action but I don't have any ideas for fixing the problem.
tecleandor 3y ago
Seems like this is culturally different for some things. And corporations can influence too. I have a friend who has four restaurants in Tokyo, and I've been several times there. If you keep attention to the restaurant reviews in Google Maps, Japanese people is very hard. They'd go like "The food is great, incredible service, surprising flavors, very good experience, best Spanish food I've had in a long time..." and then they go "...but I know they can do better and have space to improve. 2/5". In Spain we would be like "Nice beer, they gave us some tapas. 5/5" Gig economy corporations also have made this worse. You're scared to score your driver, your server or your hotel by less than a 5/5 or somebody might get punished or fired (and maybe they didn't even had a contract in the first place). And this becomes a snake biting its own tail. Now I rarely go to a place with a score of less than a 4/5, and I'll score relatively high because I know I could influence votes out of what people consider "worthy to visit".
- JohnFen 3y ago
  > I rarely go to a place with a score of less than a 4/5 I have a simpler heuristic. I ignore all customer ratings entirely, and refuse to leave any.
  - deleted 3y ago
    [deleted]
- yurishimo 3y ago
  100% it's cultural. In the Netherlands, companies brag about a 7 or 8 out of ten as proof of their amazing customer service. When I lived in America, it seemed the consensus of anything under a 9 is killing puppies. Coincidentally, NPS is entirely based on this fact. 8+ is required for a "happy customer". 7 is given a little wiggle room to win them back and under 6 is a lost cause.
  - user_7832 3y ago
    > In the Netherlands, companies brag about a 7 or 8 out of ten as proof of their amazing customer service. Huh, could you give an example of it? I am new to the Netherlands and hadn’t realized this yet.
berkes 3y ago
I really like the idea of relative ratings. But that also amplifies a problem that exists with any rating: recency bias, or any sort of time bias. I change over time. An album that I loved when I was 15 could get a 1/5 from me today. Or a book that I hated in my teens, may now be my favorite. It's complicated.
- MrVandemar 3y ago
  In which case the system would need to continually challenge you about your historical choices, perhaps based on recent choices and detecting possible paradoxes (you liked A more than B, and you liked B more than C, but you also liked C more than A).
  - berkes 3y ago
    Very good solution indeed! Maybe also allow ratings to be a list over time rather than one, single data point. That'll make calculating an average more involved, but also more realistic.
thebigspacefuck 3y ago
IIRC they initially gave you a text for each star like 1 - I hated it 2 - It was okay 3 - I liked it 4 - It was great 5 - I loved it I thought it was helpful for framing what a score would be but I haven’t seen this in the app or anywhere for a long time. At least it avoids the problem of Amazon where a review is like “Best book I’ve ever read but the copy I received had a rip on the dust jacket - 1/5”. Learned through experience to skip anything below a 4 or very high 3 now. Probably missing some interesting stuff but I don’t read that often anyway.
waspight 3y ago
I have thought about this relative rating as well. Would it be possible to use? I mean, does it scale? I also have seen some ratings where you can only rate between 5 - 10, I guess since no one ever rates below 5 anyway. I think it kind of works.
throw0101a 3y ago
For a while now I've though that maybe only three rankings are needed: * Recommend this thing * Neutral opinion * Do not recommend this thing Not sure how much "neutral" would be used: kind of like thumbs-up/down or up-down-vote mechanic: if you're neutral you may simply not vote at all.
hospitalJail 3y ago
The Modern Zelda reviews are a prime example of this. The non-industry people reviewing Zelda have 4 options, 7/10, 8/10, 9/10, 10/10. A 7/10 means the game was bad and you had a hard time finishing it, and won't be playing again. 8/10 means the game was also bad, but you had fun for a few minutes/hours. 9/10 means the game meet minimum expectations. 10/10 means you enjoyed the game, but there were countless flaws that took away from enjoying the game. 10/10 Greatest game of all time means, the game was above average. Now, this same system applied outside Nintendo's curve, you knock these numbers down 2. Its incredibly hard to figure out if a Nintendo game is good, if you treat them like other companies. Even Nintendo believes this and will yank early access/ads/etc... from gaming websites if you don't comply with the Nintendo curve.
- yurishimo 3y ago
  Devils advocate; Is it not possible that Nintendo just makes good games? They spend a lot of time on development, don't rush things, the trailers are honest about gameplay and what to expect. I can understand it's not your cup of tea or arguments that their online service is shit (it is) but reviewing a video game in isolation, many of them seem to be good games objectively. Games that don't appeal to the average CoD fan sure, but that doesn't mean they are bad games.
  - hospitalJail 3y ago
    Nah, you can see it on subreddits as people discuss the game. Complaining about a few enemies, copypaste world, etc... Its not a 10/10, but it will still be reviewed like this. Imagine you got rid of the Zelda skins and Zelda name, released it on Xbox. What would it get then? Don't get me wrong, I religiously play all the zelda's and find ways to enjoy them. I am under Nintendo's spell, but I also know these are the corporate mascots I grew up with. Nintendo markets to children, we basically need therapy if we want to be free from their grip.
    - yurishimo 3y ago
      That’s fair. I would take a minor issue with your framing (someone who didn’t play BotW wouldn’t find the map a cop out and I’m glad they put the effort in for the 2 other layers of the map) but you could make those same arguments against COD or Destiny. I do think it’s worth noting how many of the ideas from BotW have been copied in the past 6 years in other games. I guess my point is, I can understand the 10/10s but I also find it entirely reasonable to view it closer to a 7 or 8. We’re also not thinking about things in a vacuum in this discussion. Is the Xbox game released as if BotW never happened? Nintendo certainly uses its brand to buy marketing influence but importantly, I don’t think the company is pissed when games fail. Skyward Sword was a good example of that actually. It’s failure literally led to BotW. Nintendo got off their laurels and reinvented their “tried and true” formula and now it’s paying off. I am closer your position for games like 2D Mario. Zero innovation and I’ll be eagerly watching the reviews for the new game later this year.
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
TRiG_Ireland 3y ago
> I just give you two books that you've read and you tell me which is better. I can't wait to compare The Pickwick Papers against The Unfolding of Language.
- gavmor 3y ago
  A solid point: books may excel at one dimension while not even attempting another. I suppose one might have to ascertain what genre, exactly, a book falls in, and how it fares in its own genre. But then you're not comparing pairwise, and we lose the benefits of our visual cortex or whatever neural system is so good at pairwise comparisons. Constraining pairings within their genres might suffice, if possible. There are several standard ontology for books, and it's not a trivial problem.
- MrVandemar 3y ago
  I don't think it makes much sense to compare a fiction book to a non-fiction book. There's not enough points of comparison. While both can be compared on quality of prose, the fiction might be judged on characters and plot, and the non-fiction book on accuracy and detail. You might as well ask the question: Who is the better Tennis player? Roger Federer or Garry Kasparov? A system would need to draw some kind of boundary conditions so you are comparing things that have more in common than "printed on paper and bound".
cratermoon 3y ago
> Humans There's the problem there – assuming human input makes up an important part of the rankings.
sedatk 3y ago
Yes, one person I knew rated my book 2/5 because, as he said, it was beginner/mid-level-oriented and he wasn't aware when he purchased it (clearly stated on the book's page, and pretty much everywhere). A star rating system gives more emphasis on an individual's own preference of what a good rating should be which works against the consumer.
jsmith99 3y ago
It's not just because humans are bad at distributions. I would rate almost all the books I read at least 3/5 because I pick my books carefully. There are plenty of terrible books deserving 1/5 but they are often obvious from a distance. With eg movies a much greater investment is required to tell if it's terrible.
ROTMetro 3y ago
In medicine they use a pain scale with faces when they need to find out from kids how much pain they are in. Something like that might be a better system?
bstpierre 3y ago
See https://www.criticker.com/explain/ https://www.criticker.com/explain/ -- it works sort of like what you're suggesting. You rank movies from 0-100, which is different from rating them. Your percentile ranking scores are compared to other users' rankings, and then it can suggest movies that it thinks you will rank highly. IMO the real problem with something as subjective as books or movies is that even completely honest, well-reasoned reviews are going to be all over the map. My review of a Pride and Prejudice movie is going to be maybe 3/10, but my wife would give it 7/10, while we have the opposite reactions to something like The Hunt for Red October. I don't care about reviews from experts or the unwashed masses. I don't even really care about reviews that much -- I'm more interesting in ratings from people who like the same kind of stuff I do.
mahoho 3y ago
Well rating scales don't have any absolute meaning, only that which is attributed to them either by the reviewer or the audience. On any aggregate review site the scale of the aggregate score will take on its own semantics, which is the aggregate of all the users' own semantics they use for their individual scores. And for most people, it seems like the semantics of rating scales don't align with a normal distribution about the 50% mark, but something more like letter grades where 70% is decent/mediocre, while 50% is a failure (in the US). IMO this makes at least as much sense as centering around 50%; if you only score 50% of the points on a test I don't think you're competent in a subject, and if your book only meets 50% of the criteria for a great book then it might be a bad book. But again, it's all arbitrary and only makes sense in the context of the particular reviewer or aggregate community.
ohlookcake 3y ago
> the average book should have a 2.5 Disagree on this. This assumes that people read books completely at random. In reality, people read blurbs/summaries or get a recommendation and the typical book you read is likely to be better than the midpoint between the worst book you've read and the best book you've read (I'm assuming a linear scale mental model).
- CSMastermind 3y ago
  > This assumes that people read books completely at random. I'm not sure it implies that. I think that if: 1. Book quality follows a normal distribution. and 2. People are capable of accurately assessing the quality of books relative to other books. then you'd end up with an average rating of 2.5. It doesn't matter if people are disproportionately reading the better books and rating them highly because an increasing number of reviews on a book would only give you more confidence in its rating, not change the rating of the book relative to other books. To illustrate this we can imagine a world with only 100 books in it and only 100 people in it. Let's assume that a single person from that group reads all 100 books and per the two assumptions above correctly rates them on a normal distribution. Then the other 99 people read only a single book which the first person recommends as a 5. And per the two assumptions above they all also rate it as a 5. Now there are 99 books with only 1 rating normally distributed between 1 and 5 and there is 1 book with 100 ratings - all 5s. Does this change the average rating of books on the platform? Nope the average rating is still 2.5. In fact the one book could have a million ratings or a billion 5 star ratings and the average rating of books on the platform would still be a 2.5. What we see on sites like goodreads is that either book quality doesn't follow a normal distribution and/or people are not capable of accurately assessing the quality of books relative to other books.
  - therealdrag0 3y ago
    Good points. But do we have evidence GR is not a normal distribution. GP’s comment does apply to views of ratings, so we might have an impression if higher average rating because the books we look at are more often better than avg.
  - 3y ago
aleksiy123 3y ago
Elo/Trueskill sounds like a solid idea. When you go to rate a book the system asks you if it was better or worse (maybe same) then a book of you previously read closest in rating or some hidden matchmaking rating (MMR). I think having books having both a personal and global score would be nice as well. I don't think it would be possible for someone to change their rating but that may be a feature. I also don't think this solves recommendation though it may be useful as a feature.
- MrVandemar 3y ago
  > I don't think it would be possible for someone to change their rating but that may be a feature. It should definitely be possible. Take films. I had a friend who would go see a film. On the day he saw it: "OMG THE BEST THING EVER". Next Week: "Yeah, that was a good film". A month later: "Yeah it was okay I guess". Six months later: "A bit ordinary really" In 1990 if you'd asked me to rank the Alien films, I would have put ALIENS first, and ALIEN second. These days, I reverse those rankings. People's tastes change over time. It would be inaccurate to "fix" a rating in stone.
  - helges 3y ago
    Yeah, I think what was meant is that the Elo rating system might make it difficult to account for changes in the rating. People change their minds all the time (and it's a good thing), I agree.
    - aleksiy123 3y ago
      Yes, this is what I meant. Changing the rating would require storing every rating and recalculating all dependent ratings which could be quite a large DAG. While not impossible I don't think it would be particularly scalable.
kmos17 3y ago
As much as the IMDB rankings are obviously gamed (especially when a movie or series just came out) I find the ratings fairly accurate over time, a 7+ is usually pretty decent to great for 8+, ratings of 6 are usually pretty average and 5 and below are truly awful.
- therealdrag0 3y ago
  Agree. Except for Indian movies. Their ratings aren’t trustworthy. Lots of 8+ movies they’re mediocre or bad (despite high production costs).
xhevahir 3y ago
This article about Microsoft's research on recommendation algorithms was interesting. It's from several years ago, so I imagine some people have begun using it. https://news.mit.edu/2017/better-recommendation-algorithm-1206 https://news.mit.edu/2017/better-recommendation-algorithm-12...
glitcher 3y ago
> Humans just aren't very good at ranking things on a normal distribution Anecdotally this feels correct, the more common user rating distributions I see are an inverted bell curve - lots of 5's and 1's, not too many 3's. But as far as the same problem that plagues every other review system goes, I would say the paid/fake reviews are the far bigger problem.
tim-fan 3y ago
I just give you two books that you've read and you tell me which is better. I played around with a tool for sorting through my personal photos based on that idea. It gets interesting when you start thinking of how to optimally choose pairs to compare, to maximise signal and minimize redundant comparisons. This becomes more important as the size of the set of objects you are comparing becomes large. https://github.com/tim-fan/image_sorting https://github.com/tim-fan/image_sorting
gmiller123456 3y ago
I don't find the average rating very useful. I do, however, find reading 1 and 2 star reviews quite helpful. It wouldn't be very helpful to read a review as to why someone thought Fundamental Calculus was better than Moby Dick. Without the star rating system, you lose the ability to filter on the reviews from people who think the product is bad.
merdaverse 3y ago
Agree completely. Scoring is mostly a cultural thing, I presume based on how the person experienced grades in school and their social pressures to attain certain grades. As a child I moved from a country where getting any grade below 9/10 was a shame to one where 7 was good enough, a 9 was very rare. Because of this all the review systems like IMDB are flawed by design. Steam reviews are the most trustworthy for me. While they are not perfect, the binary choice normalizes these scoring differences, and can be read as a "probability that you will enjoy X if your are interested in this sort of thing" which is the best one can hope for.
mattmanser 3y ago
First only IMDb TV shows hover around 8, films hover around 6. And it is still useful. For example, from my perspective I can quickly judge a movie from an imdb score. Anything less than 6.1 is bad. 6.1-6.3, ok, 6.3-6.8 good, 6.8+ excellent. Apart from movies starring star trek characters, add one to all those scores. Goodreads is similar but tighter, anything less than 3.9 is a red flag, over 4.2 and you start getting into excellent. Though in my opinion steam seems to have got it done best with "overwhelminly positive", "very positive", "positive", "mixed", "mostly negative", etc. but occasionally gets review bombed.
Zampa 3y ago
> I just give you two books that you've read and you tell me which is better. Do this loads of times and eventually you have a solid ranking of every book you've ever read. Aggregate everyone's rankings and you have a much more robust system then "please rate this book out of 5 stars". We've done exactly this with Flickchart - https://www.flickchart.com https://www.flickchart.com - presenting the user with two movies at a time and letting them choose what, in their opinion, is the better film. We then aggregate everyone's rankings to generate a combined chart of the best films of all-time that's always in flux and changing. We take into account each user's total number of rankings (refinement), and the total number of movies they've ranked (breadth of knowledge). We also use time as a factor - given that something that just came out has had less time to be "an all-timer" than something that has existed in the world longer and had the time to be experienced by more people. We created this way back in 2009 and are currently building the next-generation of it on a modern web stack. This new iteration is also more modularized so we can expand more easily from just movies to other domains (books, games, music, TV, etc.).
bart_spoon 3y ago
It seems like it would be easy to just normalize scores. Rather than a book’s score being the average rating they’ve received, it should be a z-score based on all ratings.
Gtex555 3y ago
This reminds me of my favourite scene from the social network when movie Zuck gets the idea for facemash: Billy Olson’s sitting here and had the idea of putting some of these next to pictures of farm animals and have people vote on who’s hotter. Yea, it’s on. I’m not gonna do the farm animals but I like the idea of comparing two people together. It gives the whole thing a very “Turing” feel since people’s ratings of the pictures will be more implicit than, say, choosing a number to represent each person’s hotness like they do on hotornot.com. The first thing we’re going to need is a lot of pictures.
laurels-marts 3y ago
> For IMDB the rankings all hover around 8ish. That's definitely not the case. 8 is neither mean nor median for titles on imdb. If a film scores anywhere above 8.0 they are already in the top 250. Most titles average between 5 and 7.
Gtex555 3y ago
Couldn't you do this implicitly if I rate book A an 8 and book B a 7 then I have technically said book A is better than book B, so ignore the numerical value and take the better reading. Now put the rating out of 100 as out of 5 or 10 would lead to too many ties.
moneywoes 3y ago
> aggregated personal Elo Interestingly that seems reminiscient of data apps. What are your thoughts on their ranking philosophy?
foysauce 3y ago
I don't think I'm alone in that if I'm reading something that is anywhere near a 1-star, I don't finish it. And then go on to not leave a review. This skews my ratings indeed, because not every book I start is between 3-5 stars, but almost every book I _finish_ is.

fourmajor 3y ago

I disagree that stars should be evenly distributed between 1 to 5 stars. I think it's quite possible that most books that people choose to read end up being a 3 (good with some flaws) or 4 (good but not all-time great). It's kind of like the same thing with pizza. I'd give most pizza a 3 or 4. Very few 1s and 5s to be sure. 1 doesn't have to mean bottom 20% of pizzas. It can mean "awful, couldn't finish," where very few pizzas would fall into that category. 5 doesn't need to mean "best 20% of pizzas," it can mean "telling strangers about it the next day," again where very few pizzas would fall into that category.

deleted 3y ago
[deleted]
LVB 3y ago
I had a debate on the Criticker site about this topic since they try hard to turn your raw ratings into a normalized span. The fact is that because of ratings sites, I'm very rarely watching 1-star movies. That's good! So the low end was dominated by a 2.5/3 type ratings, which per my scale meant "just OK," but they were mapping it to "bad," somehow inferring that my rating range is 3-5.
kortilla 3y ago
“Normal distribution” doesn’t mean evenly distributed. Most being in the middle is a normal distribution.

nextos 3y ago

There's also TrueSkill, by Microsoft Research, which turns Elo into a proper generative model: https://en.wikipedia.org/wiki/TrueSkill https://en.wikipedia.org/wiki/TrueSkill It'd be definitely interesting to see a book review site like this. In the meantime, for math books, I trust curated rankings by MAA: https://www.maa.org/press/maa-reviews https://www.maa.org/press/maa-reviews

pizza_pleb 3y ago
We use this at https://languageroadmap.com https://languageroadmap.com, but for difficulty rankings between titles in the context of language learning using media. It does seem to work pretty well and we disclaim rankings with the confidence score. As for the grandparent comment: recency bias, as pointed out by another commenter is a thing, as is the tediousness of doing a bunch of pairwise decisions. I think a happier medium is to have everyone fill in tier charts (with variable number of tiers) and build the pairwise rankings from that.

pessimizer 3y ago

> I just give you two books that you've read and you tell me which is better. I've always been so for this in situations where the users also have a database of their own history on the site, like Goodreads or Boardgamegeek.com, for example. Just throw up modals every once and a while comparing two books that you know the user knows, and make it easy to opt out of. I probably wouldn't opt out, because it would appeal to my ego and feel like a game. > Humans just aren't very good at ranking things on a normal distribution, so you invariably end up with every item (books in this case) being ranked somewhere in the 3.5-4.5 range (since Goodreads is out of 5). For IMDB the rankings all hover around 8ish. When in reality the average book should have a 2.5. If you don't rate like this then you just end up with garbage. Just normalize the users against themselves. I wouldn't be giving a 4-star review, I'd be giving a pessimizer(4-star review).

deleted 3y ago

[deleted]

dreamcompiler 3y ago

And in many such systems the peer pressure to give a 5 is tremendous, which further devalues the rating system. Uber drivers for example have told me that if more than a few customers give them a 4 they will be effectively fired. Restaurants, tour companies etc have asked me to please give them the maximum number of stars on Yelp, Google, Tripadvisor, etc lest there be dire consequences for them. This is of course Goodheart's Law in action but I don't have any ideas for fixing the problem.

tecleandor 3y ago

Seems like this is culturally different for some things. And corporations can influence too. I have a friend who has four restaurants in Tokyo, and I've been several times there. If you keep attention to the restaurant reviews in Google Maps, Japanese people is very hard. They'd go like "The food is great, incredible service, surprising flavors, very good experience, best Spanish food I've had in a long time..." and then they go "...but I know they can do better and have space to improve. 2/5". In Spain we would be like "Nice beer, they gave us some tapas. 5/5" Gig economy corporations also have made this worse. You're scared to score your driver, your server or your hotel by less than a 5/5 or somebody might get punished or fired (and maybe they didn't even had a contract in the first place). And this becomes a snake biting its own tail. Now I rarely go to a place with a score of less than a 4/5, and I'll score relatively high because I know I could influence votes out of what people consider "worthy to visit".

JohnFen 3y ago
> I rarely go to a place with a score of less than a 4/5 I have a simpler heuristic. I ignore all customer ratings entirely, and refuse to leave any.
- deleted 3y ago
  [deleted]
yurishimo 3y ago
100% it's cultural. In the Netherlands, companies brag about a 7 or 8 out of ten as proof of their amazing customer service. When I lived in America, it seemed the consensus of anything under a 9 is killing puppies. Coincidentally, NPS is entirely based on this fact. 8+ is required for a "happy customer". 7 is given a little wiggle room to win them back and under 6 is a lost cause.
- user_7832 3y ago
  > In the Netherlands, companies brag about a 7 or 8 out of ten as proof of their amazing customer service. Huh, could you give an example of it? I am new to the Netherlands and hadn’t realized this yet.

berkes 3y ago

I really like the idea of relative ratings. But that also amplifies a problem that exists with any rating: recency bias, or any sort of time bias. I change over time. An album that I loved when I was 15 could get a 1/5 from me today. Or a book that I hated in my teens, may now be my favorite. It's complicated.

MrVandemar 3y ago
In which case the system would need to continually challenge you about your historical choices, perhaps based on recent choices and detecting possible paradoxes (you liked A more than B, and you liked B more than C, but you also liked C more than A).
- berkes 3y ago
  Very good solution indeed! Maybe also allow ratings to be a list over time rather than one, single data point. That'll make calculating an average more involved, but also more realistic.

thebigspacefuck 3y ago

IIRC they initially gave you a text for each star like 1 - I hated it 2 - It was okay 3 - I liked it 4 - It was great 5 - I loved it I thought it was helpful for framing what a score would be but I haven’t seen this in the app or anywhere for a long time. At least it avoids the problem of Amazon where a review is like “Best book I’ve ever read but the copy I received had a rip on the dust jacket - 1/5”. Learned through experience to skip anything below a 4 or very high 3 now. Probably missing some interesting stuff but I don’t read that often anyway.

waspight 3y ago

I have thought about this relative rating as well. Would it be possible to use? I mean, does it scale? I also have seen some ratings where you can only rate between 5 - 10, I guess since no one ever rates below 5 anyway. I think it kind of works.

throw0101a 3y ago

For a while now I've though that maybe only three rankings are needed: * Recommend this thing * Neutral opinion * Do not recommend this thing Not sure how much "neutral" would be used: kind of like thumbs-up/down or up-down-vote mechanic: if you're neutral you may simply not vote at all.

hospitalJail 3y ago

The Modern Zelda reviews are a prime example of this. The non-industry people reviewing Zelda have 4 options, 7/10, 8/10, 9/10, 10/10. A 7/10 means the game was bad and you had a hard time finishing it, and won't be playing again. 8/10 means the game was also bad, but you had fun for a few minutes/hours. 9/10 means the game meet minimum expectations. 10/10 means you enjoyed the game, but there were countless flaws that took away from enjoying the game. 10/10 Greatest game of all time means, the game was above average. Now, this same system applied outside Nintendo's curve, you knock these numbers down 2. Its incredibly hard to figure out if a Nintendo game is good, if you treat them like other companies. Even Nintendo believes this and will yank early access/ads/etc... from gaming websites if you don't comply with the Nintendo curve.

yurishimo 3y ago
Devils advocate; Is it not possible that Nintendo just makes good games? They spend a lot of time on development, don't rush things, the trailers are honest about gameplay and what to expect. I can understand it's not your cup of tea or arguments that their online service is shit (it is) but reviewing a video game in isolation, many of them seem to be good games objectively. Games that don't appeal to the average CoD fan sure, but that doesn't mean they are bad games.
- hospitalJail 3y ago
  Nah, you can see it on subreddits as people discuss the game. Complaining about a few enemies, copypaste world, etc... Its not a 10/10, but it will still be reviewed like this. Imagine you got rid of the Zelda skins and Zelda name, released it on Xbox. What would it get then? Don't get me wrong, I religiously play all the zelda's and find ways to enjoy them. I am under Nintendo's spell, but I also know these are the corporate mascots I grew up with. Nintendo markets to children, we basically need therapy if we want to be free from their grip.
  - yurishimo 3y ago
    That’s fair. I would take a minor issue with your framing (someone who didn’t play BotW wouldn’t find the map a cop out and I’m glad they put the effort in for the 2 other layers of the map) but you could make those same arguments against COD or Destiny. I do think it’s worth noting how many of the ideas from BotW have been copied in the past 6 years in other games. I guess my point is, I can understand the 10/10s but I also find it entirely reasonable to view it closer to a 7 or 8. We’re also not thinking about things in a vacuum in this discussion. Is the Xbox game released as if BotW never happened? Nintendo certainly uses its brand to buy marketing influence but importantly, I don’t think the company is pissed when games fail. Skyward Sword was a good example of that actually. It’s failure literally led to BotW. Nintendo got off their laurels and reinvented their “tried and true” formula and now it’s paying off. I am closer your position for games like 2D Mario. Zero innovation and I’ll be eagerly watching the reviews for the new game later this year.

deleted 3y ago

[deleted]

deleted 3y ago

[deleted]

TRiG_Ireland 3y ago

> I just give you two books that you've read and you tell me which is better. I can't wait to compare The Pickwick Papers against The Unfolding of Language.

gavmor 3y ago
A solid point: books may excel at one dimension while not even attempting another. I suppose one might have to ascertain what genre, exactly, a book falls in, and how it fares in its own genre. But then you're not comparing pairwise, and we lose the benefits of our visual cortex or whatever neural system is so good at pairwise comparisons. Constraining pairings within their genres might suffice, if possible. There are several standard ontology for books, and it's not a trivial problem.
MrVandemar 3y ago
I don't think it makes much sense to compare a fiction book to a non-fiction book. There's not enough points of comparison. While both can be compared on quality of prose, the fiction might be judged on characters and plot, and the non-fiction book on accuracy and detail. You might as well ask the question: Who is the better Tennis player? Roger Federer or Garry Kasparov? A system would need to draw some kind of boundary conditions so you are comparing things that have more in common than "printed on paper and bound".

cratermoon 3y ago

> Humans There's the problem there – assuming human input makes up an important part of the rankings.

sedatk 3y ago

Yes, one person I knew rated my book 2/5 because, as he said, it was beginner/mid-level-oriented and he wasn't aware when he purchased it (clearly stated on the book's page, and pretty much everywhere). A star rating system gives more emphasis on an individual's own preference of what a good rating should be which works against the consumer.

jsmith99 3y ago

It's not just because humans are bad at distributions. I would rate almost all the books I read at least 3/5 because I pick my books carefully. There are plenty of terrible books deserving 1/5 but they are often obvious from a distance. With eg movies a much greater investment is required to tell if it's terrible.

ROTMetro 3y ago

In medicine they use a pain scale with faces when they need to find out from kids how much pain they are in. Something like that might be a better system?

bstpierre 3y ago

See https://www.criticker.com/explain/ https://www.criticker.com/explain/ -- it works sort of like what you're suggesting. You rank movies from 0-100, which is different from rating them. Your percentile ranking scores are compared to other users' rankings, and then it can suggest movies that it thinks you will rank highly. IMO the real problem with something as subjective as books or movies is that even completely honest, well-reasoned reviews are going to be all over the map. My review of a Pride and Prejudice movie is going to be maybe 3/10, but my wife would give it 7/10, while we have the opposite reactions to something like The Hunt for Red October. I don't care about reviews from experts or the unwashed masses. I don't even really care about reviews that much -- I'm more interesting in ratings from people who like the same kind of stuff I do.

mahoho 3y ago

Well rating scales don't have any absolute meaning, only that which is attributed to them either by the reviewer or the audience. On any aggregate review site the scale of the aggregate score will take on its own semantics, which is the aggregate of all the users' own semantics they use for their individual scores. And for most people, it seems like the semantics of rating scales don't align with a normal distribution about the 50% mark, but something more like letter grades where 70% is decent/mediocre, while 50% is a failure (in the US). IMO this makes at least as much sense as centering around 50%; if you only score 50% of the points on a test I don't think you're competent in a subject, and if your book only meets 50% of the criteria for a great book then it might be a bad book. But again, it's all arbitrary and only makes sense in the context of the particular reviewer or aggregate community.

ohlookcake 3y ago

> the average book should have a 2.5 Disagree on this. This assumes that people read books completely at random. In reality, people read blurbs/summaries or get a recommendation and the typical book you read is likely to be better than the midpoint between the worst book you've read and the best book you've read (I'm assuming a linear scale mental model).

CSMastermind 3y ago
> This assumes that people read books completely at random. I'm not sure it implies that. I think that if: 1. Book quality follows a normal distribution. and 2. People are capable of accurately assessing the quality of books relative to other books. then you'd end up with an average rating of 2.5. It doesn't matter if people are disproportionately reading the better books and rating them highly because an increasing number of reviews on a book would only give you more confidence in its rating, not change the rating of the book relative to other books. To illustrate this we can imagine a world with only 100 books in it and only 100 people in it. Let's assume that a single person from that group reads all 100 books and per the two assumptions above correctly rates them on a normal distribution. Then the other 99 people read only a single book which the first person recommends as a 5. And per the two assumptions above they all also rate it as a 5. Now there are 99 books with only 1 rating normally distributed between 1 and 5 and there is 1 book with 100 ratings - all 5s. Does this change the average rating of books on the platform? Nope the average rating is still 2.5. In fact the one book could have a million ratings or a billion 5 star ratings and the average rating of books on the platform would still be a 2.5. What we see on sites like goodreads is that either book quality doesn't follow a normal distribution and/or people are not capable of accurately assessing the quality of books relative to other books.
- therealdrag0 3y ago
  Good points. But do we have evidence GR is not a normal distribution. GP’s comment does apply to views of ratings, so we might have an impression if higher average rating because the books we look at are more often better than avg.
- 3y ago

aleksiy123 3y ago

Elo/Trueskill sounds like a solid idea. When you go to rate a book the system asks you if it was better or worse (maybe same) then a book of you previously read closest in rating or some hidden matchmaking rating (MMR). I think having books having both a personal and global score would be nice as well. I don't think it would be possible for someone to change their rating but that may be a feature. I also don't think this solves recommendation though it may be useful as a feature.

MrVandemar 3y ago
> I don't think it would be possible for someone to change their rating but that may be a feature. It should definitely be possible. Take films. I had a friend who would go see a film. On the day he saw it: "OMG THE BEST THING EVER". Next Week: "Yeah, that was a good film". A month later: "Yeah it was okay I guess". Six months later: "A bit ordinary really" In 1990 if you'd asked me to rank the Alien films, I would have put ALIENS first, and ALIEN second. These days, I reverse those rankings. People's tastes change over time. It would be inaccurate to "fix" a rating in stone.
- helges 3y ago
  Yeah, I think what was meant is that the Elo rating system might make it difficult to account for changes in the rating. People change their minds all the time (and it's a good thing), I agree.
  - aleksiy123 3y ago
    Yes, this is what I meant. Changing the rating would require storing every rating and recalculating all dependent ratings which could be quite a large DAG. While not impossible I don't think it would be particularly scalable.

kmos17 3y ago

As much as the IMDB rankings are obviously gamed (especially when a movie or series just came out) I find the ratings fairly accurate over time, a 7+ is usually pretty decent to great for 8+, ratings of 6 are usually pretty average and 5 and below are truly awful.

therealdrag0 3y ago
Agree. Except for Indian movies. Their ratings aren’t trustworthy. Lots of 8+ movies they’re mediocre or bad (despite high production costs).

xhevahir 3y ago

This article about Microsoft's research on recommendation algorithms was interesting. It's from several years ago, so I imagine some people have begun using it. https://news.mit.edu/2017/better-recommendation-algorithm-1206 https://news.mit.edu/2017/better-recommendation-algorithm-12...

glitcher 3y ago

> Humans just aren't very good at ranking things on a normal distribution Anecdotally this feels correct, the more common user rating distributions I see are an inverted bell curve - lots of 5's and 1's, not too many 3's. But as far as the same problem that plagues every other review system goes, I would say the paid/fake reviews are the far bigger problem.

tim-fan 3y ago

I just give you two books that you've read and you tell me which is better. I played around with a tool for sorting through my personal photos based on that idea. It gets interesting when you start thinking of how to optimally choose pairs to compare, to maximise signal and minimize redundant comparisons. This becomes more important as the size of the set of objects you are comparing becomes large. https://github.com/tim-fan/image_sorting https://github.com/tim-fan/image_sorting

gmiller123456 3y ago

I don't find the average rating very useful. I do, however, find reading 1 and 2 star reviews quite helpful. It wouldn't be very helpful to read a review as to why someone thought Fundamental Calculus was better than Moby Dick. Without the star rating system, you lose the ability to filter on the reviews from people who think the product is bad.

merdaverse 3y ago

Agree completely. Scoring is mostly a cultural thing, I presume based on how the person experienced grades in school and their social pressures to attain certain grades. As a child I moved from a country where getting any grade below 9/10 was a shame to one where 7 was good enough, a 9 was very rare. Because of this all the review systems like IMDB are flawed by design. Steam reviews are the most trustworthy for me. While they are not perfect, the binary choice normalizes these scoring differences, and can be read as a "probability that you will enjoy X if your are interested in this sort of thing" which is the best one can hope for.

mattmanser 3y ago

First only IMDb TV shows hover around 8, films hover around 6. And it is still useful. For example, from my perspective I can quickly judge a movie from an imdb score. Anything less than 6.1 is bad. 6.1-6.3, ok, 6.3-6.8 good, 6.8+ excellent. Apart from movies starring star trek characters, add one to all those scores. Goodreads is similar but tighter, anything less than 3.9 is a red flag, over 4.2 and you start getting into excellent. Though in my opinion steam seems to have got it done best with "overwhelminly positive", "very positive", "positive", "mixed", "mostly negative", etc. but occasionally gets review bombed.

Zampa 3y ago

> I just give you two books that you've read and you tell me which is better. Do this loads of times and eventually you have a solid ranking of every book you've ever read. Aggregate everyone's rankings and you have a much more robust system then "please rate this book out of 5 stars". We've done exactly this with Flickchart - https://www.flickchart.com https://www.flickchart.com - presenting the user with two movies at a time and letting them choose what, in their opinion, is the better film. We then aggregate everyone's rankings to generate a combined chart of the best films of all-time that's always in flux and changing. We take into account each user's total number of rankings (refinement), and the total number of movies they've ranked (breadth of knowledge). We also use time as a factor - given that something that just came out has had less time to be "an all-timer" than something that has existed in the world longer and had the time to be experienced by more people. We created this way back in 2009 and are currently building the next-generation of it on a modern web stack. This new iteration is also more modularized so we can expand more easily from just movies to other domains (books, games, music, TV, etc.).

bart_spoon 3y ago

It seems like it would be easy to just normalize scores. Rather than a book’s score being the average rating they’ve received, it should be a z-score based on all ratings.

Gtex555 3y ago

This reminds me of my favourite scene from the social network when movie Zuck gets the idea for facemash: Billy Olson’s sitting here and had the idea of putting some of these next to pictures of farm animals and have people vote on who’s hotter. Yea, it’s on. I’m not gonna do the farm animals but I like the idea of comparing two people together. It gives the whole thing a very “Turing” feel since people’s ratings of the pictures will be more implicit than, say, choosing a number to represent each person’s hotness like they do on hotornot.com. The first thing we’re going to need is a lot of pictures.

laurels-marts 3y ago

> For IMDB the rankings all hover around 8ish. That's definitely not the case. 8 is neither mean nor median for titles on imdb. If a film scores anywhere above 8.0 they are already in the top 250. Most titles average between 5 and 7.

Gtex555 3y ago

Couldn't you do this implicitly if I rate book A an 8 and book B a 7 then I have technically said book A is better than book B, so ignore the numerical value and take the better reading. Now put the rating out of 100 as out of 5 or 10 would lead to too many ties.

moneywoes 3y ago

> aggregated personal Elo Interestingly that seems reminiscient of data apps. What are your thoughts on their ranking philosophy?

foysauce 3y ago

I don't think I'm alone in that if I'm reading something that is anywhere near a 1-star, I don't finish it. And then go on to not leave a review. This skews my ratings indeed, because not every book I start is between 3-5 stars, but almost every book I _finish_ is.