3 ms·
While a simple scale rating system is just that, simple, anything more complex quickly exceeds the amount of effort both rater and searchers are willing to put
by thwarted 18y ago
While a simple scale rating system is just that, simple, anything more complex quickly exceeds the amount of effort both rater and searchers are willing to put in to understanding it. "Rate this restaurant on a scale of 1-10" is much simpler to understand than a massive questionnaire (even ratings on something like 3 or 4 axes may be too many/too much effort). It also has the advantage of being ambiguous enough that ANY opinion can fit into it -- you would be hard pressed to find the right kind of questions that fit all the different things that can be rated (What if you liked a place but didn't experience any of the specific things being asked about? Would you rate a movie theater on service if you never used the concession stand?). A site that didn't allow a reviewer to put in prose to explain their rating would be useless; I believe all the sites the OP talked about do. The rating scale is just meant to be a summary, each one is a summary of the thing being reviewed, and in aggregate it's a summary of the community's feel for the thing being reviewed.
When flipping channels looking at movies that are on, the star rating system is largely worthless because 1) it was most likely done by some paid movie buff who has inherently different motivations and likes/dislikes than I do and 2) there is no explanation as to the reasoning why a rating was given. It's all subjective measurements, and it can't be objective. The whole point of rating systems is opinion. It's just like asking a friend if a place is worth going.
Often times, the use case is "find something good with a minimal amount of hassle". This comes down to things like "places people liked that are within a mile of where I am right now". I have five minutes to make a decision on this, the single data point of a rating scale helps me make the decision quickly and (perceptually) accurately (it may not actually be accurate, but I feel like I'm making a good decision). These sites are also designed for repeat users and users being contributors. You learn the way other people have rated things on the site over time and are more able to decide for yourself what a "1 star" means vs a "5 star" means given the context.
I think a larger problem is getting people to want to expend the effort to provide a review of a place that they aren't extremely excited about (either positively or negatively). I know I don't bother to review places I had a so-so experience at, but if I had excellent service or really bad service, I make it a point to rate them. In some way, this skews the results, but is most likely isn't that big of a deal.
I don't know what the OP has in store for his next posting, but if it's truly revolutionary, he should be starting his own competing review service.
- pmjoyce 18y agoGood analysis, and it real get to the nub of the article: the difference between the perception of making a good decision based on the available data and how it's presented versus the reality of the decision making process. What I'm proposing isn't new or revolutionary - it's simply that we need a more honest appraisal system - something that everyone can understand not just think they understand because it's presented in a familiar and reassuring format (5 stars).
- thwarted 18y agoWhat is dishonest about the current systems? And what makes it look like an opinion system could be honest? Does it seem like people are purposely misrepresenting their position in these kinds of rating systems? Everyone does understand the simple rating system, which is why it works and is popular. I think there is a perceptual accuracy, and not necessarily an absolute accuracy, to the way people interpret the data because they are looking for absolutes in a system which can not inherently contain absolutes, as it is based on opinions. Discussion of the Netflix Prize is a good place to mine data on the accuracy of rating systems and improve them.
- pmjoyce 18y agoWhat is dishonest about the current systems? The dishonesty (wilful or otherwise) arises from allowing the 5 star system to proxy for something that gives users something meaningful with which to base their decision making. And what makes it look like an opinion system could be honest? I said an opinion system could be made more honest. To be clear, I'm not suggesting that the people voting are misrepresenting their position (the users are dishonest) merely that the 5 star rating system that purports to tell us something of value, doesn't (the system is dishonest). Honesty comes from simplicity and it's my opinion that aligning online recommendations systems more closely with "real world" models is not only purer (in that we'll have a greater innate understanding) but probably more accurate and more helpful as a decision making aid. As for the Netflix Prize, I can't help feeling that it's not much more than asking how many angels can dance on the head of a pin?