4 ms·

Personally, I've consistently found that the best predictor of whether or not I was going to enjoy a movie was the NUMBER of ratings, not the rating itself. Thi

Personally, I've consistently found that the best predictor of whether or not I was going to enjoy a movie was the NUMBER of ratings, not the rating itself. This also works for restaurants and other things on sites like Yelp. It almost seems that a movie should come with a simple "recommend!" button that simply counts recommendations. But ratings are a tricky issue and I think they require a more sophisticated mathematical treatment and modeling if one wants to get it right, not just a few histograms that treat all people equal. There are a few modeling challenges that come to mind: For example, people disagree on quality of movies based on their taste. This could be modeled as a latent variable that must be inferred for every person in some graphical model. Another example of a relevant variable would be person's rating habits: some people rate movies 5 or 1, some people have a gaussian rating centered at some value. These should be explicitly modeled and normalized. Every rating could ideally be used to make a stochastic gradient update to the weights of the network, and since we are dealing with very sparse data, strong priors and Bayesian treatment seems appropriate. Ratings could then be personalized through an inference process on the graph. Has anyone heard of a more sophisticated model like this, or any efforts in this direction? I'd like to see more math, modeling and machine learning and less silly counting methods.

dredmorbius 14y ago
That metric starts to fail when you've got, say, mediocre products or brands with tons of exposure. The classic is in "readers choice" reviews of restaurants or eateries. Fast-food franchises dominate? Why? Because the philosophy of such sites is often "majority rules", and the establishment (or brand) with the most votes wins. But there are far more McDonalds or Taco Bells than Jacks Cook Shacks or Trader Vics. Even when the quality of JCS or TV exceeds TB or MD, it's not going to be reflected in the ratings. Adjustments such as taking a Likert (3-7 point scale) and adjusting reviews based on the number of reviewers, to give both the actual qualitative assessment, and the probable maximal review can help. This is how sites such as Reddit have adjusted their comments/submissions ratings. The broader and more philosophical problem is that "quality" is not a one-dimensional attribute, interpretation of quality differs among individuals, and "fitness for purpose or task" should be considered when assessing quality as well. McDonalds may very well be appropriate when your goal is a quick, inexpensive meal on the run (a conclusion I'd differ with), while Trader Vics is where you'd head to impress the boss, date, in-laws, or client. It's a tough problem. It's also one that sees a great many very poor proposed solutions.

dredmorbius 14y ago

That metric starts to fail when you've got, say, mediocre products or brands with tons of exposure. The classic is in "readers choice" reviews of restaurants or eateries. Fast-food franchises dominate? Why? Because the philosophy of such sites is often "majority rules", and the establishment (or brand) with the most votes wins. But there are far more McDonalds or Taco Bells than Jacks Cook Shacks or Trader Vics. Even when the quality of JCS or TV exceeds TB or MD, it's not going to be reflected in the ratings. Adjustments such as taking a Likert (3-7 point scale) and adjusting reviews based on the number of reviewers, to give both the actual qualitative assessment, and the probable maximal review can help. This is how sites such as Reddit have adjusted their comments/submissions ratings. The broader and more philosophical problem is that "quality" is not a one-dimensional attribute, interpretation of quality differs among individuals, and "fitness for purpose or task" should be considered when assessing quality as well. McDonalds may very well be appropriate when your goal is a quick, inexpensive meal on the run (a conclusion I'd differ with), while Trader Vics is where you'd head to impress the boss, date, in-laws, or client. It's a tough problem. It's also one that sees a great many very poor proposed solutions.