3 ms·

Question to the Machine Learning folk: are five-star ratings "better" than thumbs up/down, or is it just a matter of algorithm design? I know ratings/predictio

Question to the Machine Learning folk: are five-star ratings "better" than thumbs up/down, or is it just a matter of algorithm design? I know ratings/prediction has long been studied by the MovieLens.org scientists.

dredmorbius 10y ago
The technical term for a multi-perference rating is a Likert Scale. There are different schools on this -- an even or odd number of options (even "forces" reviewers to indicate some preference positive or negative, odd allows fence-sitting), how to compare different raters' preferences, etc. I'm finding interesting the rating system I've created and am trying to apply with some consistency to my Pocket archive of a few thousand items. Nominally it runs from 0 to 5, though I may reserve a 6 for an absolutely mind-blowing piece. A 0 is a net negative: you are less informed for having read it, it reduces teh intelligence of its reader. A 1 is, generally, a simple noting of some event. A 2 should be a general news story, without strong insight. A 3 is a news or general interest story with strong insight, or a typical scientific paper, or an undistinguished book (generally nonfiction). A 4 is a particularly good scientific paper, or a typically well-thought-out book. A 5 is a document which establishes a fundamental idea or field. Claude Shannon's original paper on information theory, say. I don't think I've run across a 6 yet, but that might be a work which ties together two or more previously unrelated fields into a common theory. My problem has been in assigning far too many '3' class articles. I've already carved out a list to re-assess and downgrade if appropriate. I'd also like to be able to report on the numbers for each classification, though Pocket's utter lack of quantitative reporting (I cannot even state how many articles I've collected in total) stymies this. I am ... increasingly dissastisfied with Pocket as an information management tool.
refrigerator 10y ago
I think the fact that Netflix saw a 200% increase in "thumbs" ratings vs "stars" in the A/B test makes it much better from a machine learning perspective, even if it might at first seem like the data is of "lower quality". One of the biggest problems for recommender systems is that the data is extremely sparse - most users will have only rated a tiny proportion of films, and most films will only have been rated by a tiny proportion of users. This is just my opinion now, but having studied recommender systems in a decent amount of depth, I don't think the design of the algorithm will need to change. The current techniques that provide the best results use matrix factorisation to simultaneously learn characteristics of films and how much each user likes each characteristic. My intuition would be that the algorithm can learn a lot more about a user from 3 up/down-ratings than from 1 star-rating, and the only reason Netflix are doing this is so that they can provide better recommendations so it's almost certainly the case. TL;DR: All else being equal, 5-star ratings carry more signal for any machine learning algorithm, but the fact that thumbs up/down will result in 200% more data is a lot more significant than the delta in the signal.

dredmorbius 10y ago

The technical term for a multi-perference rating is a Likert Scale. There are different schools on this -- an even or odd number of options (even "forces" reviewers to indicate some preference positive or negative, odd allows fence-sitting), how to compare different raters' preferences, etc. I'm finding interesting the rating system I've created and am trying to apply with some consistency to my Pocket archive of a few thousand items. Nominally it runs from 0 to 5, though I may reserve a 6 for an absolutely mind-blowing piece. A 0 is a net negative: you are less informed for having read it, it reduces teh intelligence of its reader. A 1 is, generally, a simple noting of some event. A 2 should be a general news story, without strong insight. A 3 is a news or general interest story with strong insight, or a typical scientific paper, or an undistinguished book (generally nonfiction). A 4 is a particularly good scientific paper, or a typically well-thought-out book. A 5 is a document which establishes a fundamental idea or field. Claude Shannon's original paper on information theory, say. I don't think I've run across a 6 yet, but that might be a work which ties together two or more previously unrelated fields into a common theory. My problem has been in assigning far too many '3' class articles. I've already carved out a list to re-assess and downgrade if appropriate. I'd also like to be able to report on the numbers for each classification, though Pocket's utter lack of quantitative reporting (I cannot even state how many articles I've collected in total) stymies this. I am ... increasingly dissastisfied with Pocket as an information management tool.

refrigerator 10y ago

I think the fact that Netflix saw a 200% increase in "thumbs" ratings vs "stars" in the A/B test makes it much better from a machine learning perspective, even if it might at first seem like the data is of "lower quality". One of the biggest problems for recommender systems is that the data is extremely sparse - most users will have only rated a tiny proportion of films, and most films will only have been rated by a tiny proportion of users. This is just my opinion now, but having studied recommender systems in a decent amount of depth, I don't think the design of the algorithm will need to change. The current techniques that provide the best results use matrix factorisation to simultaneously learn characteristics of films and how much each user likes each characteristic. My intuition would be that the algorithm can learn a lot more about a user from 3 up/down-ratings than from 1 star-rating, and the only reason Netflix are doing this is so that they can provide better recommendations so it's almost certainly the case. TL;DR: All else being equal, 5-star ratings carry more signal for any machine learning algorithm, but the fact that thumbs up/down will result in 200% more data is a lot more significant than the delta in the signal.