3 ms·
The problem isn't that it's impossible for humans to understand, nor that one can't hire the best in the field to implement this solution. The problem is that t
by biot 15y ago
The problem isn't that it's impossible for humans to understand, nor that one can't hire the best in the field to implement this solution. The problem is that the people who need to understand the outcome won't be able to understand it.
Case in point, the article mentions Amazon. Amazon could easily hire the best and brightest mathematicians and developers to make the implementation completely rigorous and provably correct. However, they're still going to field questions from merchants saying "My product has a 5-star average, but this competitor's product only has a 4.5-star average. Why the hell is my product shown further down in the list?"
Similarly, you may work in the acquisitions department of BigCorp and you need to purchase some equipment. When you prepare your report for Big Boss of your recommendations, how do you explain that you are recommending something which has a 4.5-star average rating instead of the 5-star average rated one?
The ratings may be 100% mathematically sound, but the users of the rating system are not the same as the producers of the rating system. So it's largely irrelevant if the producers made it perfect if the users don't understand what "perfect" means.
- gcb 15y agothe article also misses the human factor. And intent. for amazon, is it better to show the best product first or to help sell a few of the lacking-rating ones to generate more ratings and work out the uncertainty with real world data instead of crazy math?
- vog 15y agoI don't think it is a good idea to misuse your customers for that kind of testing.
- gliese1337 15y agoThey'll only have to field those questions if they display the plain averages despite not sorting on them. If we've established that average rating is the wrong metric, why would you even show it to the user/merchant/whoever? Display the same metric that you sort by, and then the merchants will be perfectly happy with higher-rated items showing up higher on the list, and Big Boss will be happy that you've chosen the #1-recommended option.
- esrauch 15y agoI really don't think that solves the problem; you look at product X, it has 0 reviews, you give it a 5 star, now it shows 1 rating and 3.7 stars?
- redguava 15y agoSo you let people rate it with 5 stars, but then show it as 3.7 magic points. Magic points can be your proprietary ranking system. You don't need to let them rate with stars and also show it as stars.
- gliese1337 15y agoIf I saw that happen, I would probably assume that the item was so popular that a bunch of other people had rated it concurrently. Unless the number of ratings were displayed simultaneously. There's a bit of a comprehension barrier in letting people know that the ordering score is not the same as the thing they're inputting, and what the relationship between them is. I don't know how to solve that problem (if I had to, I'd probably just not show the ranking numbers at all- the user already sees the sorting order, what do they need the raw numbers for?), but that's OK, 'cause I prefer to sidestep it- just don't use a graded rating scale like "5 stars". At least not unless you have a really good justification and explanation of what precisely the scale means. 5-star rating systems tend to result in large numbers of 5s and large numbers of 1s anyway, so you might as well just go with a binary like/dislike system. The fact that a ranking is a real number and your input is an "up"/"down" enumeration makes it clear that the ranking you see is not the same thing as the data that you put in. It could also be useful from a defusing confusion standpoint to make the ranking more complicated by mixing in an age factor (like HN does) or some other extra band of information. Then if a user doesn't immediately understand the relation between their rating and the final ranking, they can easily write it off as being due to some other factor that's not in their control anyway.
- aercolino 15y agoYou can do all stars but if you say you need at least X people to rate before showing any. That's what many sites do.
- lwat 15y ago> "My product has a 5-star average, but this competitor's product only has a 4.5-star average. Why the hell is my product shown further down in the list?" 'Because you have fewer votes than your competitor' I don't think it's that hard to understand.
- msg 15y agoRight. If you look at most-helpful ratios for product reviews on Amazon, you will see this. 500/600 helpful floats far above 10/10 helpful.
- georgebarnett 15y agoThe problem isn't that it's impossible for humans to understand, nor that one can't hire the best in the field to implement this solution. The problem is that the people who need to understand the outcome won't be able to understand it. I disagree that people won't be able to understand it - I propose they can't be bothered to understand it. Regular people do not give a shit about algorithms - either the computer/software works or it doesn't. Tech geeks regularly forget this simple fact. Most people have other things they care about and all they want is to press a button and have an answer come out. If its instantly clear how the answer is generated, they think about it for a very short while, agree and move on with their lives. If it's not instantly clear how the answer is generated, they assume the computer is right and move on with their lives.