4 ms·

You can do much better than just sorting. The simplest is to use Bradley-Terry. It's a very simple algorithm and will let you combine results from multiple user

by timhh 3y ago

You can do much better than just sorting. The simplest is to use Bradley-Terry. It's a very simple algorithm and will let you combine results from multiple users and gives an actual rating rather than just a ranking. It also handles the probabilistic nature of sorting better. Traditional sorting algorithms rely on comparisons being sensible (a>b and b>c implies a>b) but you probably won't get that if you use people. I explained it here: https://stats.stackexchange.com/a/131270/60526 https://stats.stackexchange.com/a/131270/60526 Quite closely related to matchmaking in computer games. I remember there was a website a while ago that used pairwise comparison to rank programming languages and I think whiskey. Does anyone remember this? I could never find it again.

pocketarc 3y ago
Wow, this is extremely helpful, I had no idea this existed and will have to read up on it properly. I think my main concern would be: What would it be like for the first user to try to rank a show (as was the case for everyone today)? All probabilities would be 50-50, no? But if it's a show that's already been ranked at least once, then this could help immensely, if I understand correctly.
- timhh 3y ago
  Due to the regularisation yeah they all start at the same rating. But you don't need many votes to start getting good ratings. I introduced this method to Dyson for objectively calculating very subjective measurements (e.g. "how frizzy does this hair look?"). We basically crowd sourced it to other engineers. I did a load of studies on different methods by ranking something that's sort of hard to rank but you know the answer to - I used 10 grey squares that only differed by 2/255 and you had to pick the brighter one. Some other things: 1. I don't remember the exact details but there's a slight extension of the method where you give each user a "how good are you" coefficient that you simultaneously solve for. This helps eliminate people that vote randomly, and also inverts the votes of people that deliberately pick the wrong answer (as long as they're consistently wrong). 2. You can put confidence limits on the values very easily too since it's a MAP estimate. Actually I showed curves for each item - basically how does the model probability vary as you sweep one rating up and down a bit. People didn't understand it at all though. 3. You can calculate the rankings incrementally very quickly (details in the answer) which means you can show users comparisons that give the most information. This usually means you end up showing users endless difficult choices which can frustrate them, especially if it's a forced choice. 4. I never found a principled way to incorporate a "they look the same" option. I tried some ad-hoc methods and IIRC a "much better, slightly better, can't tell, slightly worse, much worse" scale gave the fastest convergence but it was pretty unsatisfying that I just used some as hoc method to add the results. It was all closed source and I haven't worked there for years so the code is lost to the wind unfortunately.
  - pocketarc 3y ago
    This is honestly very interesting, thank you so much for elaborating! To be fair, after today, there are now nearly 400 TV shows with votes, so I can start seriously looking into this very soon!
michaelrpeskin 3y ago
I did something similar, in fact, the math may be the same thing and just expressed differently. But when I've had to rank non-transitive things, I use Elo (https://en.wikipedia.org/wiki/Elo_rating_system https://en.wikipedia.org/wiki/Elo_rating_system) Many years ago when I was a mid-level developer at a dysfunctional company, I was senior enough to be invited to some "strategy" meetings, but junior enough that no one ever listened to me. We (engineering, sales, marketing, etc.) spent nearly an entire summer bickering over what "important" features we were going to schedule next. I finally got fed up, took everything out of the ticketing system and made random parings and had people vote on it. Then just like a chess match, updated their Elo score based on the outcome. Then I had anyone who cared play match-ups for as long as they wanted. We ended up getting a decent ordering of features and finally ended the summer of hell meetings. I don't know if the order was the correct order, I didn't stay around long enough to see. I was just happy that sales and marketing folks thought that I had some magic math that solved their problem, and I was happy to be back developing and not sitting in useless meetings. What I like about this is that you don't have to be self consistent, as long as on average you pick the best, it will bubble to the top. And you can mix the results of other voters and see what the "true" winner is. (Of course, to be fair, you have to give each person the same number of match ups, in my case, I just served match-ups to anyone who wanted to sit at the terminal and vote, so someone could have wasted an entire day and overwhelm the system - I didn't care at the time).
zeroonetwothree 3y ago
This is so cool. I always wanted a way to do this
manx 3y ago
Interesting! We could use this algorithm to rank websites for every term in search engines. Just need a good UI to collect the data.

by timhh 3y ago

pocketarc 3y ago

Wow, this is extremely helpful, I had no idea this existed and will have to read up on it properly. I think my main concern would be: What would it be like for the first user to try to rank a show (as was the case for everyone today)? All probabilities would be 50-50, no? But if it's a show that's already been ranked at least once, then this could help immensely, if I understand correctly.

timhh 3y ago
Due to the regularisation yeah they all start at the same rating. But you don't need many votes to start getting good ratings. I introduced this method to Dyson for objectively calculating very subjective measurements (e.g. "how frizzy does this hair look?"). We basically crowd sourced it to other engineers. I did a load of studies on different methods by ranking something that's sort of hard to rank but you know the answer to - I used 10 grey squares that only differed by 2/255 and you had to pick the brighter one. Some other things: 1. I don't remember the exact details but there's a slight extension of the method where you give each user a "how good are you" coefficient that you simultaneously solve for. This helps eliminate people that vote randomly, and also inverts the votes of people that deliberately pick the wrong answer (as long as they're consistently wrong). 2. You can put confidence limits on the values very easily too since it's a MAP estimate. Actually I showed curves for each item - basically how does the model probability vary as you sweep one rating up and down a bit. People didn't understand it at all though. 3. You can calculate the rankings incrementally very quickly (details in the answer) which means you can show users comparisons that give the most information. This usually means you end up showing users endless difficult choices which can frustrate them, especially if it's a forced choice. 4. I never found a principled way to incorporate a "they look the same" option. I tried some ad-hoc methods and IIRC a "much better, slightly better, can't tell, slightly worse, much worse" scale gave the fastest convergence but it was pretty unsatisfying that I just used some as hoc method to add the results. It was all closed source and I haven't worked there for years so the code is lost to the wind unfortunately.
- pocketarc 3y ago
  This is honestly very interesting, thank you so much for elaborating! To be fair, after today, there are now nearly 400 TV shows with votes, so I can start seriously looking into this very soon!

michaelrpeskin 3y ago

I did something similar, in fact, the math may be the same thing and just expressed differently. But when I've had to rank non-transitive things, I use Elo (https://en.wikipedia.org/wiki/Elo_rating_system https://en.wikipedia.org/wiki/Elo_rating_system) Many years ago when I was a mid-level developer at a dysfunctional company, I was senior enough to be invited to some "strategy" meetings, but junior enough that no one ever listened to me. We (engineering, sales, marketing, etc.) spent nearly an entire summer bickering over what "important" features we were going to schedule next. I finally got fed up, took everything out of the ticketing system and made random parings and had people vote on it. Then just like a chess match, updated their Elo score based on the outcome. Then I had anyone who cared play match-ups for as long as they wanted. We ended up getting a decent ordering of features and finally ended the summer of hell meetings. I don't know if the order was the correct order, I didn't stay around long enough to see. I was just happy that sales and marketing folks thought that I had some magic math that solved their problem, and I was happy to be back developing and not sitting in useless meetings. What I like about this is that you don't have to be self consistent, as long as on average you pick the best, it will bubble to the top. And you can mix the results of other voters and see what the "true" winner is. (Of course, to be fair, you have to give each person the same number of match ups, in my case, I just served match-ups to anyone who wanted to sit at the terminal and vote, so someone could have wasted an entire day and overwhelm the system - I didn't care at the time).

zeroonetwothree 3y ago

This is so cool. I always wanted a way to do this

manx 3y ago

Interesting! We could use this algorithm to rank websites for every term in search engines. Just need a good UI to collect the data.