3 ms·

I actually already wrote this as a reply to another comment, but I think it merits a comment of its own. The best way to do the scoring for the applications is

by zaguios 11y ago

I actually already wrote this as a reply to another comment, but I think it merits a comment of its own. The best way to do the scoring for the applications is to actually make it into somewhat of a game. Each application should be given a starting elo of 1200. Every time a user wants to go and review applications they will be presented with two separate applications with a similar elo and then will be asked to vote on which one they think is better. Elo will then be added/subtracted with normal conditions based on the vote. This system would very quickly and accurately identify the best applications. I've seen this done before for other things and it's been proven to be very effective and prevent almost any sort of bias.

Kapow 11y ago
Would this be implemented as only showing pairs within a certain threshold of difference, or would it choose an application purely at random and then the application with the closest score (or even 50/50 the next highest or next lowest)? The former seems like it could lead to cases where applications end up in a void between clusters and stop appearing in votes, preventing late voters from giving an opinion on them. Is there a name for this kind of voting process?
- zaguios 11y ago
  To ensure each application gets the same amount of views (with the exception of the top ranked and bottom ranked applications) you can pick the next highest/lowest and it will work out fine. My concern with this implementation was that it might be possible for the same applications to be compared multiple times at the high and low ends of the bell curve where elo becomes more spaced out. A better solution might be internally listing all the applications by their elo, selecting an application and random, and then selecting from an application within 5 ranks of itself. I.E. App #342 is randomly chosen and then an application between #337 and #347 will be randomly chosen to compare it with. I am not aware of any name for this voting process. Each potential implementation has slightly better/worse corner cases, but the general quality of the ranking should still be extremely accurate regardless of the particular implementation. It takes surprisingly few comparisons for items to find their appropriate ranks.
GFischer 11y ago
Interesting, I was also going to suggest some kind of alternative review system. I was invited to review applications to CORFO (the government program behind Startup Chile), and since they crowdsource reviews between unrelated startup founders, they have a "points" system to determine whether submissions were reasonably graded (they look for big differences between graders). Your system sounds better :) , but I think they want to promote individual discussion of submissions too.
- zaguios 11y ago
  Yea, a point system would work decently, but as you mentioned there is the problem where certain reviewers give higher and lower ratings than one another. With a large enough sample size you should still get fairly accurate ratings. A 1 vs 1 system based on elo eliminates all of these issues though and helps you find accurate ratings much faster. The system doesn't need to stop individual discussion, you should be allowed to post responses to the applications as your are reviewing them to vote on. You should also be allowed to directly view and comment on an application by going to it directly as long as you aren't allowed to do anything that affects its rating.
dang 11y ago
That's pretty interesting. Is this https://en.wikipedia.org/wiki/Elo_rating_system https://en.wikipedia.org/wiki/Elo_rating_system?
- zaguios 11y ago
  Yep, implementation is fairly simple. I find the wikipedia article to be hard to follow so I'll give an example here to simplify it. Let's say the applications you want to compare have an elo of 1200 and 1600 respectively. Let's also use a kFactor of 20 (This determines how fast the elos will change based on a single comparison and can be modified to your liking). http://pastebin.com/65WGmuyN http://pastebin.com/65WGmuyN This example is for two applications fairly far apart in elo, if each application had an elo of 1200 then one would increase by 10 and the other would decrease by 10.
pbnjay 11y ago
I think in addition to a win/loss/draw judgement, it would be good to require a comment (and hence username) for each application. That would at the least give a way to trace if the Elo ratings are being gamed or users are biased.

by zaguios 11y ago

Kapow 11y ago

Would this be implemented as only showing pairs within a certain threshold of difference, or would it choose an application purely at random and then the application with the closest score (or even 50/50 the next highest or next lowest)? The former seems like it could lead to cases where applications end up in a void between clusters and stop appearing in votes, preventing late voters from giving an opinion on them. Is there a name for this kind of voting process?

zaguios 11y ago
To ensure each application gets the same amount of views (with the exception of the top ranked and bottom ranked applications) you can pick the next highest/lowest and it will work out fine. My concern with this implementation was that it might be possible for the same applications to be compared multiple times at the high and low ends of the bell curve where elo becomes more spaced out. A better solution might be internally listing all the applications by their elo, selecting an application and random, and then selecting from an application within 5 ranks of itself. I.E. App #342 is randomly chosen and then an application between #337 and #347 will be randomly chosen to compare it with. I am not aware of any name for this voting process. Each potential implementation has slightly better/worse corner cases, but the general quality of the ranking should still be extremely accurate regardless of the particular implementation. It takes surprisingly few comparisons for items to find their appropriate ranks.

GFischer 11y ago

Interesting, I was also going to suggest some kind of alternative review system. I was invited to review applications to CORFO (the government program behind Startup Chile), and since they crowdsource reviews between unrelated startup founders, they have a "points" system to determine whether submissions were reasonably graded (they look for big differences between graders). Your system sounds better :) , but I think they want to promote individual discussion of submissions too.

zaguios 11y ago
Yea, a point system would work decently, but as you mentioned there is the problem where certain reviewers give higher and lower ratings than one another. With a large enough sample size you should still get fairly accurate ratings. A 1 vs 1 system based on elo eliminates all of these issues though and helps you find accurate ratings much faster. The system doesn't need to stop individual discussion, you should be allowed to post responses to the applications as your are reviewing them to vote on. You should also be allowed to directly view and comment on an application by going to it directly as long as you aren't allowed to do anything that affects its rating.

dang 11y ago

That's pretty interesting. Is this https://en.wikipedia.org/wiki/Elo_rating_system https://en.wikipedia.org/wiki/Elo_rating_system?

zaguios 11y ago
Yep, implementation is fairly simple. I find the wikipedia article to be hard to follow so I'll give an example here to simplify it. Let's say the applications you want to compare have an elo of 1200 and 1600 respectively. Let's also use a kFactor of 20 (This determines how fast the elos will change based on a single comparison and can be modified to your liking). http://pastebin.com/65WGmuyN http://pastebin.com/65WGmuyN This example is for two applications fairly far apart in elo, if each application had an elo of 1200 then one would increase by 10 and the other would decrease by 10.

pbnjay 11y ago

I think in addition to a win/loss/draw judgement, it would be good to require a comment (and hence username) for each application. That would at the least give a way to trace if the Elo ratings are being gamed or users are biased.