A good approach would be to use Elo ratings from chess in a similar fashion to how Mark Zuckerberg used Elo ratings for Facemash when he was at Harvard. Assign

A good approach would be to use Elo ratings from chess in a similar fashion to how Mark Zuckerberg used Elo ratings for Facemash when he was at Harvard. Assign each application a rating of 1000. Randomly select two applications and show them to a partner. The partner selects the better application, or declares a tie. The Elo ratings are updated accordingly. Aggregate this over many partners and many applications, and you now have a quantifiable measure of YC's aggregate preferences for the applications. Further, you can now calculate the probability that YC will prefer one application over the other by comparing ratings, even if the two applications have never been directly compared before. Then, rank based on ratings and select the top N (N being the number they want to interview). The cool thing about this is it's very simple to implement (easily done in a day, or even a few hours). It'd be a useful measure even for YC's regular batches, which have more manageable numbers.