I've done something similar. You're essentially doing A/B comparisons across the entire set. It looks like you have it so you're basically setting where "B" i

I've done something similar. You're essentially doing A/B comparisons across the entire set. It looks like you have it so you're basically setting where "B" is before moving on to the next item. This isn't strictly necessary. You could just generate a novel pair every time and ask the user to choose between them. The thing is that you'd need a way to track a user. So you can make sure that user hasn't seen a certain pair already. Once you've exhausted all the pairs, you'll know exactly how to sort the array. You'll have an idea before. You might have an issue with circular lists though. People are fickle. You could have someone who says that A > B > C > A. In this case, I'd allow for repeat pairings after a certain amount of time. To allow the person to reevaluate essentially. You could also take the comparisons across all users and compile a general sort of "best of" ranking.