This is a problem well-known in Web search as 'position bias': top results get more clicks and therefore get 'upvoted' by machine learning algorithms. There ar

This is a problem well-known in Web search as 'position bias': top results get more clicks and therefore get 'upvoted' by machine learning algorithms. There are theoretically sound solutions. Typically, I recommend [1]. Very briefly the solution is to yield an unbiased click estimate (or for what matters here, an unbiased number of upvotes). Look at the position and cascade models as well as the solution in [1]. An approximation of this solution at the end of the article is very easy to implement while theoretically sound. Side note: another 'solution' would be to randomly generate a slightly different order for every reader (e.g. sampling based on the current number of upvotes for every comment). The readers would then spread their upvotes across more comments overall. [1] O. Chapelle and Y. Zhang. A dynamic bayesian network click model for web search ranking. In Proceedings of the 18th International World Wide Web Conference (WWW), 2009. http://olivier.chapelle.cc/pub/DBN_www2009.pdf http://olivier.chapelle.cc/pub/DBN_www2009.pdf