It's even more odd considering diversity in page ranking and novelty in recommendation are well studied (although not solved) problems in information retrieval,

It's even more odd considering diversity in page ranking and novelty in recommendation are well studied (although not solved) problems in information retrieval, so it's not like there isn't a solution. And it's not like Google doesn't have the human or computer resources to tackle it.