I tackled this for a YC-funded company in the recent past, and it's fundamentally trickier than you might think. If filtering out duplicate urls isn't good eno

I tackled this for a YC-funded company in the recent past, and it's fundamentally trickier than you might think. If filtering out duplicate urls isn't good enough, matching copied content is the next easiest way to solve this, but it requires analyzing/storing/indexing relatively massive amounts of data. And even with that cost, there's the question of how much overlapping content actually makes something a "dupe". Overlapping subjects? Crowdsourcing gets into merging comment threads and system gaming.. and the end result is still that one URL (probably from the most popular, "trusted" source) ends up on the front page anyhow. So.. lots of messy boring work and higher server expenses, for .. ? I'd call it good enough, myself