You could also try to do content-based duplicate detection: rather than looking at the URL, try to determine whether a new article is substantively similar to a

You could also try to do content-based duplicate detection: rather than looking at the URL, try to determine whether a new article is substantively similar to a previously-posted one. Getting that to work with a high degree of accuracy would take some thought, however. Perhaps there's some work on this in the information retrieval literature?