This is interesting. I used to to work with SVM document categorization, and this was a huge problem. This was years ago, so forgive me for not ever having an

This is interesting. I used to to work with SVM document categorization, and this was a huge problem. This was years ago, so forgive me for not ever having an elegant solution, but I never thought to use something similar to Markov's chains (http://en.wikipedia.org/wiki/Markov_chain http://en.wikipedia.org/wiki/Markov_chain) to detect duplicates or avoid layout data. I ended up doing custom layout detection and trying to ignore headers, navigational menus, footers. It was always a constant battle and very brittle and hackish. It worked OK for my purposes, but I still remember going through and checking a whole bunch of new entries every time the crawler pulled down new stories to see if I should add some obvious ignores to the filters.