3 ms·
You need to either work with the urls or scrape the target page; this makes it sound like you want to scrape the page. The problem with the latter is that it's
by x0x0 12y ago
You need to either work with the urls or scrape the target page; this makes it sound like you want to scrape the page. The problem with the latter is that it's a lot of work to get rid of all the cruft (header, footer, nav, ads, js) around the actual content and the python libs that claim to do this for you do an utter shit job. I've used boilerpipe and it's not helpful. diffbot claims to do this but I've never tried them.
What do you mean by blogspam?
If you had pure article text, you can do a pretty good job. An obvious technique is to build a histogram over 1,2,3 grams then do something like cos_sim. This sounds very slow but you don't have that much content and can be easily sped up via lsh. That doesn't solve your containment problem, but it helps w/ dups. Containment could potentially be done via windowing.
edit -- i'm sure you've reached out, but there's a yc founder with a lot of experience doing both of these.
edit2 -- of the two challenges: 1 - turning url into extracted text, and 2 - fingerprint matching, #1 strikes me as much harder
edit3 -- I'm going to suggest the plagiarism detection is probably misleading because they're working with already clean text, eg word or txt representations. If you want fingerprinting to work, you need to test it with all the page cruft or your best attempt at quickly cleaning page cruft. Again, I think you'll find this harder.