3 ms·
I have an idea: Crawl the page, create similarity hashes of the text and image contents, and keep track of the metadata and outbound links, which will probably
by krapp 7y ago
I have an idea:
Crawl the page, create similarity hashes of the text and image contents, and keep track of the metadata and outbound links, which will probably match or mostly match between page versions, and related pages (particularly if they contain social media tags, or author and publisher, etc.)
If any two submissions match text and images within some margin, or share the same links, consider one a possible dupe and ask the user if they want to post it or just vote up what seems to be the existing article (this will allow user curation to reduce false positives.) Also, automatically flag reposts for mod review later.