4 ms·
3 days ago: https://news.ycombinator.com/item?id=20589870 https://news.ycombinator.com/item?id=20589870
by nothrabannosir 7y ago
3 days ago: https://news.ycombinator.com/item?id=20589870 https://news.ycombinator.com/item?id=20589870
- rwmj 7y agoSorry about that - I thought HN automatically deduped submissions.
- Jaruzel 7y agoIt does, but only if the URL is the same. Original URL: https://www.vice.com/en_us/article/d3naek/how-to-make-a-phone-farm https://www.vice.com/en_us/article/d3naek/how-to-make-a-phon... Your submitted URL: https://www.vice.com/en_uk/article/d3naek/how-to-make-a-phone-farm https://www.vice.com/en_uk/article/d3naek/how-to-make-a-phon...
- nothrabannosir 7y agoDoesn’t matter, it should and if it doesn’t it’s not your fault. Never blame the user ;)
- dang 7y agoJust out of curiosity, how would you write software to catch that those two URLs pointed to the same thing?
- krapp 7y agoI have an idea: Crawl the page, create similarity hashes of the text and image contents, and keep track of the metadata and outbound links, which will probably match or mostly match between page versions, and related pages (particularly if they contain social media tags, or author and publisher, etc.) If any two submissions match text and images within some margin, or share the same links, consider one a possible dupe and ask the user if they want to post it or just vote up what seems to be the existing article (this will allow user curation to reduce false positives.) Also, automatically flag reposts for mod review later.
- nothrabannosir 7y agoIf you pressed me for an answer: similar to how Stackoverflow suggests possibly related questions. Start with a distance function from url to url. Easiest would be based on simple URL string distance (would have shown this as very similar). If you want to get fancy, fetch the content and do a distance function on that. I guess you can endlessly tweak that distance function, incorporate <meta canonical, etc. Probably unnecessary. Then: order by distance asc limit 5, ask users “is your URL a dupe of any of these?” Although a pretty solid alternative solution is: let the community deal with it. It’s not a huge problem at the moment, I don’t think anyone particularly minds (?). HN doesn’t do all it “should”, and (personally) that’s precisely what’s enjoyable. So what if there’s a dupe? Someone posts the original, problem solved :D As long as nobody feels bad, everything’s rosy.
- hombre_fatal 7y agoDon't worry. HN threads are essentially dead after 24 hours because nobody is going to see your comment after that point. Much less in a three day old submission.