3 ms·
Makes sense to me. It also seems that a good part of the dupes come from the fact that the full URL is used as the identifier. And often times that URL has a
by brk 17y ago
Makes sense to me. It also seems that a good part of the dupes come from the fact that the full URL is used as the identifier. And often times that URL has a lot of referrer garbage and similar data in it. It would seem there could be some logic to try and pare URLs down somewhat in the comparator code.
- RiderOfGiraffes 17y agoFor some pages the "stuff" that follows the obvious root URL is in fact required. There are no easy rules for paring down the URL - that would need to be done by hand, and most people who submit items don't take the time to do that.
- brk 17y agoSometimes it IS certainly required, but many times there are extraneous switches tacked on, which follow standard HTML patterns. Take this URL for example: http://www.wired.com/thisdayintech/2009/12/1223Shockley-Bardeen-Brattain-transistor?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+wired%2Findex+%28Wired%3A+Index+3+%28Top+Stories+2%29%29&utm_content=Google+Feedfetcher http://www.wired.com/thisdayintech/2009/12/1223Shockley-Bard... The question mark is the demarcation point between the story URL and the "garbage". While some cases may exist, I don't recall recently seeing a URL of the format: http://mydomain.com/foo/storyFetcherScript?storyID=TheOneAboutTheDonkey http://mydomain.com/foo/storyFetcherScript?storyID=TheOneAbo... Even so, one possibility would be to compare URL's up to the first question mark, or other obvious demarcation characters, and if a dupe is found ask the submitter to verify their submission is unique. Like most other things at HN, I think that for the time being an honor system yes/no question is all that would really be needed to weed out 95% of the dupe submissions.