3 ms·

For some pages the "stuff" that follows the obvious root URL is in fact required. There are no easy rules for paring down the URL - that would need to be done

For some pages the "stuff" that follows the obvious root URL is in fact required. There are no easy rules for paring down the URL - that would need to be done by hand, and most people who submit items don't take the time to do that.

brk 17y ago
Sometimes it IS certainly required, but many times there are extraneous switches tacked on, which follow standard HTML patterns. Take this URL for example: http://www.wired.com/thisdayintech/2009/12/1223Shockley-Bardeen-Brattain-transistor?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+wired%2Findex+%28Wired%3A+Index+3+%28Top+Stories+2%29%29&utm_content=Google+Feedfetcher http://www.wired.com/thisdayintech/2009/12/1223Shockley-Bard... The question mark is the demarcation point between the story URL and the "garbage". While some cases may exist, I don't recall recently seeing a URL of the format: http://mydomain.com/foo/storyFetcherScript?storyID=TheOneAboutTheDonkey http://mydomain.com/foo/storyFetcherScript?storyID=TheOneAbo... Even so, one possibility would be to compare URL's up to the first question mark, or other obvious demarcation characters, and if a dupe is found ask the submitter to verify their submission is unique. Like most other things at HN, I think that for the time being an honor system yes/no question is all that would really be needed to weed out 95% of the dupe submissions.