4 ms·

If HN would do a dup check on the canonical url instead of just the submitted one it would avoid these duplicates. Isn't that just what the canonical meta tag i

If HN would do a dup check on the canonical url instead of just the submitted one it would avoid these duplicates. Isn't that just what the canonical meta tag is there for?

RiderOfGiraffes 16y ago
Not all sites use it. I suggested a while ago fetching the page and extracting the "<title>" Then compare the URL up to any "?" or other stuff on the end, and checking the title. This would catch every duplicate I've seen.
- Encosia 16y ago
  Ignoring the querystring would break for a lot of sites' URLs, like the default WordPress URLs (Hacker News' too, to be recursive).
  - panic 16y ago
    The system doesn't have to be automatic. It could look for a list of possible duplicates by ignoring the query string, then show them to the user if the list is shorter than some threshold.
  - RiderOfGiraffes 16y ago
    That's what I get for trying to describe a longer algorithm when I don't have time. Which I don't. Ignoring the query string is only done when comparing the titles. If the initial section of the URL matches and the title matches then it's almost always a duplicate, and I only say "almost always" because there's a chance I'm wrong when I say "always" - I've never seen it. But nothing's going to be done, and I don't have time to implement anything more robust than the quick hack I've done for myself, so I'm not going to get involved.
- MichaelApproved 16y ago
  All sites don't have to use it to implement it. Check it for sites that have it and fall back to just url checking for sites that don't.
blasdel 16y ago
HN doesn't even do a consistent equality-based dupe check of the submitted URL! Submitted URLs are only checked against the URLs of past stories that are cached in memory, and the Arc process crashes regularly, which invalidates all comment forms and starts the cache fresh again. The result is that the duplicate check is only processed against stories that have been displayed in some form in a response since the last crash. That end result is actually kind of elegant — you can only submit a direct dupe if nobody cares about the old one, but the means to that end is pretty fucked up. A worthwhile accident.

If HN would do a dup check on the canonical url instead of just the submitted one it would avoid these duplicates. Isn't that just what the canonical meta tag is there for?

RiderOfGiraffes 16y ago

Not all sites use it. I suggested a while ago fetching the page and extracting the "<title>" Then compare the URL up to any "?" or other stuff on the end, and checking the title. This would catch every duplicate I've seen.

Encosia 16y ago
Ignoring the querystring would break for a lot of sites' URLs, like the default WordPress URLs (Hacker News' too, to be recursive).
- panic 16y ago
  The system doesn't have to be automatic. It could look for a list of possible duplicates by ignoring the query string, then show them to the user if the list is shorter than some threshold.
- RiderOfGiraffes 16y ago
  That's what I get for trying to describe a longer algorithm when I don't have time. Which I don't. Ignoring the query string is only done when comparing the titles. If the initial section of the URL matches and the title matches then it's almost always a duplicate, and I only say "almost always" because there's a chance I'm wrong when I say "always" - I've never seen it. But nothing's going to be done, and I don't have time to implement anything more robust than the quick hack I've done for myself, so I'm not going to get involved.
MichaelApproved 16y ago
All sites don't have to use it to implement it. Check it for sites that have it and fall back to just url checking for sites that don't.

blasdel 16y ago

HN doesn't even do a consistent equality-based dupe check of the submitted URL! Submitted URLs are only checked against the URLs of past stories that are cached in memory, and the Arc process crashes regularly, which invalidates all comment forms and starts the cache fresh again. The result is that the duplicate check is only processed against stories that have been displayed in some form in a response since the last crash. That end result is actually kind of elegant — you can only submit a direct dupe if nobody cares about the old one, but the means to that end is pretty fucked up. A worthwhile accident.