3 ms·

@Dang (HN "meta" Q&A) Has HN ever looked into clustering news using Google News to do it? Reason I ask is how HN's de-dup filter works is puzzling to me and r

@Dang (HN "meta" Q&A) Has HN ever looked into clustering news using Google News to do it? Reason I ask is how HN's de-dup filter works is puzzling to me and related stories would likely be useful to pool comments/de-dup/etc on a single news event. Example of dups, clusters, themes on HN on this event: https://hn.algolia.com/?query=westinghouse&sort=byDate&prefix=false&page=0&dateRange=all&type=story https://hn.algolia.com/?query=westinghouse&sort=byDate&prefi... Google News cluster: https://news.google.com/news/m/more?ncl=dGww1eRZGqsOScMMCg3U3tq_4ODEM&authuser=0&ned=us&topic=b https://news.google.com/news/m/more?ncl=dGww1eRZGqsOScMMCg3U...

striking 10y ago
The de-dupe filter is based on how much discussion went on in past posts. But that actually doesn't affect the current case we're looking at here, as there are no duplicates of this link: https://hn.algolia.com/?query=https:%2F%2Fwww.nytimes.com%2F2017%2F03%2F29%2Fbusiness%2Fwestinghouse-toshiba-nuclear-bankruptcy.html&sort=byDate&prefix=false&page=0&dateRange=all&type=story https://hn.algolia.com/?query=https:%2F%2Fwww.nytimes.com%2F... Now, in a sense, there was a duplicate, but because the URL didn't match, it didn't count. https://hn.algolia.com/?query=https:%2F%2F*.nytimes.com%2F2017%2F03%2F29%2Fbusiness%2Fwestinghouse-toshiba-nuclear-bankruptcy.html&sort=byDate&prefix&page=0&dateRange=all&type=story https://hn.algolia.com/?query=https:%2F%2F*.nytimes.com%2F20... The de-dupe filter would've turned new submissions of your post into votes for your post, if it had caught the duplicate. That being said, I don't think HN is designed to pool discussion over time, preferring to archive discussion of each story. At this point, you can't even upvote older posts; they can only be added to your favorites.
- wonderous 10y ago
  Yes the URL is different, but the canonical URL is the same. Not concerned about the votes, just curious if the topic of canonical URLs, clusters, etc. had come up.
  - striking 10y ago
    You raise an interesting point. I'd recommend shooting the HN team an email about making use of canonical URLs for de-duping. mailto:hn@ycombinator.com
    - wonderous 10y ago
      Thanks. I'm aware of the HN support email, not interested in private dialogs; whole topic of being open, accessible, referenable, etc. is a different topic, and not something I'm looking to cover in the thread unless it's requested.

striking 10y ago

The de-dupe filter is based on how much discussion went on in past posts. But that actually doesn't affect the current case we're looking at here, as there are no duplicates of this link: https://hn.algolia.com/?query=https:%2F%2Fwww.nytimes.com%2F2017%2F03%2F29%2Fbusiness%2Fwestinghouse-toshiba-nuclear-bankruptcy.html&sort=byDate&prefix=false&page=0&dateRange=all&type=story https://hn.algolia.com/?query=https:%2F%2Fwww.nytimes.com%2F... Now, in a sense, there was a duplicate, but because the URL didn't match, it didn't count. https://hn.algolia.com/?query=https:%2F%2F*.nytimes.com%2F2017%2F03%2F29%2Fbusiness%2Fwestinghouse-toshiba-nuclear-bankruptcy.html&sort=byDate&prefix&page=0&dateRange=all&type=story https://hn.algolia.com/?query=https:%2F%2F*.nytimes.com%2F20... The de-dupe filter would've turned new submissions of your post into votes for your post, if it had caught the duplicate. That being said, I don't think HN is designed to pool discussion over time, preferring to archive discussion of each story. At this point, you can't even upvote older posts; they can only be added to your favorites.

wonderous 10y ago
Yes the URL is different, but the canonical URL is the same. Not concerned about the votes, just curious if the topic of canonical URLs, clusters, etc. had come up.
- striking 10y ago
  You raise an interesting point. I'd recommend shooting the HN team an email about making use of canonical URLs for de-duping. mailto:hn@ycombinator.com
  - wonderous 10y ago
    Thanks. I'm aware of the HN support email, not interested in private dialogs; whole topic of being open, accessible, referenable, etc. is a different topic, and not something I'm looking to cover in the thread unless it's requested.