4 ms·
Ask HN: Why aren't tweets picked up by the duplicate filter?
I'm not sure Elon Musk shitposts constitute news at any time, but why are people able to post the same tweet over and over without being flagged by HN's otherwise-efficient duplicate filter? HN strips query data from URLs before they are posted, maybe it should do so prior to the duplicate check too since most Twitter URLs come with a large helping of cruft.
And please, check the New page before submitting. There are 4 links to the same evidence-free tweet in the last 10 minutes.
- PaulHoule 4y agoThere's a button for that and you have enough karma to use it. But really I can't decide if links to Twitter are better or worse than links to Medium. 1 time in 20 or so somebody posts a highly intelligent series of tweets that are worth reading, but most of the time a link to twitter points to a poster that doesn't really have anything to say, a bunch of comments that say at best "me too!" and a lot of irrelevant junk. (Almost as bad a link to a "good" discussion on reddit that is spammed up with irrelevant discussions like a porn site where you have the hardest time telling your wife, 'ignore the 15 animated gifs of pistoning action and look at that picture which I think is really hot')
- gnicholas 4y agoWhat's the button, and what's the karma threshold?
- anigbrowl 4y agoFlagging posts/comments you dislike. I think the threshold is 2000 or something but I haven't thought about it in years. Unfortunately the comment you replied to misunderstood my question. I wasn't complaining about tweets in general (though I find them mildly annoying) but expressing confusion about why a specific tweet (with an inherently unique URL) was being posted over and over when the HN dupe filter should have prevented that.
- PaulHoule 4y agoIt's not intended to mean "dislike" but rather that you believe it is a violation of the guidelines. It's part of the system for keeping the front page clean. Anybody who is posting anything about the twitterpocalypse however is pretty oblivious because if they were reading the front page they'd notice that nothing about it is making the front page, including the people who posted an "Ask HN: Can you knock it off posting about Twitter?"
- anigbrowl 4y agoThe question was about the duplication filter specifically, not about twitter in general or the twitterocalypse or why HN users post tweets.
- gnicholas 4y agoMy question about buttons was about this also. Is there a dupe button? How do posts get the [dupe] label if not? Is it manually applied after people generically flag it?
- anigbrowl 4y agoNormally if you submit the same link as someone else it will be automatically flagged as a duplicate, and you'll get a link to the original post. This works well most of the time, which is why I was surprised that individual tweets (with their very specific URLs) didn't set it off. The duplication filter is run by HN rather than user flags. Most of the time it's effective, although if someone posts an interesting story at the wrong time of day and it dies for lack of attention the dupe filter can defeat subsequent efforts by people to repost it. However it's easy to circumvent by modifying the submission URL, just don't abuse it.
- PaulHoule 4y agoSee https://github.com/minimaxir/hacker-news-undocumented#flaggingvouching https://github.com/minimaxir/hacker-news-undocumented#flaggi...
- Trouble_007 4y agoI have no affiliation with HN. The duplicate filter, along with all of HN code is a work in constant review, rewrite, test, and publish. Setting the parameters for such filtering requires time, https://news.ycombinator.com/from?site=twitter.com/elonmusk https://news.ycombinator.com/from?site=twitter.com/elonmusk to adjust window of operation, frequency, etc. https://news.ycombinator.com/from?site=twitter.com https://news.ycombinator.com/from?site=twitter.com
- r721 4y agoPossibly related - looks like recent posts are missing in Algolia search index: https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=musk&sort=byDate&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu... https://hn.algolia.com/?dateRange=all&page=0&prefix=false&query=apple&sort=byDate&type=story https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...
- aaron695 4y ago
- r721 4y agoDang's reply: https://news.ycombinator.com/item?id=33790811 https://news.ycombinator.com/item?id=33790811 >they've found the problem and are currently backfilling the missing data