20 ms·

Stories that Hacker News removes from the front page

• PleaseHelpMe 10y ago
  > The stories that Hacker News removes from the front page (sangaline.com) is : http://sangaline.com/post/reverse-engineering-the-hacker-news-ranking-algorithm/ http://sangaline.com/post/reverse-engineering-the-hacker-new...
• eli 10y ago
  I find this sort of conspiratorial meta complaint boring and (IMHO) off topic. I didn't flag it, but I'd understand if others did.
  • suby 10y ago
    It seems to me that it's healthy if we as a community are able to discuss issues with how the community is policed. Some may find it boring, but I'd argue that it's useful.
    • alex_hitchins 10y ago
      I agree. I thought the write-up was fair and reasonably level headed. If something popular (and surely by definition of being on the front page it is) is suddenly removed, people are bound to be interested in the reason why? Was the source discredited? Was it just a copyright issue? A simple filter for spiked stories would be good, just with a note on the reason why. Of course HN don't have to implement this, but it would be of benefit to the community.
      • DanBC 10y ago
        But the answer is always going to be "users flagged it", or "the flamewar detector was triggered", or "the vote ring detector was triggered". And advice is always going to be "if you notice something unusual email the mods to discuss it, because they can fix it if you send them an email; they might not see it if you leave a comment in a thread".
        • alex_hitchins 10y ago
          My point is you don't notice every post that gets removed and there is currently no transparancy on the reason why. HN doesn't have to be transparent, it's just a site with it's own agenda (by that I don't mean evil agenda, but it is there for a reason) but if you want to grow the community, I think clearly identifying why things were removed is a reasonable thing to ask. If every one it marked "flagged by users" I'd worry that there is no manual intervention.
    • 3131s 10y ago
      I did an Ask HN one time something along the lines of "how would you game the HN ranking algorithm?". It seems like good security practice IMO to have a discussion on what is publicly known / discernible about the ranking algorithm. This was during the time of the election so I was thinking along the lines of political astroturfing, but also to guard against companies unfairly promoting their products or suppressing posts related to a rival company. For instance, if someone really wanted to keep a discussion off HN, all it would take is to tangentially start a flame war over some sensitive issue and watch the ranking algorithm punish the ensuing vitriol.
  • _benedict 10y ago
    I don't really read this terribly as complaining. It is an interesting analysis of some posts (including his own) that seemed to behave atypically in a non-transparent manner to the user base, that may suggest some manipulation of the ranking. At no point does it seem to take any accusatory tone, or complain. The hypothesis may be incorrect, but it is not unreasonable, and it does seem relevant to readers of hacker news. I don't think being "bored" by something is generally considered a good reason to flag content. I don't see how it could possibly be considered off-topic either.
  • deleted 10y ago
    [deleted]
• paulpauper 10y ago
  Posting on a social news site is a privilege, not a right. I've had stuff removed from Reddit and a few from Hacker News that got flagged after going on the front page...I've had stories on Reddit that got many upvotes and still removed just as it was going viral because the mods didn't like it...it's just the way it goes. It def. can seem unfair at times. No question about it.
  • literallycancer 10y ago
    That may be so, but it's useful for us to know whether/and what kind of things get removed.
  • jrochkind1 10y ago
    What's this got to do with anything in the OP? Did the OP say anything to the contrary? It is appropriate that we readers/"consumers" have an interest in how our sources of information are controlled.
  • acdha 10y ago
    This is technically true but misses important social aspects: does the average HN reader know which stories were removed at all, and why? If a certain political viewpoint, criticism of popular companies, etc. is being moderated or flag-killed that'll come as a surprise to anyone who never saw it and would have been interested if they'd had the chance.
  • forgotpwtomain 10y ago
    > Posting on a social news site is a privilege, not a right. I certainly don't share the sentiment. There are lots of places I or other people can visit and post to, what makes this particular site commend value is precisely the community and user-base it has fostered over the years. Thus, I believe that this kind of meta discussion does provide considerable value to the community.
    • nottorp 10y ago
      It may be a privilege, but in this case my reading Hacker News is a privilege for them, and not their right. And I want to know what stories are manually suppressed in order to decide if i still grant them that privilege. Also, check out Tim Berner Lee's article about the internet being hijacked by the likes of Facebook and Google in order to understand why your 'I am grateful for the privilege' attitude is wrong... Edit: argh, I attached this to the wrong post. I was responding to paulpauper of course.
• s_dev 10y ago
  Cache/Mirror: http://webcache.googleusercontent.com/search?q=cache:http://sangaline.com/post/the-stories-that-hacker-news-removes-from-the-front-page/ http://webcache.googleusercontent.com/search?q=cache:http://...
  • remotebug 10y ago
    Archive: http://web.archive.org/web/20170313124045/http://sangaline.com/post/the-stories-that-hacker-news-removes-from-the-front-page/ http://web.archive.org/web/20170313124045/http://sangaline.c...
• hardwaresofton 10y ago
  The site is being crushed by traffic right now -- but without reading the article, I've also found that some stories that I thought were important that were scrubbed from HN's front page just about as soon as I saw it (when I doubled back to read the comments)... While I realize I'm not entitled to explanations, some transparency would be appreciated. Maybe it could even be automatic, whenever a mod removed something forcibly from front, they could leave a comment and it'd show up on some page? [EDIT] - After reading the article, if a mod did indeed take down the post because it discussed reverse engineering the rank algorithm, I think that's pretty naive. Security through obscurity isn't a thing, and the better response is just to make a better algorithm, not try and suppress knowledge about it. I say this naively myself, as I've never had to maintain a ranking algorithm with these many users who depend on it (or any at all for that matter), but surely the problem isn't intractable?
  • DanBC 10y ago
    > whenever a mod removed something forcibly from front Mods (as far as I know, I'm not one) never forcibly remove things from the front page. These are almost always the result of user flagging.
    • koolba 10y ago
      There can be the reverse though. A controversial post may get flagged repeatedly but the impact of flagging could be disabled by mods to not disable the post.
    • Mithaldu 10y ago
      Those are marked first "flagged" as a warning, and then "dead" when the user flags reach a high enough level. This article however is about stories that disappear without receiving either.
      • DanBC 10y ago
        When does the flag label appear though? Here's an example of a submission that does not include the [flagged] label, but which has a moderator saying it got flagged, and that the flags are the reason it's not on the front page. https://news.ycombinator.com/item?id=13741276 https://news.ycombinator.com/item?id=13741276
• iplaw 10y ago
  I didn't understand why the guy's PayPal story was removed from the front page -- the one about PayPal seizing a $40k USD balance without warning, allegedly due to a 2% chargeback rate over several years of doing business and hundreds of thousands of dollars in successful transactions. I thought that the point of HN was auto-moderation? Perhaps now that HN has seen great increases in popularity, the quality of content has to be more carefully controlled, lest the quality of posts on the HN front page slowly enter a death spiral towards that of reddit. “Think of how stupid the average person is, and realize half of them are stupider than that.” ― George Carlin
  • jasode 10y ago
    I don't think the Paypal story was "removed". That story is now on page 2. Yesterday, it stayed on the front page all day. It's decay from the 1st page to the 2nd page a day later seems comparable to other stories with ~200 comments. The author's meaning of "remove from the frontpage" means abruptly disappearing from the front page and you still don't see it in the subsequent 6 or more pages.
    • tptacek 10y ago
      Right. Stories that get buried fall from the front page to, like, the 50th. Not to the 2nd.
      • sillysaurus3 10y ago
        That's not quite true. "Buried" just means "gravity has been increased." Sometimes the gravity is adjusted massively, which moves stories to the 50th page. But it's very common for stories to go from page 1 to page 2. Even more common is to go from rank ~5 to rank ~29.
  • minimaxir 10y ago
    In addition to what jasode said, Ask HNs/self-posts have an additional penalty. (which is something I disagree with)
  • 10y ago
• laurent123456 10y ago
  As much as I like HN, I'm not a big fan of the secrecy around moderator interventions - what gets censored, what posts get re-titled, etc. I can understand they might want to keep the ranking algorithm and anti-spam techniques secret, but stuff that are manually censored by a moderator should be indicated as such, maybe by some automatic message like "This post was removed due to [reason]". Some websites manage to fight spam while remaining reasonably transparent (eg. StackExchange, where pretty much everything is documented - flags, closing reasons, edits, etc.).
  • DanBC 10y ago
    > I'm not a big fan of the secrecy around moderator interventions - what gets censored, what posts get re-titled, etc. This feels pretty transparent: https://news.ycombinator.com/threads?id=dang https://news.ycombinator.com/threads?id=dang https://news.ycombinator.com/threads?id=sctb https://news.ycombinator.com/threads?id=sctb
    • Mithaldu 10y ago
      The problem with that is that they have various tools like adjusting things off the frontpage or banning comment threads to the bottom of the list; but the UI itself does not provide information on those. If the mods do any of these things but don't also leave a human comment on them, then it's unlikely people will notice.
      • brudgers 10y ago
        A user can see those things by turning on "Show Dead" in their profile.
        • Mithaldu 10y ago
          I have show dead on since a long time. The things i mentioned have no UI to note them. Show dead only makes, well, "dead" posts render their text.
• moomin 10y ago
  It's very interesting to note the number one pulled story: https://news.ycombinator.com/item?id=13741276 https://news.ycombinator.com/item?id=13741276. If you take a look at the comments, it's theorized there that the story got pulled not because of moderator action, but because people abused the flagging mechanism. Given the content, and given the principal person under discussion, this seems pretty likely to me.
  • willvarfar 10y ago
    In the comments you link to, dang is the moderator, so he isn't theorizing when he says: > It is the usual tug of war between upvotes and flags
    • moomin 10y ago
      Good point :)
  • belorn 10y ago
    Why would it be abuse if people flag it as being off-topic for HN? The article is about some community code of conduct text which the article author comments on by adding their own interpretations to it. It look political and it uses inflammatory language. The tug of war between upvotes and flags, as commented by dang, seems a strong sign that the article is very much political in nature. The question is if it also gratifies one's intellectual curiosity, but personally, in this case it did not do that for me.
    • moomin 10y ago
      I'd say tech culture is 100% fair game for Hacker News. In fact, it's been the subject of a fair number of front page articles recently (look at the Fowler stories, for instance). The flagging suggests to me that they didn't like what was being said, rather than they thought it was off-topic. A good test for this that the moderators could run is to see if those that flagged this article flagged any other articles about LambdaConf's policies and if so, which. Another one is to compare its performance to other, similarly political articles by the same author (e.g. his free software posts) and see how they performed.
• minimaxir 10y ago
  Historically, the mods have not killed a post for discussing Hacker News meta, although on occasion they apply a penalty to meta submissions. (the original post only had 32 upvotes, which is enough to get swallowed) Indeed, HN recently allowed a post that advocated gaming the system because it encouraged debate: https://news.ycombinator.com/item?id=13676362 https://news.ycombinator.com/item?id=13676362 A conspiracy theory, even backed by data, is not the best application of Occam's Razor.
  • Analog24 10y ago
    Presenting the facts out of context isn't helpful. The original post might have "only had 32 upvotes" but, as mentioned in this article, it received about 20 in less than an hour. More than enough to get to the front page, where it was steadily receiving more upvotes until it just vanished. The post was not "swallowed".
    • minimaxir 10y ago
      "20 in less than an hour" is not a fast velocity.
      • Analog24 10y ago
        It was enough to get on the front page. The point is that the post wasn't swallowed in the usual sense. We now know that the original post fell drastically because it was being flagged by users.
    • brudgers 10y ago
      When a story gets to the front page more people will see it and some may upvote it. Likewise more people will see it and some may flag it. I'd hypothesize that longer term or more active users are among the people who are more likely to use flags. Likewise, I suspect that long term and active users are more likely to not want to see meta posts on the front page.
• laktak 10y ago
  Don't forget that HN tries to detect vote fraud. So some articles might simply disappear because the OP asked too many friends for upvotes or because of false positives.
  • lovelearning 10y ago
    Another possibility is that voting rings for _other_ posts flag maliciously to remove the competition and give their posts a relative boost. Maybe author can find correlations with other posts that were showing on the front page at the same time as the flagged ones and see how _their_ rankings got boosted due to the flagging.
    • chrisrhoden 10y ago
      The article is about posts that never receive the "flagged" designation.
      • DanBC 10y ago
        A story is not marked as [flagged] as soon as someone flags it, and flags have influence even if the submission is not marked as [flagged]. See dang's comment in this thread: https://news.ycombinator.com/item?id=13741276 https://news.ycombinator.com/item?id=13741276
  • willvarfar 10y ago
    I've chatted to mods on various sites over the years and I've heard that "voting rings", particularly the automated ones, often try to obscure their upvotes by also upvoting obscure stories and downvoting competing stories.
  • Shivetya 10y ago
    Does HN actually suffer this? I know its done on Reddit and by who in one case; just having a family member active in politics gets you good insight how many sites they try to manipulate. I was hoping that HN's flag system would sufficient for the community to self censor. Perhaps we need a flag on comments too, something you can't see. I would also prefer that posts don't transition through the lighter grays as they get down voted but disappear completely once the dead threshold is met. That would prevent some piling on that does happen
• DanBC 10y ago
  If you notice something weird happening with your submissions you should email the mods. They're happy to explain what's going on and to have a look to see if the flags are fair.
• paulpauper 10y ago
  It's probably a combination of flagging and other factors. Obvious the mods aren't going to want to make their algos transparent. Flagging removes story completely but perhaps mods have the power to bump a story off the page but without flagging it.
• intralizee 10y ago
  Transparency is nice but most social news sites don't care for it. If there was a middle ground it probably would be a section where you can specifically view threads that were removed from view.
• DanBC 10y ago
  This blog suggest that moderator action must have been necessary, but seems to say that flags are unimportant. I disagree. Just a few flags can cause a story to drop off the front page.
  • jacquesm 10y ago
    HN is like reddit where the 'downvotes' on a story outweigh the upvotes 30 to one or so.
• ploggingdev 10y ago
  I can comment on the "Books that Aaron Swartz read, loved and hated" story. OP posted that link to HN and it made the front page. The story was soon flagged by the community (people viewed it as Amazon affiliate spam) and it fell off the rankings. When the moderator dang noticed that the story had been flagged, he decided to restore it (https://news.ycombinator.com/item?id=13840869 https://news.ycombinator.com/item?id=13840869). So yeah, that explains why this particular story fell off the front page quickly but does not have any [flagged] or [dupe] tags attached now. I've learned that it's best not to jump to conclusions based on what you think is true (however sound your analysis might be). Always ask the other side(s) for an explanation. In this case, you could have sent an email to the mods asking for an explanation. If you find their response unsatisfactory, go ahead and write a post explaining why.
  • stordoff 10y ago
    FWIW, the post does acknowledge that: > There are also a few articles that are thinly veiled affiliate spam. For example, the ShelfJoy links about books that Aaron Swartz and David Bowie loved fall into this category. They get called out in the comments for being very low-effort lists of Amazon affiliate links.
    • ploggingdev 10y ago
      OP uses this story as an example of having "moderator fingerprints on them" i.e moderators censoring posts, which is not true. What happened was the opposite : the moderator uncensored a post flagged by the community.
      • azrazalea 10y ago
        Well, the author tried to account for that by saying that normal flagged posts don't go down near as fast. If what is being said in other threads is true, high karma accounts clicking flag could push a post off the front page without showing the "flagged" tag, then that could be an explanation instead. Which would then mean that maybe these high karma users have too much power.
• logicallee 10y ago
  This is called "burying" and it is unremarkable. (This story is likely to be buried.) The stories that are buried are not appropriate for the front page. The reason you come to Hacker News is because it has a better front page, with better comments under it, than other places. You experience the benefit of this editorial intervention each and every day. I've had a story buried as it was gaining a lot of traction very quickly: this one. https://news.ycombinator.com/item?id=11920431 https://news.ycombinator.com/item?id=11920431 The quality of the comments was inordinately low and it didn't look like it would be improving, which is the reason it was buried. No complaints from me around this. You can email the moderators if you want to know their reasoning. (I'm not one.) People here need to understand and be thankful for the extraordinary and ongoing work that the moderators do every single day to keep this place an appropriate place for interesting, deep discussion along the editorial lines chosen. It is not a democracy (see: reddit) but I find the moderators generally extremely fair. As far as I understand the moderators bury tons of stories (often political, link-bait, etc), which do however get traction quickly until they do so. It is easy to get traction through click-bait. Generating serious discussion is harder. For example, this title promises "the stories that Hacker News removes" -- but is not really about the stories that Hacker News removes. For example the author does not analyze the comments under them or see why it derails or is not a good contribution to HN. It is more of a click-bait title is bait-and-switch, and is designed to generate easy outrage. There's nothing remarkable here despite the traction this story is getting. It is part of the hidden workings that keep HN great. Dan and Scott (the moderators) do an extremely good and thankless job keeping the principles of this place alive. You have no idea how hard they work and I've seen them make difficult and intricate decisions. (Sometimes as simple as detaching a thread that was derailing an important discussion.) In my opinion this story does not belong on the front page.
  • minimaxir 10y ago
    > I've had a story buried as it was gaining a lot of traction very quickly: this one. https://news.ycombinator.com/item?id=11920431 https://news.ycombinator.com/item?id=11920431 I would have flagged that submission if I saw it since it's flamebait.
    • 10y ago
• midgetjones 10y ago
  I guess the acid test will be what happens to this post. If the first one was removed via manual intervention for whatever reason, then this one surely will be too.
• codr4life 10y ago
  Hacker News is nothing but a censored echo chamber, pretending otherwise at this point is pure ignorance. Anything that doesn't fit the narrative will be beaten down or removed.
  • icebraining 10y ago
    If it's ignorance, then it's not pretense :) In any case, I don't necessarily disagree, but I've yet to see good evidence of the shadowy cabal, rather than user-directed flagging. I mean, just look at the list in the article: does it point to any kind of "beating down stuff that doesn't fit the narrative"?
    • nol13 10y ago
      I mean at some level, completely understandable/expected that the community at news.ycombinator.com will have the corporate/VC dick up its ass to a degree. And not always a bad thing, even if I think this place could use a GNAA troll every now and then. I don't need 10 cynical articles about big tech every morning. Those views need to be heard but at some point it's not interesting to me, that's not the stuff that keeps me refreshing HN while I should be programming.
  • torrent-of-ions 10y ago
    Including this very comment.
  • kdamken 10y ago
    It's sad but true. Everyone here likes to think they're an open minded intellectual, but when you see completely reasonable comments get downvoted for not fitting the narrative, it's hard to believe that.
• qznc 10y ago
  Afaik there is a mechanism to remove flame bait articles. If a link gets too many comments too quickly, it gets removed.
  • turc1656 10y ago
    If that's true, that seems like the sort of thing that would lead to a ton of false positives getting flagged for removal. I'm sure big news events like the Wikileaks Vault7 release would generate lots of comments. Doing a quick search on HN, I see the top posts related to Vault7 have <15 points and no more than single digit comments. I find this worrisome. Vault7 is the type of thing that I would expect to generate significant discussion here.
    • jsnell 10y ago
      It's been covered to death here. There were at least half a dozen posts on that in the last week that made over 200 points (the main one at 2697). https://news.ycombinator.com/item?id=13810015 https://news.ycombinator.com/item?id=13810015 https://news.ycombinator.com/item?id=13824032 https://news.ycombinator.com/item?id=13824032 https://news.ycombinator.com/item?id=13844389 https://news.ycombinator.com/item?id=13844389 https://news.ycombinator.com/item?id=13811914 https://news.ycombinator.com/item?id=13811914 https://news.ycombinator.com/item?id=13813160 https://news.ycombinator.com/item?id=13813160 https://news.ycombinator.com/item?id=13840544 https://news.ycombinator.com/item?id=13840544
• 65827 10y ago
  Was it critical of Google or one of the other mega corps? I've noticed oftentimes info that is disappears quickly on here.
• Paul-ish 10y ago
  Could it be that moderators see things we can't, eg vote manipulation and astro-turfing?
  • raverbashing 10y ago
    Possible and very likely
  • lightedman 10y ago
    More likely Paul Graham being his usual self: https://jcs.org/notaweblog/2012/06/13/hellbanned_from_hacker_news/ https://jcs.org/notaweblog/2012/06/13/hellbanned_from_hacker...
• brudgers 10y ago
  A useful data point missing from the article would have been the moderator's response to an email inquiring as to the story's history.
• timthelion 10y ago
  I've had one post removed from the top spot on the front page: https://news.ycombinator.com/item?id=5576041 https://news.ycombinator.com/item?id=5576041 I origionally posted with the title "For a moment, I thought bing was down" or something (I don't remember the origional title). The title was later changed to: Title: "Bing doesn't support SSL" https://news.ycombinator.com/item?id=5576041 https://news.ycombinator.com/item?id=5576041 Later, the story was was removed entirely after I wrote the following comment: " Actually, it's been like this a really long time. I just noticed, that HN stories which have nondescript titles fare better, so I decided to conduct a little experiment. 1st spot on the front page seems to confirm my hypothesis. " https://news.ycombinator.com/item?id=5576342 https://news.ycombinator.com/item?id=5576342 I certainly understand why the mods removed the "story", but at the same time, I felt that the discussion of the "non-descript title bias" would have been an interesting one to have.
  • timthelion 10y ago
    It seems like Hacker News doesn't like hackers, given the downvotes.
• anotheryou 10y ago
  Does anyone know why some posts have a rel="nofollow" in the link? (not just these no-comment ycombinator promotional posts) I asked this before and a mod said I should ask again via mail, but never got a response from hn@ycombinator.com.
  • gus_massa 10y ago
    At some time, the post with few upvotes (perhaps < 5 or something) had the rel="nofollow" attribute to discourage spammers to get a "follow" until the post was flagged. I'm not sure if that is still the current criteria or if they are using some additional signal to enable/disable the "nofollow" attribute. Probably the mods don't want to disclose the complete criteria, because it may change constantly without warning. Try to send again an email again, but I guess you will get in the reply only a general idea of the system. If you see something horribly misclassified, try sending an email to the mods.
    • anotheryou 10y ago
      Thanks, that's reasonable and makes sense!
• ifdefdebug 10y ago
  The list of stories the author claims "have moderator fingerprints on them" does not seem to have any kind of common pattern at all which would support such a claim. Edit: to be clear, with common pattern I mean the topics of the submission (obviously they have one common pattern, which is dropping out of the front page quickly). They do not reveal some secret agenda moderators might follow or something like that.
• johnlbevan2 10y ago
  Other theories: - This is a bug / happens randomly; you just noticed it because you were looking (i.e. as you analyse this data); all the posts it's happened to before and since went unnoticed. That's supported by the evidence of your analysis; most of the results don't look any different to other posts. - It's not the link, but the related activity. Presumably if you're running analysis on HN data, there are a lot of HN requests coming from your machine. Maybe any posts made by your IP are therefore treated as suspect (i.e. the sort of protection you'd expect to avoid automated posting or upvoting... just without that extra sophistication). Perhaps the other posters had something similar... Would be good to see if any of those posts were by the same author; as that may add weight to this theory. - Other variables... Maybe the algorithm has rules which cause this behaviour under some conditions; e.g. posts made the previous day (not 24 hours ago; but rather before midnight UTC / something like that) lose weight when midnight hits; so posts made moments before suddenly lose enough score to knock them off the top spot; whilst those which had more score before midnight, or were posted just after survive... Many other possibilities such as this may exist; and we'd only know by looking at those variables in the data... What else is common about the posts which are in your post's club vs those which aren't?
  • foob 10y ago
    Good points! I use the official Hacker News API [1] and my requests come from an ip address that is completely disconnected from my personal account. Even if the API usage were a red flag, there would be no way to automatically connect it to me. You're definitely right about there being miscellanious rules in there. Something that I mentioned in passing in the article is that many stories exhibit a significant drop in position once they're 15 hours old. If you look closely at the typical story trajectories you can also see various other jumps of about 10-30 positions which I would guess are triggered by these various rules. The stories listed in the article exhibit very different behavior where they jump hundreds of positions instantaneously. It's absolutely possible that this is triggered by some automatic mechanism but if that's the case then there's an enormous amount of signifance being assigned to the corresponding rules. If there's some random component to the ranking then I highly doubt that it would be responsible for jumps of this magnitude. I try to emphasize in the article that I do think it's possible that there's a hidden flagging threshold that's responsible and that the data can't tell us with certainty whether or not that's the case. I just personally find it unlikely that that's what happened for all of these stories. If you ran a site like Hacker News then would you put an admin link next to each post that pushes it off of the front page? I know that I would. [1] - https://github.com/HackerNews/API https://github.com/HackerNews/API
• Sean1708 10y ago
  I don't think I understand your figures correctly, because none of them (except the last) seem to show any significant drop. There's a drop of about 10 positions in one of them but I wouldn't call that particularly significant, and it even climbs back up after twenty minutes or so.
  • foob 10y ago
    Sorry if I didn't make this clear enough in the article but all of the stories in "Suppressed Story Trajectories" either drop off by hundreds of positions or disappear completely after their plotted line ends. They all match the criteria laid out in the previous paragraph. I used a larger y-scale on the last figure to show the full trajectory because that was a special case where the story didn't disappear completely and the trajectory after the jump was relevant.
• yAnonymous 10y ago
  The main problem I have with this is that similar to Reddit, HN is advertised as open-minded, free-talking place by the team themselves. When you then secretly censor stuff, because it doesn't fit your agenda, be it politically or financially, it makes you look even more like a hypocrite. Branding yourself liberal while employing fascist methods (censoring and banning) seems to be a trend, not only on the internet.
  • antocv 10y ago
    Ive noticed a lack of Assange, Wikileaks and Snowden discussions up in here. CIA Vault 7, like 1 thread or something, yeah right.
• aaronchall 10y ago
  It seems to me that the stories (all headlines read as controversial, as do the texts that I dug into) are removed due to flagging by users. It would seem to me that if you're looking to grind your political axe, this is not the best place to do so.
• jonathanstrange 10y ago
  I'm pretty happy with the way HackerNews handles posts and can only recommend to them to aggressively moderate in the future, too. There are way to many toxic users, trolls, shills, astro-turfers, voting rings, paid advertising, political organisations, disinformation campaigns, and other 'special interest' parties on the Net to be able to do without strong moderation.
  • lightedman 10y ago
    "There are way to many toxic users, trolls, shills, astro-turfers, voting rings, paid advertising, political organisations, disinformation campaigns, and other 'special interest' parties on the Net to be able to do without strong moderation." That you worry about these things on the internet instead of real life (like the ones in our government) is rather telling.
• najajomo 10y ago
  The rules here do seem to be petty and arbitrary, modding people down and banning accounts because the management object to the opinions expressed therein.
  • dang 10y ago
    If you're going to accuse us of something that awful you ought to supply links so (a) we can answer and (b) people can make up their own minds.
• topbagsvk 10y ago
  Would be good to see if any of those posts were by the same author - http://www.handbagsyou.com/replica-handbags.html http://www.handbagsyou.com/replica-handbags.html
• threepipeproblm 10y ago
  >> That’s about 2.1% so it’s not a particularly common occurrence but it is happening on a daily basis. I would say that 1/50 front page stories being buried is particularly common.
• golemotron 10y ago
  I predict that one of the biggest issues in tech over the next few years will be 'silent moderation.' Tech companies like to present the illusion that it is all 'just an algorithm' but that is deceptive. Silent curation and other practices like shadow-bannning are unethical and symptomatic of a mentality that seeks to avoid confrontation. If things go well we'll see more transparency over time. A good start for a site like HN would be to create another page that shows just the titles of the submissions rejected (no links). People can google for those titles if they are interested.
• falcolas 10y ago
  Obligatory pimping of a tool I use (not mine, though): http://hckrnews.com/ http://hckrnews.com/ Shows stories which have hit the front page ever, in the order of their posting. If it's currently on the front page, the link is orange. If it's not, it's black. It's very interesting to watch how frequently highly upvoted and commented posts turn black, while their temporal peers remain. Anecdotally, there appears to be trend of positive/neutral news about YC companies remaining on the front page the longest, the latest shiny technology sticks around for awhile longer than average, and pretty much any negative news disappears almost instantly. For example, as of this instant in time, there is an article about Angular2 which remains on the front page while more highly upvoted and commented articles about laptop security, AT&T discrimination, and a Nintendo Switch CVE discussion are all gone from the front page.
  • onewaystreet 10y ago
    > For example, as of this instant in time, there is an article about Angular2 which remains on the front page while more highly upvoted and commented articles about laptop security For front page rankings time matters more than total votes. A submission with 10 votes can be ranked higher than one with 50 if those 10 votes came in minutes rather than over the course of a day. It's not surprising that controversial submissions that get less up-votes (and more flags) don't get ranked as high.
    • stymaar 10y ago
      > For front page rankings time matters more than total votes Yes, but it's not the only thing at stake because you often see more recent and more up-voted posts being below older and less up-voted ones.
    • falcolas 10y ago
      The three articles I mentioned were all newer than the Angular article.
  • 91bananas 10y ago
    This is super cool. Thanks.
• wookoouk 10y ago
  My universities firewall blocked the link :(
• Jerry2 10y ago
  Interesting. Another aspect of HN that I've noticed are stories that are not censored but actually promoted. Some time ago, I've noticed that some stories were few hours old and had 4 votes yet they were in top 15 position on HN. I never understood how that's possible without some kind of manipulation. I've collected some of these anomalies. Peruse them and analyze them in this album: https://imgur.com/a/6OvnE https://imgur.com/a/6OvnE Maybe OP can find a pattern in these.
  • gus_massa 10y ago
    Some stories are cherrypicked by the mods to be shown for a short time in the front page. It's like a second chance for some good submissions. Old explanation by dang: https://news.ycombinator.com/item?id=10705926 https://news.ycombinator.com/item?id=10705926
• fidla 10y ago
  I think it was removed because of the blatant linking to your site, considered spam by most of us
  • minimaxir 10y ago
    Hacker News encourages original content as long as it is high quality/people aren't spamming it every other day.
  • aaron695 10y ago
    HN supports self promotion as long as it's on topic.
• j_s 10y ago
  http://hnrankings.info http://hnrankings.info is a nice UI to see this happening in real time. Here is an example of three submissions, two flagged enough to get kicked off the front page (for poor use of sources on a contentious topic) but not get marked as 'flagged': http://hnrankings.info/13714928,13714987,13715552/ http://hnrankings.info/13714928,13714987,13715552/
• grimmdude 10y ago
  Good writeup. Ironic that this story is #1 on the front page now.
• mirimir 10y ago
  Conversely, I've wondered how some stories (e.g., https://news.ycombinator.com/item?id=13857880 https://news.ycombinator.com/item?id=13857880) hit the top of the front page with only a few points. Edit: Thanks all. I get it now :)
  • Neliquat 10y ago
    I have a pet theory that lisp posts are weighted higher on here.
  • c3534l 10y ago
    Protip: every post hits the front page. Some posts just stay there longer.
    • mirimir 10y ago
      OK, I'll look at the algorithm :)
    • phpnode 10y ago
      * if it receives at least 4 upvotes before falling off the "new" page.
    • gus_massa 10y ago
      Not every story. But some stories are cherrypicked by the mods to be shown for a short time in the front page. It's like a second chance for some good submissions. Old explanation by dang: https://news.ycombinator.com/item?id=10705926 https://news.ycombinator.com/item?id=10705926
  • DashRattlesnake 10y ago
    My gut feel is posts like that are very new and have gotten a few upvotes in a short time.
• turc1656 10y ago
  I would like to see a monthly version of this post, similar to the recurring "Who's Hiring" that happens every month.
• c3534l 10y ago
  There's a subreddit for pages that get deleted from their front page. There's often good discussion around those pages and it's usually for a good reason they were removed. But sometimes there does appear to be real bias. It'd be nice if I could see what sort of stuff is getting removed and be able to discuss why.
• jacquesm 10y ago
  Nice article. Not so nice to rip the HN comments and display them below the article.
• danso 10y ago
  I didn't find this particular OP to be as enlightening as its current upvote count suggests, but the previous post that was supposedly disappeared was very interesting and well-written. Considering how many upvotes this OP gotten, the purported short-shrift the previous post got was fortuitous ;)
• dang 10y ago
  The story the OP is complaining about was flagged by users. Moderators never saw it (edit: wrong, we put 2010 on the title by mistake, see downthread [1]). Had we seen it, we would have turned off the flags. There's a long tradition of people looking at HN data and posting about it. Edit #2: since the 2010 thing was our mistake (an accident of sleep deprivation by the looks of it!) I've invited foob to repost the original article using the second-chance mechanism described at https://news.ycombinator.com/item?id=11662380 https://news.ycombinator.com/item?id=11662380 and the links there. I think he's planning to do that tomorrow. The [flagged] annotation only shows up on stories that are heavily flagged, i.e. enough to kill the post. User flags have downweighting effects long before that. Story rank on HN is determined by upvotes, flags, software, and moderators. Moderators downweight stories to the degree that they don't fit the site guidelines. This doesn't happen by upvotes alone, unfortunately; certain stories routinely get tons of upvotes regardless of how good they are for HN—e.g. anything sensational, indignant, or meta. If we didn't have a compensating factor, those stories would dominate the front page every day and HN would no longer follow its primary rule: "anything that gratifies one's intellectual curiosity". Of course that means HN is subject to our interpretation of what counts as "intellectual curiosity". HN has always worked that way; before we did it, pg did, and he trained us to do it so it would work as before. There's no way around the need for moderator intervention on a site like HN—the clue is in the word 'moderator' itself: left to its own devices the system runs to extremes and it needs a negative feedback loop to dampen it. When YC is involved, we do this less than usual as a matter of principle. When HN itself is involved it's a little bit different, because the hypnotic power of all things meta causes HN upvoters to go into an upvoting trance. Meta is basically crack, so we routinely downweight such posts—but only so much, to compensate for the crack effect. That's what I've done here, which is why the post is now at #7 rather than #1. It should probably be lower, but I want to illustrate the point that we intervene less, not more, when judgments about ourselves are involved. As a further example, a moderator actually turned off software penalties and user flags on this post this morning, which is probably why it went to #1 in the first place. That's more than I would have done but it shows how seriously we take that principle. None of this is new information, btw. I've posted about it plenty over the years and am always happy to answer questions. https://hn.algolia.com/?sort=byDate&prefix&page=0&dateRange=all&type=comment&query=by:dang%20downweight https://hn.algolia.com/?sort=byDate&prefix&page=0&dateRange=... https://hn.algolia.com/?sort=byDate&prefix&page=0&dateRange=all&type=comment&query=by:dang%20countervail https://hn.algolia.com/?sort=byDate&prefix&page=0&dateRange=... https://hn.algolia.com/?sort=byDate&prefix&page=0&dateRange=all&type=comment&query=by:dang%20constitutional https://hn.algolia.com/?sort=byDate&prefix&page=0&dateRange=... https://hn.algolia.com/?sort=byPopularity&prefix&page=0&dateRange=all&type=comment&query=by:dang%20curate https://hn.algolia.com/?sort=byPopularity&prefix&page=0&date... 1. https://news.ycombinator.com/item?id=13858850 https://news.ycombinator.com/item?id=13858850
• pvaldes 10y ago
  Some of those links are closed for voting for some reason. It seems that you can upwote the rest (including this "Stories that HN removes..." thread) but the vote count will not change anyway. Maybe is just a bug, maybe not. I don't know. Updated: Probably just a bandwidth issue. After 20 minutes the vote count is changing again.
• aaronhoffman 10y ago
  @foob, thought you might be interested in this "visual front page" of HN https://www.sizzleanalytics.com/HackerNews https://www.sizzleanalytics.com/HackerNews
• probably_wrong 10y ago
  Apparently I'm going to go against the flow to say that I'm perfectly fine with the way HN is moderated. In fact, I think the moderation is probably the reason why I come here. If magazines, TV shows, and journals have editors, it seems completely natural to me that a quality website should have one too. More often than not, keeping a community from turning into 4chan requires some heavy moderation (reddit's AskHistorians comes to mind, with entire threads nuked at once), and it's often a thankless job. I'm happy that HN managed to keep it's overall spirit, and I thank the mod team for that.
• Sir_Cmpwn 10y ago
  A few months ago I wrote a tool that scrapes HN and detects moderator activity (and provides interesting stats in general). I posted it twice and it didn't receive much attention, and the database storage requirements starting to get out of hand, so I killed it. It seems like an appropriate time to bring it up again. The source code is here: https://github.com/SirCmpwn/hn-transparency https://github.com/SirCmpwn/hn-transparency Some screenshots: https://sr.ht/WBEt.png https://sr.ht/WBEt.png https://sr.ht/30Tv.png https://sr.ht/30Tv.png If there's interest, I can put this back up and start pruning old data so it's more maintainable. The data I collected shows a lot of questionable moderator activity and a lot of abuse of flagging. I'm also unhappy with HN sending all comments on paywalled posts (which are against the rules) to /dev/null, when they're usually at least willing to talk about things.
  • nkurz 10y ago
    I wrote a tool that scrapes HN and detects moderator activity (and provides interesting stats in general). Sounds great! It might be good to distinguish between "moderator activity" (which to me implies that a human reads and makes a decision) and "moderation activity" (which covers automatic adjustments such as the frequently faulty "flame detection algorithm"). If there's interest, I can put this back up and start pruning old data so it's more maintainable. Yes, I think it would be very useful to have this available.
    • Sir_Cmpwn 10y ago
      >Sounds great! It might be good to distinguish between "moderator activity" (which to me implies that a human reads and makes a decision) and "moderation activity" (which covers automatic adjustments such as the frequently faulty "flame detection algorithm"). Unfortunately this information isn't made available. More analysis of the data is necessary to identify trends that can be used to distingish these.
• crablar 10y ago
  This is why we are working on Mineranker, an open source newsfeed and ranking platform: https://github.com/francisypl/mineranker https://github.com/francisypl/mineranker If you are interested in more open newsfeed ranking systems, check it out.
• kens 10y ago
  I wrote in detail in 2013 about how the Hacker News algorithm works and the penalties that can drop stories from the front page. My analysis was based on reverse engineering the algorithm from observed behavior and comparing with the published Arc code. This latest analysis seems kind of reinventing the wheel. http://www.righto.com/2013/11/how-hacker-news-ranking-really-works.html http://www.righto.com/2013/11/how-hacker-news-ranking-really... Interestingly, my 2013 article also suddenly dropped off the front page. Apparently it somehow triggered "voting ring detection" and was penalized. (I'm not part of a voting ring of course.)
• UhUhUhUh 10y ago
  Interesting discussion, and therefore, logically, interesting post. Oddly, I begin to wonder whether a variation on a bet system wouldn't be useful. Flagging or up-voting don't come with any cost attached. The only current cost I can see is related to pissing off a group or sub-group, which is not conducive to productive exchange. Just a thought from someone who has never flagged anything.
• euphoria83 10y ago
  I had, in the past, found that a post was deleted. I was so enraged by that action that I had abandoned HN for almost a year. It wasn't my post. But, I found the action arbitrary. I do understand that it is important to keep the spirit of HN and actively discourage posts that might take HN the same way as infinite other internet forums. Since the guidelines cannot be clearly interpreted, there will always be some controversy about what should or should not have been removed. This case is different though, because the post was down-voted out by the community.
• kristopolous 10y ago
  I had something that was removed from the front page of reddit and hn nearly simultaneously in 2011 - and I hadn't posted it on either, it was just my content. I saw it in the logs; some giant cliff. I've always wondered if there's cross collaboration since then.
• omouse 10y ago
  Mention "unions" or "professionalism" or anything that involves giving the actual value that a developer produces to the developer and the story will disappear very quickly. Maybe that's because of the mods or maybe it's because of the flagging by readers who have the pro-state-capitalist argument so ingrained in them. Whatever it is, it biases Hacker News in a bad way.
• pooptasticpoop 10y ago
  I'm glad I saw this here. HN has some of the strangest draconian moderation of any website i've ever visited. Though, i'm continually driven back here because of the insanely high quality of the comments here.
• leot 10y ago
  I've been wondering what happened to this story, which was doing better than most but didn't appear anywhere on first 15 hn pages. https://news.ycombinator.com/item?id=12557020 https://news.ycombinator.com/item?id=12557020 https://twitter.com/trottiest/status/779036903517491200 https://twitter.com/trottiest/status/779036903517491200
  • dang 10y ago
    It set off the HN flamewar detector. That's a software penalty that kicks in when discussions get overheated relative to upvotes. We eventually turned that penalty off on it.
    • leot 10y ago
      Interesting. Thanks!
• chmaynard 10y ago
  I still browse HN daily, but I lost confidence in the moderators after I posted an article that trended rapidly with an interesting, useful discussion and then suddenly dropped off the front page. I asked the moderators why this happened. Their explanation was that the article I posted was a duplicate, and therefore created a distraction for readers who wanted to comment on new material. This struck me as total bullshit, but I tried to be constructive and proposed a method of merging multiple threads on the same article. I never got a response.