8 ms·

Reverse Engineering the Hacker News Ranking Algorithm

349 points by foob 10y ago

foob 10y ago
This is the article that was discussed in yesterday's "The stories that Hacker News removes from the front page" [1]. After speaking with @dang, it sounds like what happened with the original submission was that a moderator accidentally put "(2010)" in the title and users flagged it because they incorrectly thought it was old. He invited me to resubmit the article today to allow for real discussion and to demonstrate that what happened to the first submission was accidental. I know that this analysis will get less attention than the one from yesterday, but I personally find it far more interesting and hope that it can stand on its own merits. I'll be around to answer any questions that might come up. [1] - https://news.ycombinator.com/item?id=13857086 https://news.ycombinator.com/item?id=13857086
- newbie2017 10y ago
  > it sounds like what happened with the original submission was that a moderator accidentally put "(2010)" in the title and users flagged it because they incorrectly thought it was old mhm, I'm sure that's what really happened
  - dang 10y ago
    Which bit? The 2010 thing is precisely what happened. It was a case of sleep deprivation, which is one lesson of how trying too hard to make this place good can mess with a person. The other bit was just my attempt to explain why users might have flagged the post. User flags were what demoted its rank, and it isn't obvious why people flagged it. There's also the issue that meta posts aren't great for HN in the first place, but those rarely lack for upvotes.
    - saycheese 10y ago
      It would be nice if there was non-filtered view of HN available for users with a rep above 500, much like the "show dead" option for user comments that have been hidden. Basically allowing the submissions to be placed in the submission rank as if their ranking was not pinged due to users flagging the submission.
      - bradleyankrom 10y ago
        I like this idea, but it'd be nice to have it not based on karma (says a person that doesn't comment very often and has low karma).
        saycheese 10y ago
        Understand though I could easily see spammers using the data and getting 500 really should not take more than a month if you post a handful of comments a day; for example: 25 days, 5 comments a day, average of 4 upvotes per comment. Worth noting with a rep of 500+ HN users are able to downvote comments.
        bradleyankrom 10y ago
        Yeah, and I think the karma system works well overall. It isn't ideal for people that are significantly more consumer than producer of content, but I would guess that is the minority.
        tylorr 10y ago
        I feel there are a lot more consumers than you might imagine, myself included. I'm on HN constantly but I rarely ever comment. The karma system is not for everyone.
        DanBC 10y ago
        I think there are a few people with > 500 karma, more with an account but less karma, and even more without an account. Pareto principle and all.
    - deleted 10y ago
      [deleted]
    - rimliu 10y ago
      But this is not just some random meta post it's about ranking algorithms, I think those are doomed to be popular whether it is HN or Reddit.
    - scoot 10y ago
      It was asked by someone yesterday, but the question got lost in the noise, whether the voting ring detection extends to flagging rings (whether on posts or comments)? It would be naive to assume that it doesn't happen...
    - jacquesm 10y ago
      I suspect that in part this is because having HN's actual ranking algorithm as implemented today (including the secret sauce) would not benefit HN.
  - deleted 10y ago
    [deleted]
  - gthtjtkt 10y ago
    On the one hand, I'm also skeptical. On the other hand, I could see someone glancing at the date and accidentally reading "Mar 10, 2017" as Mar '10.
  - qeternity 10y ago
    Hanlon's razor: "Never attribute to malice that which is adequately explained by stupidity"
    - pcmaffey 10y ago
      Nice. Summarizes my longstanding argument with conspiracy theorists.
- saycheese 10y ago
  Being old is not a reason to flag a submission, at best appending the title with the year would be all that would be required.
  - deleted 10y ago
    [deleted]
  - mbrookes 10y ago
    Not by itself, but if the age suggest that the content is stale it could be. How the HN algorithm worked 7 years (assuming it has changed) isn't of that much interest, even though the analysis might be.
    - saycheese 10y ago
      Flags are for content that's spam, off-topic, inappropriate. While you might claim that the content is dated, it would likely be better to post a comment providing a more recent source for the topic, not upvote the submission, and allow the community to decide for themselves what the best source of information is on the topic.
      - Jach 10y ago
        Flags are downvotes, given the lack of an actual downvote, and hence have much broader uses. What they should be used for is irrelevant to what they're actually used for.
        saycheese 10y ago
        Realize they are used as downvotes which is why using them as part of the ranking system makes no sense. Adding a downvote button also makes no sense. If you see something you do not want to see, but others do want to see it, click "hide" and move on.
        Jach 10y ago
        That entirely depends on how you see yourself and other users as a member of the community. Do users have a way to influence what content the community has besides what they individually submit and upvote? Yes, flagging. Similarly with comments, we have more than one's own comments and upvotes, we have downvotes. Downvotes and flags both serve as useful signals to others what the community wants to discourage. Hiding / ignoring sends no signals, at least no different signals from never having seen the item at all.
- startupdiscuss 10y ago
  [innocent yet incisive comment removed due to excessive down-voting]
  - dang 10y ago
    That's... not how the internet works. But it doesn't matter; we invited foob to repost it as a way of owning our own mistake.
- Peroni 10y ago
  Hey foob. The level of detail in your post is great. Any plans to research things like optimal posting times?
  - foob 10y ago
    Thanks! That's a good question. I feel like I've seen analyses like that before a few times but I'm having trouble finding one right now. I would guess that submitting around 8 EST is probably best in terms of getting the most views because you'll catch most of the US audience during the day. The probability of making it to the front page is another question though and it would definitely require looking at the data there. I can actually think of some other relevant metrics here that I don't think I've seen quantified before. I'll probably play around with this a bit at some point and if the results are interesting then I'll write them up. If I do though then it will most likely be a few months down the line. I've already written twice as many HN meta articles as I was planning on and need to take a little break :-).
  - andrewstuart2 10y ago
    I did a brief analysis a few years ago after shoving the HN Algolia results into Postgres. IIRC, the optimal time (for highest median score) was on a Sunday afternoon or something like that. I figured that meant that you're not competing as much for views on Sunday afternoon and will get higher-on-average points. Then you may still be around on page 1 or 2 for the Monday morning rush and get a lot more traffic.
nstj 10y ago
Or you could just search for Paul Graham's posts[0] :) > (= gravity* 1.8 timebase* 120 front-threshold* 1 nourl-factor* .4 lightweight-factor* .17 gag-factor* .1) (def frontpage-rank (s (o scorefn realscore) (o gravity gravity*)) (* (/ (let base (- (scorefn s) 1) (if (> base 0) (expt base .8) base)) (expt (/ (+ (item-age s) timebase*) 60) gravity)) (if (no (in s!type 'story 'poll)) .8 (blank s!url) nourl-factor* (mem 'bury s!keys) .001 (* (contro-factor s) (if (mem 'gag s!keys) gag-factor* (lightweight s) lightweight-factor* 1))))) [0]: https://news.ycombinator.com/item?id=1781417 https://news.ycombinator.com/item?id=1781417
- overcast 10y ago
  Yeh I'm not sure why this is even a thing. I thought everyone knew that algorithm at this point? Hell, I'm using the same thing on a couple projects.
  - foob 10y ago
    From the article: I find this to be a particularly interesting question, not because I actually care about the answer, but because it feels like the data should be able to tell us the answer... My main goal was to tease out the ranking algorithm from the data in a simple and elegant fashion. This made it a little more interesting as an endeavor and hopefully makes it a more interesting read as well. You're right, the details of the algorithm aren't hugely interesting and are generally available. The point here was to use the data to uncover it in a somewhat novel way. Figuring things out can be fun in and of itself, even if the answers are already available.
- sillysaurus3 10y ago
  That's almost seven years out of date. The algorithm has changed significantly since then. Example: Flags affect a story's position, but that algorithm doesn't mention flags at all.
saycheese 10y ago
Anyone interested in the topic of HN's ranking algorithm should look through the HN's submission archives: https://hn.algolia.com/?query=How%20Hacker%20News%20ranking%20algorithm&sort=byPopularity&prefix&page=0&dateRange=all&type=story https://hn.algolia.com/?query=How%20Hacker%20News%20ranking%... For example: https://medium.com/hacking-and-gonzo/how-hacker-news-ranking-algorithm-works-1d9b0cf2c08d#.x86yaez0u https://medium.com/hacking-and-gonzo/how-hacker-news-ranking...
jkchu 10y ago
I wrote a piece a while back around implementing your own ranking algorithm. Thought this audience might find it relevant/interesting. https://jkchu.com/2016/02/17/designing-and-implementing-a-ranking-algorithm/ https://jkchu.com/2016/02/17/designing-and-implementing-a-ra...

349 points by foob 10y ago

foob 10y ago

This is the article that was discussed in yesterday's "The stories that Hacker News removes from the front page" [1]. After speaking with @dang, it sounds like what happened with the original submission was that a moderator accidentally put "(2010)" in the title and users flagged it because they incorrectly thought it was old. He invited me to resubmit the article today to allow for real discussion and to demonstrate that what happened to the first submission was accidental. I know that this analysis will get less attention than the one from yesterday, but I personally find it far more interesting and hope that it can stand on its own merits. I'll be around to answer any questions that might come up. [1] - https://news.ycombinator.com/item?id=13857086 https://news.ycombinator.com/item?id=13857086

newbie2017 10y ago
> it sounds like what happened with the original submission was that a moderator accidentally put "(2010)" in the title and users flagged it because they incorrectly thought it was old mhm, I'm sure that's what really happened
- dang 10y ago
  Which bit? The 2010 thing is precisely what happened. It was a case of sleep deprivation, which is one lesson of how trying too hard to make this place good can mess with a person. The other bit was just my attempt to explain why users might have flagged the post. User flags were what demoted its rank, and it isn't obvious why people flagged it. There's also the issue that meta posts aren't great for HN in the first place, but those rarely lack for upvotes.
  - saycheese 10y ago
    It would be nice if there was non-filtered view of HN available for users with a rep above 500, much like the "show dead" option for user comments that have been hidden. Basically allowing the submissions to be placed in the submission rank as if their ranking was not pinged due to users flagging the submission.
    - bradleyankrom 10y ago
      I like this idea, but it'd be nice to have it not based on karma (says a person that doesn't comment very often and has low karma).
      - saycheese 10y ago
        Understand though I could easily see spammers using the data and getting 500 really should not take more than a month if you post a handful of comments a day; for example: 25 days, 5 comments a day, average of 4 upvotes per comment. Worth noting with a rep of 500+ HN users are able to downvote comments.
        bradleyankrom 10y ago
        Yeah, and I think the karma system works well overall. It isn't ideal for people that are significantly more consumer than producer of content, but I would guess that is the minority.
        tylorr 10y ago
        I feel there are a lot more consumers than you might imagine, myself included. I'm on HN constantly but I rarely ever comment. The karma system is not for everyone.
        DanBC 10y ago
        I think there are a few people with > 500 karma, more with an account but less karma, and even more without an account. Pareto principle and all.
  - deleted 10y ago
    [deleted]
  - rimliu 10y ago
    But this is not just some random meta post it's about ranking algorithms, I think those are doomed to be popular whether it is HN or Reddit.
  - scoot 10y ago
    It was asked by someone yesterday, but the question got lost in the noise, whether the voting ring detection extends to flagging rings (whether on posts or comments)? It would be naive to assume that it doesn't happen...
  - jacquesm 10y ago
    I suspect that in part this is because having HN's actual ranking algorithm as implemented today (including the secret sauce) would not benefit HN.
- deleted 10y ago
  [deleted]
- gthtjtkt 10y ago
  On the one hand, I'm also skeptical. On the other hand, I could see someone glancing at the date and accidentally reading "Mar 10, 2017" as Mar '10.
- qeternity 10y ago
  Hanlon's razor: "Never attribute to malice that which is adequately explained by stupidity"
  - pcmaffey 10y ago
    Nice. Summarizes my longstanding argument with conspiracy theorists.
saycheese 10y ago
Being old is not a reason to flag a submission, at best appending the title with the year would be all that would be required.
- deleted 10y ago
  [deleted]
- mbrookes 10y ago
  Not by itself, but if the age suggest that the content is stale it could be. How the HN algorithm worked 7 years (assuming it has changed) isn't of that much interest, even though the analysis might be.
  - saycheese 10y ago
    Flags are for content that's spam, off-topic, inappropriate. While you might claim that the content is dated, it would likely be better to post a comment providing a more recent source for the topic, not upvote the submission, and allow the community to decide for themselves what the best source of information is on the topic.
    - Jach 10y ago
      Flags are downvotes, given the lack of an actual downvote, and hence have much broader uses. What they should be used for is irrelevant to what they're actually used for.
      - saycheese 10y ago
        Realize they are used as downvotes which is why using them as part of the ranking system makes no sense. Adding a downvote button also makes no sense. If you see something you do not want to see, but others do want to see it, click "hide" and move on.
        Jach 10y ago
        That entirely depends on how you see yourself and other users as a member of the community. Do users have a way to influence what content the community has besides what they individually submit and upvote? Yes, flagging. Similarly with comments, we have more than one's own comments and upvotes, we have downvotes. Downvotes and flags both serve as useful signals to others what the community wants to discourage. Hiding / ignoring sends no signals, at least no different signals from never having seen the item at all.
startupdiscuss 10y ago
[innocent yet incisive comment removed due to excessive down-voting]
- dang 10y ago
  That's... not how the internet works. But it doesn't matter; we invited foob to repost it as a way of owning our own mistake.
Peroni 10y ago
Hey foob. The level of detail in your post is great. Any plans to research things like optimal posting times?
- foob 10y ago
  Thanks! That's a good question. I feel like I've seen analyses like that before a few times but I'm having trouble finding one right now. I would guess that submitting around 8 EST is probably best in terms of getting the most views because you'll catch most of the US audience during the day. The probability of making it to the front page is another question though and it would definitely require looking at the data there. I can actually think of some other relevant metrics here that I don't think I've seen quantified before. I'll probably play around with this a bit at some point and if the results are interesting then I'll write them up. If I do though then it will most likely be a few months down the line. I've already written twice as many HN meta articles as I was planning on and need to take a little break :-).
- andrewstuart2 10y ago
  I did a brief analysis a few years ago after shoving the HN Algolia results into Postgres. IIRC, the optimal time (for highest median score) was on a Sunday afternoon or something like that. I figured that meant that you're not competing as much for views on Sunday afternoon and will get higher-on-average points. Then you may still be around on page 1 or 2 for the Monday morning rush and get a lot more traffic.

nstj 10y ago

Or you could just search for Paul Graham's posts[0] :) > (= gravity* 1.8 timebase* 120 front-threshold* 1 nourl-factor* .4 lightweight-factor* .17 gag-factor* .1) (def frontpage-rank (s (o scorefn realscore) (o gravity gravity*)) (* (/ (let base (- (scorefn s) 1) (if (> base 0) (expt base .8) base)) (expt (/ (+ (item-age s) timebase*) 60) gravity)) (if (no (in s!type 'story 'poll)) .8 (blank s!url) nourl-factor* (mem 'bury s!keys) .001 (* (contro-factor s) (if (mem 'gag s!keys) gag-factor* (lightweight s) lightweight-factor* 1))))) [0]: https://news.ycombinator.com/item?id=1781417 https://news.ycombinator.com/item?id=1781417

overcast 10y ago
Yeh I'm not sure why this is even a thing. I thought everyone knew that algorithm at this point? Hell, I'm using the same thing on a couple projects.
- foob 10y ago
  From the article: I find this to be a particularly interesting question, not because I actually care about the answer, but because it feels like the data should be able to tell us the answer... My main goal was to tease out the ranking algorithm from the data in a simple and elegant fashion. This made it a little more interesting as an endeavor and hopefully makes it a more interesting read as well. You're right, the details of the algorithm aren't hugely interesting and are generally available. The point here was to use the data to uncover it in a somewhat novel way. Figuring things out can be fun in and of itself, even if the answers are already available.
sillysaurus3 10y ago
That's almost seven years out of date. The algorithm has changed significantly since then. Example: Flags affect a story's position, but that algorithm doesn't mention flags at all.

saycheese 10y ago

Anyone interested in the topic of HN's ranking algorithm should look through the HN's submission archives: https://hn.algolia.com/?query=How%20Hacker%20News%20ranking%20algorithm&sort=byPopularity&prefix&page=0&dateRange=all&type=story https://hn.algolia.com/?query=How%20Hacker%20News%20ranking%... For example: https://medium.com/hacking-and-gonzo/how-hacker-news-ranking-algorithm-works-1d9b0cf2c08d#.x86yaez0u https://medium.com/hacking-and-gonzo/how-hacker-news-ranking...

jkchu 10y ago

I wrote a piece a while back around implementing your own ranking algorithm. Thought this audience might find it relevant/interesting. https://jkchu.com/2016/02/17/designing-and-implementing-a-ranking-algorithm/ https://jkchu.com/2016/02/17/designing-and-implementing-a-ra...