2 ms·

How do you decide the top 10 links of the day if you scrape every 5 minutes? (the top links keep changing). Couldn't you just scrape once a day?

• cperciva 16y ago
  The /news page ranks links based on score and time since submission. I'm only ranking links based on score (and whether it has been on a previous daily). I could scrape the entire site at midnight each day, but I think PG would be very unhappy with me if I did that. Scaping /news every 5 minutes imposes much less load, and since highly ranked links get almost all of their votes prior to falling off the front page, this gives me almost as much information.
  • revorad 16y ago
    I'm guessing that for your purpose, you might catch more interesting stories from the /classic page than /news. Do you mind sharing the scraping code?
    • cperciva 16y ago
      I'd prefer to not post the code publicly, simply because I don't want to encourage people to put extra load on the HN server -- but if you want a copy, send me an email and I'll provide it.
      • revorad 16y ago
        Thanks. Just emailed you.