3 ms·

I'd argue that one broad crawl every 2-3 months in addition to their updated-daily news crawl[0] should be good enough to make a rudimentary search engine. [0]

• mdaniel 4y ago
  You're right about the "rudimentary" part, because I don't know how they do it but the major players have some not-kidding-around freshness: https://www.google.com/search?hl=en&q=%22thrill%20me%22%20%22common%20crawl%22%20site%3Anews.ycombinator.com&tbs=qdr%3Ad https://www.google.com/search?hl=en&q=%22thrill%20me%22%20%2... https://www.bing.com/search?q=%22thrill+me%22+%22common+crawl%22+site%3Anews.ycombinator.com https://www.bing.com/search?q=%22thrill+me%22+%22common+craw... (and DDG similarly, because bing) ed: I was curious if maybe HN publishes a sitemap, and it seems no. Then again, hnreplies knows about the HN API so maybe it's special-cased by the big crawlers https://github.com/ggerganov/hnreplies#hnreplies https://github.com/ggerganov/hnreplies#hnreplies
  • jjeaff 4y ago
    I think it is as simple as an algorithm that rates how frequently new updates show up vs how many people visit a site. If you rarely make updates to your site, Google crawls it infrequently and new things won't show up very quickly. But if you do have frequent updates and lots of traffic, like any popular forum style site, you will get lots of crawler traffic. And I would bet the algo does the same for all endpoints on a site. So the "about us" page on a popular site probably ends up not being crawled nearly as much as the new threads page.