4 ms·

What is the reason for this?

• zitterbewegung 17y ago
  Probably to stop linkspam?
  • Frazzydee 17y ago
    That doesn't make sense. Why not just add rel=nofollow to all links?
  • deleted 17y ago
    [deleted]
  • jsrfded 17y ago
    It's unfortunate because I often use site: to find old threads on ycombinator. especially since news.ycombinator.com doesn't have its own search facility, external search engines are the old way to comb through old posts here.
    • johnfn 17y ago
      http://www.searchyc.com/ http://www.searchyc.com/ still exists, but it's up in the air if they will continue.
      • chengmi 17y ago
        Not unless pg replies to this with "Please stop".
  • mikecane 17y ago
    OK, thanks.
• drhodes 17y ago
  maybe to see what firms ignore robot.txt files?
• idoh 17y ago
  My bet is that it is for technical reasons. Whenever you see a reply link (among other things) the server has to persist a continuation for it. Getting all of those pages crawled must generate a massive number of continuations to persist. I know that this is one of the reasons that vote-links look the way they do. If that's not the reason, then my #2 guess is that getting the whole site crawled is moving too many comments into the cache. If I remember correctly, whenever an uncached comment gets accessed, it gets cached (basically adding it to a hash), and thus added to the in-memory ram. That gradually raises the memory requirements for running the app, until pg does a restart, which resets the cache as well.
  • blasdel 17y ago
    And the regular restarts invalidate every single continuation-based URL