3 ms·

I'd be really worried about some algorithm somewhere deciding to crawl less frequently even after the temporary ban has been lifted.

• jsrfded 17y ago
  Robot bans can render your archive.org records inaccessable, zap all your former code 200 urls out of the major search indices, and yes, kill your inclusion in the list of topical sites. Much better to set a temporary Crawl-Delay directive. Otherwise you're not just telling the engines to pause crawling you, you're telling them "take all of my pages out of your index."
  • Andys 17y ago
    What are the details of crawl-delay? If its supposed to be "number of seconds between hits", for some sites, 1 hit per second may as well be "not crawled at all"..
    • ComputerGuru 17y ago
      Right, but his point is that at least this way you won't get delisted :)