5 ms·

Twitter.com/robots.txt # Crawl-delay: 10 -- Googlebot ignores crawl-delay ftl

• danielamitay 16y ago
  The most honest robots.txt file
• zzleeper 16y ago
  A quick question: What is "crawl-delay ftl". I googled but couldn't find much, besides that this particular robots.txt dates from April 2007 (~4 years ago) and the first mention of it was in 2008
  • arctangent 16y ago
    It may be an acronym for "For The Loss/Lose", which would be a corruption of "FTW" (=="For The Win"). They're saying that it sucks that Google doesn't respect rate-limiting with the crawl-delay parameter.
    • zzleeper 16y ago
      ty!