4 ms·
A quick question: What is "crawl-delay ftl". I googled but couldn't find much, besides that this particular robots.txt dates from April 2007 (~4 years ago) and
by zzleeper 16y ago
A quick question: What is "crawl-delay ftl". I googled but couldn't find much, besides that this particular robots.txt dates from April 2007 (~4 years ago) and the first mention of it was in 2008
- arctangent 16y agoIt may be an acronym for "For The Loss/Lose", which would be a corruption of "FTW" (=="For The Win"). They're saying that it sucks that Google doesn't respect rate-limiting with the crawl-delay parameter.
- zzleeper 16y agoty!