2 ms·
Google, and I presume other big search engines too, cache robots.txt for a week by default. We're well within the window for them to still be indexing CL.
by mooism2 14y ago
Google, and I presume other big search engines too, cache robots.txt for a week by default. We're well within the window for them to still be indexing CL.
- sigmadelta 14y agoIf that's true, how does 3taps know that CL is blocking search engines? Also: $ wget -q -O- --save-headers http://www.craigslist.org/robots.txt http://www.craigslist.org/robots.txt | fgrep Last-Modified Last-Modified: Fri, 04 Nov 2011 18:13:24 GMT