3 ms·
Interestingly, Craigslist is being consistent when it comes to not allowing scrapers, including Google. http://www.craigslist.org/robots.txt http://www.craigsl
by nostromo 14y ago
Interestingly, Craigslist is being consistent when it comes to not allowing scrapers, including Google.
http://www.craigslist.org/robots.txt http://www.craigslist.org/robots.txt
edit: oops! they're not actually blocking google from the apartment listings. thanks smackfu.
- smackfu 14y agoAren't they only disallowing particular pages there? Like apartments are at *.craigslist.org/apa/number.html which doesn't seem blocked by that robots.txt.