4 ms·
Craigslist doesn't needs to be removed from the search index. They just need to prevent caching of posts, which is what 3Taps was abusing. Noarchive in robots
by bluesnowmonkey 14y ago
Craigslist doesn't needs to be removed from the search index. They just need to prevent caching of posts, which is what 3Taps was abusing. Noarchive in robots.txt ought to take care of that.
- IanDrake 14y agoTo your knowledge, is it against the law to not follow a site's robots.txt? I'm not aware of one, but I am curious.
- debacle 14y agoI don't know if it's illegal, but Google would follow the site's robots.txt.
- BenOfTomorrow 14y agoAmbiguous. See http://news.ycombinator.com/item?id=4286857 http://news.ycombinator.com/item?id=4286857.