6 ms·
To your knowledge, is it against the law to not follow a site's robots.txt? I'm not aware of one, but I am curious.
by IanDrake 14y ago
To your knowledge, is it against the law to not follow a site's robots.txt? I'm not aware of one, but I am curious.
- debacle 14y agoI don't know if it's illegal, but Google would follow the site's robots.txt.
- BenOfTomorrow 14y agoAmbiguous. See http://news.ycombinator.com/item?id=4286857 http://news.ycombinator.com/item?id=4286857.