3 ms·
I don't know. The pathological case could include a rapidly changing robots.txt. Think about archive.org's policy. If they suddenly find new restrictions on a d
by rhema 9y ago
I don't know. The pathological case could include a rapidly changing robots.txt. Think about archive.org's policy. If they suddenly find new restrictions on a domain, they hide it in their waybackmachine. Sometimes an old site will go down and be replaced by totally new owners. This breaks some domains of the waybackmachine retroactively.
- ikeboy 9y agoI think judges are able to ask some questions and tell the difference between an honest mistake and a flagrant disregard for robots.txt, if that were to be the legal standard.