3 ms·
Honoring robots.txt isn't a legal requirement, and I don't think a crawler can read or agree to TOS. Personally I'm in favour of anything publicly accessible be
by nly 13y ago
Honoring robots.txt isn't a legal requirement, and I don't think a crawler can read or agree to TOS. Personally I'm in favour of anything publicly accessible being fair-game for indexing purposes, but this kind of news about database access makes me uncomfortable.
- gojomo 13y agoRe: honoring robots.txt isn't a legal requirement In common-law jurisdictions (like the US), I wouldn't be so sure of that. It has a lot of precedent behind it, as a longstanding convention for indicating site-owner/rightsholder wishes. Ignoring it – deploying software designed or configured to be oblivious to it – could create legal risk. A literal reading of copyright law and laws about 'authorized' use of computer systems would assess all bulk copying/reuse of web content without explicit advance permission as illegal. It's the force of traditional/reasonable industry practices (like robots.txt), and offsetting considerations like fair-use, that make it legally defensible.