3 ms·
Well, typically you need consent from the service being used. If you look at how the "robots.txt" standard works, consent is assumed even for basic web crawling
by skybrian 7y ago
Well, typically you need consent from the service being used.
If you look at how the "robots.txt" standard works, consent is assumed even for basic web crawling by search engines. (At least for legitimate businesses; many crawlers do ignore robots.txt.)
- AstralStorm 7y agoThat document should not be considered legally binding. It is not a contract. Robots.txt is not even really a standard, best it was is a draft, set to expire on January 2nd next year. The purpose of the file was to preserve performance, not to hide anything. It targets crawlers not API calls. Consent of a non-single person business entity takes months at the very least, if it ever happens. It's easier for them to deny (incl. lawyers) and intimidate than compete and handle support tickets. Welcome to capitalism.
- skybrian 7y agoIn practice, the crawlers that read robots.txt don't do adversarial interoperability. If a website doesn't want to be in Google's index, they can opt out, and no lawyers are needed to do that.