3 ms·
That really blows my mind. I mean, how can they say that's any kind of "agreement"? I someone writes a curl/wget script wrapper & points it to the top 10 webs
by johnvschmitt 13y ago
That really blows my mind. I mean, how can they say that's any kind of "agreement"?
I someone writes a curl/wget script wrapper & points it to the top 10 websites, they don't enter into any kind of written contract or agreement.
- MichaelApproved 13y agoYou're quoting "agreement" as if its literally in their robots file. It's not. They're telling the public that it does not have permission to crawl the site which try have the right to do. What is the problem with that?
- yeukhon 13y agoWhat is the point of having such silly prohibition? It's silly because anyone can crawl it if they want, Facebook may block such DDoS attack, but why would they bother to put up such sign when they know it's useless?
- sfall 13y agomy guess lawyers
- MichaelApproved 13y agoThey have such a large network, anything they could do to prevent unwanted crawling is probably helpful.
- declan 13y agoThe operator of a crawler doesn't need to sign an agreement for the prohibition to be enforceable. See eBay v. Bidder's Edge. This was 14 years ago, folks.
- usrusr 13y agoIt's probably just their way of explaining how those user-agents that do not get the catchall Disallow: / treatment got into that robots.txt file. Also, including some lawyerisms might be quite effective at reminding upstart scrapers that faking the googlebot UA would be even less cool than simply ignoring robots.txt.