4 ms·

Robots.txt doesnt create a legal obligation. It’s just a set of rules saying “if you don’t follow these rules to politely crawl our site, we’ll block you from c

Robots.txt doesnt create a legal obligation. It’s just a set of rules saying “if you don’t follow these rules to politely crawl our site, we’ll block you from crawling our site”. Obviously “anything goes” in civil suits however - if someone is being absurdly egregious with their crawling there’s usually some exposure to one tort or another.

Joel_Mckay 2y ago
The posted site access/usage policy is legally enforceable in most jurisdictions as far as I know... And Reddit has definitely become more proactive about scrapers. ;-)
- echoangle 2y ago
  Legally enforceable as in “you can block people who don’t follow the policy” or enforceable as in “you can sue them for money”?
  - Joel_Mckay 2y ago
    If I recall it is considered theft-of-service if you bypass the posted site usage terms with an agent like a spider, and certainly a copyright violation for unauthorized content usage (especially in the context of a commercial venture.) One may be sued, but not because you parsed robots.txt wrong =3
    - reaperman 2y ago
      > it is considered theft-of-service if you bypass the posted site usage terms My understanding is that this is not accurate. HiQ v LinkedIn established that this is only the case if you actually agreed to the terms of service. Such "agreement" only happens if the information is walled behind an account creation process, e.g. Facebook, Inc. v. Power Ventures, Inc. If it's just scraping publicly available webpages, the only legal issue with scraping would be unreasonably or obviously negligent scraping practices which lead to degradation or denial-of-service. And obviously the line for that would have to be determined in civil court. eBay v. Bidder's Edge (2000) is the last case that I could find which even considered violation of robots.txt as very minor part of the judgement, but the findings were based far more on other things. Intel Corp. v. Hamidi also implicitly overruled the judgement in that that ruling (though not related to robots.txt, which was really just a very minor point in the first place).
      - Joel_Mckay 2y ago
        Hard to say, I seem to recall it was because some spider authors used session cookies to bypass the EULA (the page probe auto-clicks "I agree" to capture the session cookie), and faked user agent strings to spoof gogglebot to gain access to site content. One thing is for certain, is its jurisdictional... and way too messy to be responsible for maintaining/hosting (the ambiguous copyright protection outside a research context looked way too risky.) =3
- reaperman 2y ago
  Please review HiQ vs. LinkedIn - it hinged on the fact that HiQ hired crowdsourced workers (“turkers”) to create fake profiles through which to access LinkedIn’s platform (who had to agree to the ToS to create these accounts). The court found that hiQ expressly agreed to the user agreement when it created its corporate account on LinkedIn’s platform. This doesn't apply if you don't ever agree to anything - which is the case if the information is not locked behind account creation.
  - Joel_Mckay 2y ago
    This gets complex fast, as a click-army is not necessarily violating the EULA. However, if they scraped the content using these account credentials, than it becomes a problem in a commercial context. If I recall, only journalists and academics could argue Fair use at that point. Anyway, I didn't touch the project mostly for copyright and trademark risk concerns. Have a great day =3

Joel_Mckay 2y ago

The posted site access/usage policy is legally enforceable in most jurisdictions as far as I know... And Reddit has definitely become more proactive about scrapers. ;-)

echoangle 2y ago
Legally enforceable as in “you can block people who don’t follow the policy” or enforceable as in “you can sue them for money”?
- Joel_Mckay 2y ago
  If I recall it is considered theft-of-service if you bypass the posted site usage terms with an agent like a spider, and certainly a copyright violation for unauthorized content usage (especially in the context of a commercial venture.) One may be sued, but not because you parsed robots.txt wrong =3
  - reaperman 2y ago
    > it is considered theft-of-service if you bypass the posted site usage terms My understanding is that this is not accurate. HiQ v LinkedIn established that this is only the case if you actually agreed to the terms of service. Such "agreement" only happens if the information is walled behind an account creation process, e.g. Facebook, Inc. v. Power Ventures, Inc. If it's just scraping publicly available webpages, the only legal issue with scraping would be unreasonably or obviously negligent scraping practices which lead to degradation or denial-of-service. And obviously the line for that would have to be determined in civil court. eBay v. Bidder's Edge (2000) is the last case that I could find which even considered violation of robots.txt as very minor part of the judgement, but the findings were based far more on other things. Intel Corp. v. Hamidi also implicitly overruled the judgement in that that ruling (though not related to robots.txt, which was really just a very minor point in the first place).
    - Joel_Mckay 2y ago
      Hard to say, I seem to recall it was because some spider authors used session cookies to bypass the EULA (the page probe auto-clicks "I agree" to capture the session cookie), and faked user agent strings to spoof gogglebot to gain access to site content. One thing is for certain, is its jurisdictional... and way too messy to be responsible for maintaining/hosting (the ambiguous copyright protection outside a research context looked way too risky.) =3
reaperman 2y ago
Please review HiQ vs. LinkedIn - it hinged on the fact that HiQ hired crowdsourced workers (“turkers”) to create fake profiles through which to access LinkedIn’s platform (who had to agree to the ToS to create these accounts). The court found that hiQ expressly agreed to the user agreement when it created its corporate account on LinkedIn’s platform. This doesn't apply if you don't ever agree to anything - which is the case if the information is not locked behind account creation.
- Joel_Mckay 2y ago
  This gets complex fast, as a click-army is not necessarily violating the EULA. However, if they scraped the content using these account credentials, than it becomes a problem in a commercial context. If I recall, only journalists and academics could argue Fair use at that point. Anyway, I didn't touch the project mostly for copyright and trademark risk concerns. Have a great day =3