2 ms·

Had a conversation with a firm that wanted a distributed scraper built, and they really did not care about site usage policies. You would be fooling yourselves

Had a conversation with a firm that wanted a distributed scraper built, and they really did not care about site usage policies. You would be fooling yourselves if you think such a firm cared about robots.txt or page tags. We warned them they would be sued eventually, to contact the site owners for legal access to the data, and issued a hard pass on the project. Probably they assumed if the indexing process was out of another jurisdiction their domestic firm wouldn't be liable for theft of service or copyright infringement. It was my understanding AI/ML does not change legal obligations in business, but the firm probably found someone to build that dubious project eventually... Spider traps and rate-limiting are good options too. =3

reaperman 2y ago
Robots.txt doesnt create a legal obligation. It’s just a set of rules saying “if you don’t follow these rules to politely crawl our site, we’ll block you from crawling our site”. Obviously “anything goes” in civil suits however - if someone is being absurdly egregious with their crawling there’s usually some exposure to one tort or another.
- Joel_Mckay 2y ago
  The posted site access/usage policy is legally enforceable in most jurisdictions as far as I know... And Reddit has definitely become more proactive about scrapers. ;-)
  - echoangle 2y ago
    Legally enforceable as in “you can block people who don’t follow the policy” or enforceable as in “you can sue them for money”?
    - Joel_Mckay 2y ago
      If I recall it is considered theft-of-service if you bypass the posted site usage terms with an agent like a spider, and certainly a copyright violation for unauthorized content usage (especially in the context of a commercial venture.) One may be sued, but not because you parsed robots.txt wrong =3
      - reaperman 2y ago
        > it is considered theft-of-service if you bypass the posted site usage terms My understanding is that this is not accurate. HiQ v LinkedIn established that this is only the case if you actually agreed to the terms of service. Such "agreement" only happens if the information is walled behind an account creation process, e.g. Facebook, Inc. v. Power Ventures, Inc. If it's just scraping publicly available webpages, the only legal issue with scraping would be unreasonably or obviously negligent scraping practices which lead to degradation or denial-of-service. And obviously the line for that would have to be determined in civil court. eBay v. Bidder's Edge (2000) is the last case that I could find which even considered violation of robots.txt as very minor part of the judgement, but the findings were based far more on other things. Intel Corp. v. Hamidi also implicitly overruled the judgement in that that ruling (though not related to robots.txt, which was really just a very minor point in the first place).
        Joel_Mckay 2y ago
        Hard to say, I seem to recall it was because some spider authors used session cookies to bypass the EULA (the page probe auto-clicks "I agree" to capture the session cookie), and faked user agent strings to spoof gogglebot to gain access to site content. One thing is for certain, is its jurisdictional... and way too messy to be responsible for maintaining/hosting (the ambiguous copyright protection outside a research context looked way too risky.) =3
  - reaperman 2y ago
    Please review HiQ vs. LinkedIn - it hinged on the fact that HiQ hired crowdsourced workers (“turkers”) to create fake profiles through which to access LinkedIn’s platform (who had to agree to the ToS to create these accounts). The court found that hiQ expressly agreed to the user agreement when it created its corporate account on LinkedIn’s platform. This doesn't apply if you don't ever agree to anything - which is the case if the information is not locked behind account creation.
    - Joel_Mckay 2y ago
      This gets complex fast, as a click-army is not necessarily violating the EULA. However, if they scraped the content using these account credentials, than it becomes a problem in a commercial context. If I recall, only journalists and academics could argue Fair use at that point. Anyway, I didn't touch the project mostly for copyright and trademark risk concerns. Have a great day =3
Atotalnoob 2y ago
AI/ML has created a race to suck all the data on the internet regardless of copyright or status and use it. OpenAI introduced gptbot in August 2023… they already took everything
- Joel_Mckay 2y ago
  Unlikely, site-generator hosts are still happily providing a limitless supply of remixed well-structured nonsense, random images with noise, and valid links to popular sites. In this case, they showed up to the data buffet long after it went rotten due to SEO. Have a nice day =3

reaperman 2y ago

Robots.txt doesnt create a legal obligation. It’s just a set of rules saying “if you don’t follow these rules to politely crawl our site, we’ll block you from crawling our site”. Obviously “anything goes” in civil suits however - if someone is being absurdly egregious with their crawling there’s usually some exposure to one tort or another.

Joel_Mckay 2y ago
The posted site access/usage policy is legally enforceable in most jurisdictions as far as I know... And Reddit has definitely become more proactive about scrapers. ;-)
- echoangle 2y ago
  Legally enforceable as in “you can block people who don’t follow the policy” or enforceable as in “you can sue them for money”?
  - Joel_Mckay 2y ago
    If I recall it is considered theft-of-service if you bypass the posted site usage terms with an agent like a spider, and certainly a copyright violation for unauthorized content usage (especially in the context of a commercial venture.) One may be sued, but not because you parsed robots.txt wrong =3
    - reaperman 2y ago
      > it is considered theft-of-service if you bypass the posted site usage terms My understanding is that this is not accurate. HiQ v LinkedIn established that this is only the case if you actually agreed to the terms of service. Such "agreement" only happens if the information is walled behind an account creation process, e.g. Facebook, Inc. v. Power Ventures, Inc. If it's just scraping publicly available webpages, the only legal issue with scraping would be unreasonably or obviously negligent scraping practices which lead to degradation or denial-of-service. And obviously the line for that would have to be determined in civil court. eBay v. Bidder's Edge (2000) is the last case that I could find which even considered violation of robots.txt as very minor part of the judgement, but the findings were based far more on other things. Intel Corp. v. Hamidi also implicitly overruled the judgement in that that ruling (though not related to robots.txt, which was really just a very minor point in the first place).
      - Joel_Mckay 2y ago
        Hard to say, I seem to recall it was because some spider authors used session cookies to bypass the EULA (the page probe auto-clicks "I agree" to capture the session cookie), and faked user agent strings to spoof gogglebot to gain access to site content. One thing is for certain, is its jurisdictional... and way too messy to be responsible for maintaining/hosting (the ambiguous copyright protection outside a research context looked way too risky.) =3
- reaperman 2y ago
  Please review HiQ vs. LinkedIn - it hinged on the fact that HiQ hired crowdsourced workers (“turkers”) to create fake profiles through which to access LinkedIn’s platform (who had to agree to the ToS to create these accounts). The court found that hiQ expressly agreed to the user agreement when it created its corporate account on LinkedIn’s platform. This doesn't apply if you don't ever agree to anything - which is the case if the information is not locked behind account creation.
  - Joel_Mckay 2y ago
    This gets complex fast, as a click-army is not necessarily violating the EULA. However, if they scraped the content using these account credentials, than it becomes a problem in a commercial context. If I recall, only journalists and academics could argue Fair use at that point. Anyway, I didn't touch the project mostly for copyright and trademark risk concerns. Have a great day =3

Atotalnoob 2y ago

AI/ML has created a race to suck all the data on the internet regardless of copyright or status and use it. OpenAI introduced gptbot in August 2023… they already took everything

Joel_Mckay 2y ago
Unlikely, site-generator hosts are still happily providing a limitless supply of remixed well-structured nonsense, random images with noise, and valid links to popular sites. In this case, they showed up to the data buffet long after it went rotten due to SEO. Have a nice day =3