3 ms·

Why? My site, my robots.txt - if the scrapers do not obey, then I'll block them.

kuschku 11y ago
No. Your site is public. If it is available via HTTP without authorization, I can scrape it (but not republish). There have been in many countries (except the US) court decisions about this. There are ways to force authorization, if you don’t use them, your fault. If I just use TOR or VPNs or similar to get around your block, also, again, your issue. Unless you require authorization, your service is, by law, public. And if I want to store the site offline for later reading, or read it now in a browser, is nothing that you have to decide.
- chinathrow 11y ago
  Well, copyright does prohibit redistributing the scraped content. And that is what all 3rd party content scraping services essentially do. That violates copyright law in quite a few countries. If you scrape with your own tools, own servers and use it only within your company, I would think it's a different legal matter. No, it's not "by law" public, or could you cite a few laws for that, both US and EU. Would appreciate that.
  - kuschku 11y ago
    It is public because the HTTP standard defines it as public, and, if you claim to use this standard, its definitions apply to you (if you claim to support a standard, you have to accept that its definitions will be used in court against you) HTTP directly says you should use authorization to make stuff non-public. And third-party content scraping tools are NOT violating copyright law, as they do not redistribute it freely. Otherwise services like Opera Turbo – which scrapes a website, removes cruft, compresses data, and sends it to your phone – would also break copyright law, and there is a nice exception for this case, under US law, it’s covered under fair use. (Unless the scraping service makes your content accessible publicly – and even then, fair use applies usually, as in the case of pure archival like archive.org or archive.is)
    - chinathrow 11y ago
      You can have your content blocked from archive.org also by robots.txt - didn't check archive.is Are you a lawyer? I am not - anyway, I can't really agree with that falls under fair use. If I charge money to scrape someones site and then hand over the material in whatever form my customer uses - I am bound run into problems sooner or later.

No. Your site is public. If it is available via HTTP without authorization, I can scrape it (but not republish). There have been in many countries (except the US) court decisions about this. There are ways to force authorization, if you don’t use them, your fault. If I just use TOR or VPNs or similar to get around your block, also, again, your issue. Unless you require authorization, your service is, by law, public. And if I want to store the site offline for later reading, or read it now in a browser, is nothing that you have to decide.

chinathrow 11y ago
Well, copyright does prohibit redistributing the scraped content. And that is what all 3rd party content scraping services essentially do. That violates copyright law in quite a few countries. If you scrape with your own tools, own servers and use it only within your company, I would think it's a different legal matter. No, it's not "by law" public, or could you cite a few laws for that, both US and EU. Would appreciate that.
- kuschku 11y ago
  It is public because the HTTP standard defines it as public, and, if you claim to use this standard, its definitions apply to you (if you claim to support a standard, you have to accept that its definitions will be used in court against you) HTTP directly says you should use authorization to make stuff non-public. And third-party content scraping tools are NOT violating copyright law, as they do not redistribute it freely. Otherwise services like Opera Turbo – which scrapes a website, removes cruft, compresses data, and sends it to your phone – would also break copyright law, and there is a nice exception for this case, under US law, it’s covered under fair use. (Unless the scraping service makes your content accessible publicly – and even then, fair use applies usually, as in the case of pure archival like archive.org or archive.is)
  - chinathrow 11y ago
    You can have your content blocked from archive.org also by robots.txt - didn't check archive.is Are you a lawyer? I am not - anyway, I can't really agree with that falls under fair use. If I charge money to scrape someones site and then hand over the material in whatever form my customer uses - I am bound run into problems sooner or later.