4 ms·

2. robots.txt is not legally enforceable. It's a gentleman's agreement, a widely accepted standard to codify into a scraper for it to observe and respect. 3. T

2. robots.txt is not legally enforceable. It's a gentleman's agreement, a widely accepted standard to codify into a scraper for it to observe and respect. 3. That's why it's called an opinion. No shit. Maybe instead of talking about promoting terrorism, you should have a level head and talk about fair use.

Retric 3y ago
Robots.txt is permission it allows you to do something that is otherwise illegal. You are not allowed to do this stuff by default. Something people don’t understand is “Right click file save as…” on a copyrighted image breaks copyright, as you don’t have permission to make a copy. The do have implied permission to make incidental copies to view a website, but that’s it.
- srslack 3y ago
  Are you saying that after all these years of omitting robots.txt from my public personal website of priceless blog posts and code repositories, I can take Google and Microsoft to court for scraping, linking to, and reproducing what I deem as substantial portions? They've violated my rights! They didn't have permission! In all seriousness, that's total shit. It's publicly accessible. That's the permission. There have been different cases like weev where the judge has viewed it differently because of specific details like carrying out the action of enumeration, or guessing IDs, to reveal non-public information, but that had to do with the CFAA. If you reproduce content in total in another work, resell, etc., that's another thing.
  - Retric 3y ago
    > It's publicly accessible. That's the permission. Fair use is permission. Every library book is accessible that doesn’t have copyright implications.
    - srslack 3y ago
      You brought up robots.txt as if it was illegal to scrape or crawl the artist's website since his site had one that was supposed to "turn off scraping." That's what I was referring to. And that's just flat out laughably wrong. In the context you're now talking about by the letter of copyright law, downloading a photo, image, file of something or other, a "copyrighted work" where it's publicly accessible, and has a particular license specified that you may not reproduce it, may technically be "unlawful" letter by letter of the law, but I doubt any judge is going to actually see it that way versus intent of the law unless you literally reproduce or share a substantial portion of it, sell a complete copy, etc. It's almost certainly fair use to study, and use portions of the copyrighted work in your own copyrighted works.
      - Retric 3y ago
        The legality of scrapping without explicit permission depends on the context as has been demonstrated by multiple court cases. Robots.txt short circuits that by giving permission to people who might not otherwise qualify. As to making a copy with intent to X, that’s what fair use is. A student may photocopy the full text of an short article so they can accurate quote it in their term paper. They can’t simply photocopy an article because they are a student with access to the article and a photocopier nor can the photocopy a full book because they want to use a short quote in their paper. Copying incidental to acceptable use becomes retroactively acceptable. This distinction may seem crazy to you, but the fact that intent matters means you can’t judge an action without context. Fair use in commercial context is looked at with vastly more suspicion than fair use in academic context, which again demonstrates specific actions on their own aren’t always enough information to say if something is allowed.
        srslack 3y ago
        >The legality of scrapping without explicit permission depends on the context as has been demonstrated by multiple court cases. Robots.txt short circuits that by giving permission to people who might not otherwise qualify. It's been determined that it's legal, if it's publicly accessible, and you don't receive a cease-and-desist letter telling you to stop. If it's public, that's permission. Simply "turning off scraping" by putting a robots.txt there doesn't make the content and linked images of a public web page any less public and restricted from being scraped, legally. Just last year, the LinkedIn case: LinkedIn had a robots.txt, and the judge didn't give a fuck. Nor did he care what their terms of use said. Rather, it was hiQ's continued scraping of LinkedIn data even after LinkedIn's cease-and-desist letter to them that constituted access of data "without authorization." >As to making a copy with intent to X, that’s what fair use is. Yes, and? >This distinction may seem crazy to you It's not a crazy distinction, that's what I'd basically said previously, so perhaps we're talking past each other.

Retric 3y ago

Robots.txt is permission it allows you to do something that is otherwise illegal. You are not allowed to do this stuff by default. Something people don’t understand is “Right click file save as…” on a copyrighted image breaks copyright, as you don’t have permission to make a copy. The do have implied permission to make incidental copies to view a website, but that’s it.

srslack 3y ago
Are you saying that after all these years of omitting robots.txt from my public personal website of priceless blog posts and code repositories, I can take Google and Microsoft to court for scraping, linking to, and reproducing what I deem as substantial portions? They've violated my rights! They didn't have permission! In all seriousness, that's total shit. It's publicly accessible. That's the permission. There have been different cases like weev where the judge has viewed it differently because of specific details like carrying out the action of enumeration, or guessing IDs, to reveal non-public information, but that had to do with the CFAA. If you reproduce content in total in another work, resell, etc., that's another thing.
- Retric 3y ago
  > It's publicly accessible. That's the permission. Fair use is permission. Every library book is accessible that doesn’t have copyright implications.
  - srslack 3y ago
    You brought up robots.txt as if it was illegal to scrape or crawl the artist's website since his site had one that was supposed to "turn off scraping." That's what I was referring to. And that's just flat out laughably wrong. In the context you're now talking about by the letter of copyright law, downloading a photo, image, file of something or other, a "copyrighted work" where it's publicly accessible, and has a particular license specified that you may not reproduce it, may technically be "unlawful" letter by letter of the law, but I doubt any judge is going to actually see it that way versus intent of the law unless you literally reproduce or share a substantial portion of it, sell a complete copy, etc. It's almost certainly fair use to study, and use portions of the copyrighted work in your own copyrighted works.
    - Retric 3y ago
      The legality of scrapping without explicit permission depends on the context as has been demonstrated by multiple court cases. Robots.txt short circuits that by giving permission to people who might not otherwise qualify. As to making a copy with intent to X, that’s what fair use is. A student may photocopy the full text of an short article so they can accurate quote it in their term paper. They can’t simply photocopy an article because they are a student with access to the article and a photocopier nor can the photocopy a full book because they want to use a short quote in their paper. Copying incidental to acceptable use becomes retroactively acceptable. This distinction may seem crazy to you, but the fact that intent matters means you can’t judge an action without context. Fair use in commercial context is looked at with vastly more suspicion than fair use in academic context, which again demonstrates specific actions on their own aren’t always enough information to say if something is allowed.
      - srslack 3y ago
        >The legality of scrapping without explicit permission depends on the context as has been demonstrated by multiple court cases. Robots.txt short circuits that by giving permission to people who might not otherwise qualify. It's been determined that it's legal, if it's publicly accessible, and you don't receive a cease-and-desist letter telling you to stop. If it's public, that's permission. Simply "turning off scraping" by putting a robots.txt there doesn't make the content and linked images of a public web page any less public and restricted from being scraped, legally. Just last year, the LinkedIn case: LinkedIn had a robots.txt, and the judge didn't give a fuck. Nor did he care what their terms of use said. Rather, it was hiQ's continued scraping of LinkedIn data even after LinkedIn's cease-and-desist letter to them that constituted access of data "without authorization." >As to making a copy with intent to X, that’s what fair use is. Yes, and? >This distinction may seem crazy to you It's not a crazy distinction, that's what I'd basically said previously, so perhaps we're talking past each other.