3 ms·

You brought up robots.txt as if it was illegal to scrape or crawl the artist's website since his site had one that was supposed to "turn off scraping." That's w

You brought up robots.txt as if it was illegal to scrape or crawl the artist's website since his site had one that was supposed to "turn off scraping." That's what I was referring to. And that's just flat out laughably wrong. In the context you're now talking about by the letter of copyright law, downloading a photo, image, file of something or other, a "copyrighted work" where it's publicly accessible, and has a particular license specified that you may not reproduce it, may technically be "unlawful" letter by letter of the law, but I doubt any judge is going to actually see it that way versus intent of the law unless you literally reproduce or share a substantial portion of it, sell a complete copy, etc. It's almost certainly fair use to study, and use portions of the copyrighted work in your own copyrighted works.

Retric 3y ago
The legality of scrapping without explicit permission depends on the context as has been demonstrated by multiple court cases. Robots.txt short circuits that by giving permission to people who might not otherwise qualify. As to making a copy with intent to X, that’s what fair use is. A student may photocopy the full text of an short article so they can accurate quote it in their term paper. They can’t simply photocopy an article because they are a student with access to the article and a photocopier nor can the photocopy a full book because they want to use a short quote in their paper. Copying incidental to acceptable use becomes retroactively acceptable. This distinction may seem crazy to you, but the fact that intent matters means you can’t judge an action without context. Fair use in commercial context is looked at with vastly more suspicion than fair use in academic context, which again demonstrates specific actions on their own aren’t always enough information to say if something is allowed.
- srslack 3y ago
  >The legality of scrapping without explicit permission depends on the context as has been demonstrated by multiple court cases. Robots.txt short circuits that by giving permission to people who might not otherwise qualify. It's been determined that it's legal, if it's publicly accessible, and you don't receive a cease-and-desist letter telling you to stop. If it's public, that's permission. Simply "turning off scraping" by putting a robots.txt there doesn't make the content and linked images of a public web page any less public and restricted from being scraped, legally. Just last year, the LinkedIn case: LinkedIn had a robots.txt, and the judge didn't give a fuck. Nor did he care what their terms of use said. Rather, it was hiQ's continued scraping of LinkedIn data even after LinkedIn's cease-and-desist letter to them that constituted access of data "without authorization." >As to making a copy with intent to X, that’s what fair use is. Yes, and? >This distinction may seem crazy to you It's not a crazy distinction, that's what I'd basically said previously, so perhaps we're talking past each other.
  - Retric 3y ago
    > LinkedIn There’s a lot of precedent here showing scrapping isn’t guaranteed to be acceptable. https://en.wikipedia.org/wiki/Facebook,_Inc._v._Power_Ventures,_Inc https://en.wikipedia.org/wiki/Facebook,_Inc._v._Power_Ventur.... $79,640.50 in compensatory damages + $39,796.73 discovery sanction I could go on, but I am not sure what exactly you’re trying to argue here.
    - srslack 3y ago
      Power Ventures' actions violated the CFAA because they bypassed security measures intended to make the content not-exactly-public. The judge dismissed claims of copyright infringement despite them hosting "cached" versions of the scraped profiles using Facebook's trade dress. The damages + discovery sanction had to do with them bypassing security restrictions with their scraping, creating profiles and using bots to scrape with those profiles to access further information than was public, and Power Ventures' ignoring their explicit cease-and-desist the first time, and their non-compliance with discovery in some context. Read the case. >I could go on, but I am not sure what exactly you’re trying to argue here. That public is public. If you leave the door open in the real world, someone CAN enter your home. If you host your image on a public webpage, they can scrape it. robots.txt is not a security measure, nor is it a contract that magically gives the right to scrape where it wasn't given previously, it is a gentleman's agreement that you can ignore if you want to be a dick about it, and know about the robots.txt. Ethically, that's wrong, but it is how it is. Not to mention, as I came to understand while reading during this discussion, LAION wasn't even crawling: they were using a public commoncrawl dump to gather their images. commoncrawl had crawled the author's site previously. They just took that data and got image links out of it. 1. they weren't selling a dataset 2. the artist didn't "disable" scraping in any meaningful way, legally 3. linking to the image is not illegal, and they're justified to respond with an invoice in Germany to recover legal fees for this dumb copyright complaint 4. it may fall under fair use to download images and train neural nets on them, it may not be. it always depends on the context and the specific case. pretty simple stuff.
      - Retric 3y ago

Retric 3y ago

The legality of scrapping without explicit permission depends on the context as has been demonstrated by multiple court cases. Robots.txt short circuits that by giving permission to people who might not otherwise qualify. As to making a copy with intent to X, that’s what fair use is. A student may photocopy the full text of an short article so they can accurate quote it in their term paper. They can’t simply photocopy an article because they are a student with access to the article and a photocopier nor can the photocopy a full book because they want to use a short quote in their paper. Copying incidental to acceptable use becomes retroactively acceptable. This distinction may seem crazy to you, but the fact that intent matters means you can’t judge an action without context. Fair use in commercial context is looked at with vastly more suspicion than fair use in academic context, which again demonstrates specific actions on their own aren’t always enough information to say if something is allowed.

srslack 3y ago
>The legality of scrapping without explicit permission depends on the context as has been demonstrated by multiple court cases. Robots.txt short circuits that by giving permission to people who might not otherwise qualify. It's been determined that it's legal, if it's publicly accessible, and you don't receive a cease-and-desist letter telling you to stop. If it's public, that's permission. Simply "turning off scraping" by putting a robots.txt there doesn't make the content and linked images of a public web page any less public and restricted from being scraped, legally. Just last year, the LinkedIn case: LinkedIn had a robots.txt, and the judge didn't give a fuck. Nor did he care what their terms of use said. Rather, it was hiQ's continued scraping of LinkedIn data even after LinkedIn's cease-and-desist letter to them that constituted access of data "without authorization." >As to making a copy with intent to X, that’s what fair use is. Yes, and? >This distinction may seem crazy to you It's not a crazy distinction, that's what I'd basically said previously, so perhaps we're talking past each other.
- Retric 3y ago
  > LinkedIn There’s a lot of precedent here showing scrapping isn’t guaranteed to be acceptable. https://en.wikipedia.org/wiki/Facebook,_Inc._v._Power_Ventures,_Inc https://en.wikipedia.org/wiki/Facebook,_Inc._v._Power_Ventur.... $79,640.50 in compensatory damages + $39,796.73 discovery sanction I could go on, but I am not sure what exactly you’re trying to argue here.
  - srslack 3y ago
    Power Ventures' actions violated the CFAA because they bypassed security measures intended to make the content not-exactly-public. The judge dismissed claims of copyright infringement despite them hosting "cached" versions of the scraped profiles using Facebook's trade dress. The damages + discovery sanction had to do with them bypassing security restrictions with their scraping, creating profiles and using bots to scrape with those profiles to access further information than was public, and Power Ventures' ignoring their explicit cease-and-desist the first time, and their non-compliance with discovery in some context. Read the case. >I could go on, but I am not sure what exactly you’re trying to argue here. That public is public. If you leave the door open in the real world, someone CAN enter your home. If you host your image on a public webpage, they can scrape it. robots.txt is not a security measure, nor is it a contract that magically gives the right to scrape where it wasn't given previously, it is a gentleman's agreement that you can ignore if you want to be a dick about it, and know about the robots.txt. Ethically, that's wrong, but it is how it is. Not to mention, as I came to understand while reading during this discussion, LAION wasn't even crawling: they were using a public commoncrawl dump to gather their images. commoncrawl had crawled the author's site previously. They just took that data and got image links out of it. 1. they weren't selling a dataset 2. the artist didn't "disable" scraping in any meaningful way, legally 3. linking to the image is not illegal, and they're justified to respond with an invoice in Germany to recover legal fees for this dumb copyright complaint 4. it may fall under fair use to download images and train neural nets on them, it may not be. it always depends on the context and the specific case. pretty simple stuff.
    - Retric 3y ago