3 ms·

> Sites start to close up behind more anti-copy protection or literal paywalls to make things not-so-freely accessible. And that ends the free internet as we kn

by Ukv 3y ago

> Sites start to close up behind more anti-copy protection or literal paywalls to make things not-so-freely accessible. And that ends the free internet as we know it. We are indeed starting to see pieces of this happen in real time Sites have been pushing towards "to continue reading, download our app" or "you've reached your free article limit, subscribe for just $X" for a long time, because it's profitable. I don't think there's sense in retroactively placing the blame for this behavior on crawlers - it's a tactic designed for humans, to the extent that it's often implemented in ways entirely ineffective against bots such as putting an element in front of the content, using cookies to track viewed article count, or even intentionally letting bot user-agents through the paywall. Sites are capable of restricting GPTBot/CCBot/etc., or putting a rate limit on crawling, without impacting human readers. > You can reguritate the source with enough prompting. This seems to imply that some server out there is in fact just storing hoards of source material on itself. If that server is commercialized and has TBs of copyright material, that seems to go past the fair use argument of "limited use". The model isn't retrieving content from some storage server during inference if that's what you're thinking, but consider Google Books which does actually do that - storing a huge number of in-copyright books internally on Google's servers and retrieving snippets when searched. As found in Authors Guild v. Google, this is Fair Use - in particular: > > What matters in such cases is not so much "the amount and substantiality of the portion used" in making a copy, but rather the amount and substantiality of what is thereby made accessible to a public for which it may serve as a competing substitute.

johnnyanmac 3y ago
>Sites have been pushing towards "to continue reading, download our app" or "you've reached your free article limit, subscribe for just $X" for a long time, because it's profitable. I don't think there's sense in retroactively placing the blame for this behavior on crawlers And websites have been getting paid for a long time by selling user data (which again is only useful for human data, not robot data). I don't think it's a coincidence that the paywalling of more and more sites came as GDPR and other data privacy regulations started to pop up. Crawlers are one of those smaller symptoms that gets knocked out along the way as well. Can't make money one way, they switch to another (even before taking the pandemic and interests rates into account). >Sites are capable of restricting GPTBot/CCBot/etc., or putting a rate limit on crawling, without impacting human readers. all those captchas I'm being hit with these days tells me otherwise. Maybe it is possible, but perhaps too expensive. Or the capthas themselves are great for training data. I don't know the real reason, but the internet has definitely become most hostile towards my human eyes in an attempt to curb bots. And the worst part is that it doesn't even entirely block out the bots. >The model isn't retrieving content from some storage server during inference if that's what you're thinking I don't admit to know the full details, but some exploits discovered lately seem to suggest that this isn't just "reading a page" and then moving on: https://arxiv.org/pdf/2301.13188.pdf https://arxiv.org/pdf/2301.13188.pdf (PDF warning). Apparently somewhere in the memory there is those raw assets being stored for "reference". Again, I don't know enough to debate on it, but I've seen enough pieces to be scrutinous of how the process currently works. Scraping is a gray area, but storing copyright data in your commercial product is a much harder issue to argue. Youtube screws its creators over backwards to prevent litigation over a very similar concept.
- Ukv 3y ago
  > I don't think it's a coincidence that the paywalling of more and more sites came as GDPR and other data privacy regulations started to pop up Data protection regulations being the cause of paywalls seems a different theory than what you were previously suggesting. > all those captchas I'm being hit with these days tells me otherwise. Maybe it is possible, but perhaps too expensive. Captchas are generally to prevent spambots. CCBot/GPTBot crawlers identify themselves by their user-agent (so the server can just choose not to serve the content) and also respect robots.txt. > https://arxiv.org/pdf/2301.13188.pdf https://arxiv.org/pdf/2301.13188.pdf (PDF warning). Apparently somewhere in the memory there is those raw assets being stored for "reference". Stable Diffusion is ~4GB and trained on 5 billion images - for the most part it's not plausible for it to have stored the "raw assets". There is a real issue of overfitting on a subset of content that the model saw many times during training such that it was able to memorize details, but the model isn't retrieving the source material from some server to reference (that's known as retrieval augmented generation, which isn't used by default SD/GPT). Would also keep in mind that for papers like this, where they're generating hundreds of millions of images and digging into CLIP embeddings to construct prompts that intentionally recreate images, that there's some degree of similarity you'd expect just by the conditioning given, the large number of comparisons being automatically made, and coincidence - even had the model not seen the image in question. > Youtube screws its creators over backwards to prevent litigation over a very similar concept. Youtube makes the video available to the public. The Google Books analogy properly preserves the aspect that full works are stored but only small portions made available.
  - johnnyanmac 3y ago
    >CCBot/GPTBot crawlers identify themselves by their user-agent (so the server can just choose not to serve the content) and also respect robots.txt. Sure. I guess we'll see how that holds up in the lawsuit over the coming years. > for the most part it's not plausible for it to have stored the "raw assets" I wasn't talking as literal pngs on a server. I'm sure a company backed by Microsoft has much more clever ways to store such data. My main point was that it seems possible to reproduce the asset so enough of it is kept to perform such operations. >Would also keep in mind that for papers like this, where they're generating hundreds of millions of images and digging into CLIP embeddings to construct prompts that intentionally recreate images, that there's some degree of similarity you'd expect just by the conditioning given, the large number of comparisons being automatically made, and coincidence - even had the model not seen the image in question. coincidence, perhaps. I can't say and tracing isn't technically illegal either (although extremely faux pas for artists doing anything more than learning how to draw). But that is definitely where the metaphor of human motor skills and machine reproduction breaks down. Humans can more easily be cast off as coincidence based on context (which can be easy or hard to deduce. E. G. Did artist B follow artist A on Twitter? How long were they active? Are they known to have a similar style?) bots' context is closed up as company IP. That alone will give it a lot more scrutiny. Scrutiny that can potentiallybe subpeona's, so again: TBD. >Youtube makes the video available to the public. The Google Books analogy properly preserves the aspect that full works are stored but only small portions made available. I don't really know why you're so fixated on the Google books metaphor. Google books is also an actual bookstore as well as some place for users to upload their library. I can upload pirated content for private use becsuse Google isn't scanning all its customers 24/7. It's technically not legal but also hard to enforce on my private library of 1 that I share with no one. I'd run into more issues if I tried to sell it and someone caught me. You can DMCA a book from Google just as easily as a YouTube video. There's simply less money in the game to DMCA troll literature (and for many books of interest to HN, the primary customer isn't individuals but institutions and education).

by Ukv 3y ago

johnnyanmac 3y ago

>Sites have been pushing towards "to continue reading, download our app" or "you've reached your free article limit, subscribe for just $X" for a long time, because it's profitable. I don't think there's sense in retroactively placing the blame for this behavior on crawlers And websites have been getting paid for a long time by selling user data (which again is only useful for human data, not robot data). I don't think it's a coincidence that the paywalling of more and more sites came as GDPR and other data privacy regulations started to pop up. Crawlers are one of those smaller symptoms that gets knocked out along the way as well. Can't make money one way, they switch to another (even before taking the pandemic and interests rates into account). >Sites are capable of restricting GPTBot/CCBot/etc., or putting a rate limit on crawling, without impacting human readers. all those captchas I'm being hit with these days tells me otherwise. Maybe it is possible, but perhaps too expensive. Or the capthas themselves are great for training data. I don't know the real reason, but the internet has definitely become most hostile towards my human eyes in an attempt to curb bots. And the worst part is that it doesn't even entirely block out the bots. >The model isn't retrieving content from some storage server during inference if that's what you're thinking I don't admit to know the full details, but some exploits discovered lately seem to suggest that this isn't just "reading a page" and then moving on: https://arxiv.org/pdf/2301.13188.pdf https://arxiv.org/pdf/2301.13188.pdf (PDF warning). Apparently somewhere in the memory there is those raw assets being stored for "reference". Again, I don't know enough to debate on it, but I've seen enough pieces to be scrutinous of how the process currently works. Scraping is a gray area, but storing copyright data in your commercial product is a much harder issue to argue. Youtube screws its creators over backwards to prevent litigation over a very similar concept.

Ukv 3y ago
> I don't think it's a coincidence that the paywalling of more and more sites came as GDPR and other data privacy regulations started to pop up Data protection regulations being the cause of paywalls seems a different theory than what you were previously suggesting. > all those captchas I'm being hit with these days tells me otherwise. Maybe it is possible, but perhaps too expensive. Captchas are generally to prevent spambots. CCBot/GPTBot crawlers identify themselves by their user-agent (so the server can just choose not to serve the content) and also respect robots.txt. > https://arxiv.org/pdf/2301.13188.pdf https://arxiv.org/pdf/2301.13188.pdf (PDF warning). Apparently somewhere in the memory there is those raw assets being stored for "reference". Stable Diffusion is ~4GB and trained on 5 billion images - for the most part it's not plausible for it to have stored the "raw assets". There is a real issue of overfitting on a subset of content that the model saw many times during training such that it was able to memorize details, but the model isn't retrieving the source material from some server to reference (that's known as retrieval augmented generation, which isn't used by default SD/GPT). Would also keep in mind that for papers like this, where they're generating hundreds of millions of images and digging into CLIP embeddings to construct prompts that intentionally recreate images, that there's some degree of similarity you'd expect just by the conditioning given, the large number of comparisons being automatically made, and coincidence - even had the model not seen the image in question. > Youtube screws its creators over backwards to prevent litigation over a very similar concept. Youtube makes the video available to the public. The Google Books analogy properly preserves the aspect that full works are stored but only small portions made available.
- johnnyanmac 3y ago
  >CCBot/GPTBot crawlers identify themselves by their user-agent (so the server can just choose not to serve the content) and also respect robots.txt. Sure. I guess we'll see how that holds up in the lawsuit over the coming years. > for the most part it's not plausible for it to have stored the "raw assets" I wasn't talking as literal pngs on a server. I'm sure a company backed by Microsoft has much more clever ways to store such data. My main point was that it seems possible to reproduce the asset so enough of it is kept to perform such operations. >Would also keep in mind that for papers like this, where they're generating hundreds of millions of images and digging into CLIP embeddings to construct prompts that intentionally recreate images, that there's some degree of similarity you'd expect just by the conditioning given, the large number of comparisons being automatically made, and coincidence - even had the model not seen the image in question. coincidence, perhaps. I can't say and tracing isn't technically illegal either (although extremely faux pas for artists doing anything more than learning how to draw). But that is definitely where the metaphor of human motor skills and machine reproduction breaks down. Humans can more easily be cast off as coincidence based on context (which can be easy or hard to deduce. E. G. Did artist B follow artist A on Twitter? How long were they active? Are they known to have a similar style?) bots' context is closed up as company IP. That alone will give it a lot more scrutiny. Scrutiny that can potentiallybe subpeona's, so again: TBD. >Youtube makes the video available to the public. The Google Books analogy properly preserves the aspect that full works are stored but only small portions made available. I don't really know why you're so fixated on the Google books metaphor. Google books is also an actual bookstore as well as some place for users to upload their library. I can upload pirated content for private use becsuse Google isn't scanning all its customers 24/7. It's technically not legal but also hard to enforce on my private library of 1 that I share with no one. I'd run into more issues if I tried to sell it and someone caught me. You can DMCA a book from Google just as easily as a YouTube video. There's simply less money in the game to DMCA troll literature (and for many books of interest to HN, the primary customer isn't individuals but institutions and education).