4 ms·

Another way of stating it: now that worthless text and images can be made to be worth money people no longer want to have public websites and are trying to chan

Another way of stating it: now that worthless text and images can be made to be worth money people no longer want to have public websites and are trying to change the longstanding culture of an open remixable web to fit their new paranoia over not getting their cut. I welcome all useragents and have been delighted to see openai, anthropic, microsoft, and other AI related crawlers mirroring my public website(s). Just the same as when I see googlebot or bingbot or firefox using humans. For information that I don't want to be public I simply don't put it on public websites.

hn_throwaway_99 2y ago
Another way of stating it: how dare all those greedy web publishers want to eat! /s The first couple decades of the web were build on an implicit promise: publishers could put their content out on freely accessible websites, and search engines would direct traffic to those sites. It was a mutually beneficial arrangement: publishers got the benefit of traffic that they could monetize in different ways if they saw fit (even hosting ads from the search engines a la AdSense), and Google etc. could earn mountains in AdWords revenue. I'm not saying it was always a fair tradeoff on both sides, but both sides had an incentive to share content openly. AI breaks that model. Publishers create all the content, but then the big search engines and AI companies can answer user questions without giving any reference at all to the original sites (companies like Google are providing source citations, but I guarantee the click-through rates go WAY down). This breakdown in the web's economic model has been happening for a while even before AI (e.g. with Google hosting more and more "informational content" directly on SERP pages - see https://news.ycombinator.com/item?id=24105465 https://news.ycombinator.com/item?id=24105465), but with AI the breakdown is really complete. So no, I don't fault people at all who don't want all the toils of their labor to be sucked up by trillion dollar megacorps.
- nickpsecurity 2y ago
  That’s their expectation but not the original design. The design of HTTP is that, if you run a web server, you are willing to send content to anyone that sends you a HTTP request. Anyone putting content online with a system like that should expect the data to go out to as many as request it. Alternative systems, like paywalls with prominent terms and conditions, exist to protect copyrighted content. They intentionally avoid them. So, they’re using a system designed for unrestricted distribution to publish content they hope people will use in restricted ways. I think the default, reasonable expectation should be that anyone can scrape public data for at least personal use.
  - probably_wrong 2y ago
    > they’re using a system designed for unrestricted distribution to publish content they hope people will use in restricted ways HTTP is not designed for unrestricted distribution - if that were the case we wouldn't have the HTTP status codes 401 (Unauthorized), 402 (Payment required) and 403 (Forbidden).
    - nickpsecurity 2y ago
      Good point. I forgot about those codes. So, the expectation would be that the content is available so long as HTTP sends it without such a code.
  - hn_throwaway_99 2y ago
    I don't really disagree with you that, if I understand you correctly, it's not really possible to make web content open for some purposes but not for other, and I commented as much here, https://news.ycombinator.com/item?id=41418603 https://news.ycombinator.com/item?id=41418603. But there is currently a lot of content that is published openly solely because the pre-AI economic model of the web made it viable. That economic model is now going away, so you'll see a lot more content put behind paywalls (or just not published at all) because AI means that it no longer makes sense for some publishers to spend time and money to produce content they can't get a return on.
croes 2y ago
>For information that I don't want to be public I simply don't put it on public websites. And there is no middle ground? I want others to read it but not to sell it?

hn_throwaway_99 2y ago

Another way of stating it: how dare all those greedy web publishers want to eat! /s The first couple decades of the web were build on an implicit promise: publishers could put their content out on freely accessible websites, and search engines would direct traffic to those sites. It was a mutually beneficial arrangement: publishers got the benefit of traffic that they could monetize in different ways if they saw fit (even hosting ads from the search engines a la AdSense), and Google etc. could earn mountains in AdWords revenue. I'm not saying it was always a fair tradeoff on both sides, but both sides had an incentive to share content openly. AI breaks that model. Publishers create all the content, but then the big search engines and AI companies can answer user questions without giving any reference at all to the original sites (companies like Google are providing source citations, but I guarantee the click-through rates go WAY down). This breakdown in the web's economic model has been happening for a while even before AI (e.g. with Google hosting more and more "informational content" directly on SERP pages - see https://news.ycombinator.com/item?id=24105465 https://news.ycombinator.com/item?id=24105465), but with AI the breakdown is really complete. So no, I don't fault people at all who don't want all the toils of their labor to be sucked up by trillion dollar megacorps.

nickpsecurity 2y ago
That’s their expectation but not the original design. The design of HTTP is that, if you run a web server, you are willing to send content to anyone that sends you a HTTP request. Anyone putting content online with a system like that should expect the data to go out to as many as request it. Alternative systems, like paywalls with prominent terms and conditions, exist to protect copyrighted content. They intentionally avoid them. So, they’re using a system designed for unrestricted distribution to publish content they hope people will use in restricted ways. I think the default, reasonable expectation should be that anyone can scrape public data for at least personal use.
- probably_wrong 2y ago
  > they’re using a system designed for unrestricted distribution to publish content they hope people will use in restricted ways HTTP is not designed for unrestricted distribution - if that were the case we wouldn't have the HTTP status codes 401 (Unauthorized), 402 (Payment required) and 403 (Forbidden).
  - nickpsecurity 2y ago
    Good point. I forgot about those codes. So, the expectation would be that the content is available so long as HTTP sends it without such a code.
- hn_throwaway_99 2y ago
  I don't really disagree with you that, if I understand you correctly, it's not really possible to make web content open for some purposes but not for other, and I commented as much here, https://news.ycombinator.com/item?id=41418603 https://news.ycombinator.com/item?id=41418603. But there is currently a lot of content that is published openly solely because the pre-AI economic model of the web made it viable. That economic model is now going away, so you'll see a lot more content put behind paywalls (or just not published at all) because AI means that it no longer makes sense for some publishers to spend time and money to produce content they can't get a return on.

croes 2y ago

>For information that I don't want to be public I simply don't put it on public websites. And there is no middle ground? I want others to read it but not to sell it?