3 ms·

That’s their expectation but not the original design. The design of HTTP is that, if you run a web server, you are willing to send content to anyone that sends

That’s their expectation but not the original design. The design of HTTP is that, if you run a web server, you are willing to send content to anyone that sends you a HTTP request. Anyone putting content online with a system like that should expect the data to go out to as many as request it. Alternative systems, like paywalls with prominent terms and conditions, exist to protect copyrighted content. They intentionally avoid them. So, they’re using a system designed for unrestricted distribution to publish content they hope people will use in restricted ways. I think the default, reasonable expectation should be that anyone can scrape public data for at least personal use.

probably_wrong 2y ago
> they’re using a system designed for unrestricted distribution to publish content they hope people will use in restricted ways HTTP is not designed for unrestricted distribution - if that were the case we wouldn't have the HTTP status codes 401 (Unauthorized), 402 (Payment required) and 403 (Forbidden).
- nickpsecurity 2y ago
  Good point. I forgot about those codes. So, the expectation would be that the content is available so long as HTTP sends it without such a code.
hn_throwaway_99 2y ago
I don't really disagree with you that, if I understand you correctly, it's not really possible to make web content open for some purposes but not for other, and I commented as much here, https://news.ycombinator.com/item?id=41418603 https://news.ycombinator.com/item?id=41418603. But there is currently a lot of content that is published openly solely because the pre-AI economic model of the web made it viable. That economic model is now going away, so you'll see a lot more content put behind paywalls (or just not published at all) because AI means that it no longer makes sense for some publishers to spend time and money to produce content they can't get a return on.

probably_wrong 2y ago

> they’re using a system designed for unrestricted distribution to publish content they hope people will use in restricted ways HTTP is not designed for unrestricted distribution - if that were the case we wouldn't have the HTTP status codes 401 (Unauthorized), 402 (Payment required) and 403 (Forbidden).

nickpsecurity 2y ago
Good point. I forgot about those codes. So, the expectation would be that the content is available so long as HTTP sends it without such a code.

hn_throwaway_99 2y ago

I don't really disagree with you that, if I understand you correctly, it's not really possible to make web content open for some purposes but not for other, and I commented as much here, https://news.ycombinator.com/item?id=41418603 https://news.ycombinator.com/item?id=41418603. But there is currently a lot of content that is published openly solely because the pre-AI economic model of the web made it viable. That economic model is now going away, so you'll see a lot more content put behind paywalls (or just not published at all) because AI means that it no longer makes sense for some publishers to spend time and money to produce content they can't get a return on.