4 ms·

I think it's an issue of scale. The next step in your progression here might be: If / when people have personal research bots that go and look for answers acr

I think it's an issue of scale. The next step in your progression here might be: If / when people have personal research bots that go and look for answers across a number of sites, requesting many pages much faster than humans do - what's the tipping point? Is personal web crawling ok? What if it gets a bit smarter and tried to anticipate what you'll ask and does a bunch of crawling to gather information regularly to try to stay up to date on things (from your machine)? Or is it when you tip the scale further and do general / mass crawling for many users to consume that it becomes a problem?

cj 1y ago
Doesn't o3 sort of already do this? Whenever I ask it something, it makes it look like it simultaneously opens 3-8 pages (something a human can't do). Seems like a reasonable stance would be something like "Following the no crawl directive is especially necessary when navigating websites faster than humans can." > What if it gets a bit smarter and tried to anticipate what you'll ask and does a bunch of crawling to gather information regularly to try to stay up to date on things (from your machine)? To be fair, Google Chrome already (somewhat) does this by preloading links it thinks you might click, before you click it. But your point is still valid. We tolerate it because as website owners, we want our sites to load fast for users. But if we're just serving pages to robots and the data is repackaged to users without citing the original source, then yea... let's rethink that.
- Spivak 1y ago
  You don't middle click a bunch of links when doing research? Of all the things to point to I wouldn't have thought "opens a bunch of tabs" to be one of the differentiating behaviors between browsing with Firefox and browsing with an LLM.
  - daviddanielng 1y ago
    > simultaneously opens 3-8 pages (something a human can't do). Can't you read?
- fauigerzigerk 1y ago
  >Doesn't o3 sort of already do this? ChatGPT probably uses a cache though. Theoretically, the average load on the original sites could be far less than users accessing them directly.
- skeezyboy 1y ago
  how do you propose we do anything about this? any law you propose would have to be global
fxtentacle 1y ago
Maybe we should just institutionalize and explicitly legalize the Internet Archive and Archive Team. Then, I can download a complete and halfway current crawl of domain X from the IA and that way, no additional costs are incurred for domain X. But of course, most website publishers would hate that. Because they don't want people to access their content, they want people to look at the ads that pay them. That's why to them, the IA crawling their website is akin to stealing. Because it's taking away some of their ad impressions.
- ivape 1y ago
  Or websites can monetize their data via paid apis and downloadable archives. That's what makes Reddit the most valuable data trove for regular users.
  - ccgreg 1y ago
    I don't think Reddit pays the people who voluntarily write Reddit content. Valuable to Reddit, I guess.
  - p3rls 1y ago
    [flagged]
- palmfacehn 1y ago
  https://commoncrawl.org/ https://commoncrawl.org/ >Common Crawl maintains a free, open repository of web crawl data that can be used by anyone.
  - fxtentacle 1y ago
    The problem is that many websites and domains are missing from it.
- stanmancan 1y ago
  I have mixed feelings on this. Many websites (especially the bigger ones) are just businesses. They pay people to produce content, hopefully make enough ad revenue to make a profit, and repeat. Anything that reproduces their content and steals their views has a direct effect on their income and their ability to stay in business. Maybe IA should have a way for websites to register to collect payment for lost views or something. I think it’s negligible now, there are likely no websites losing meaningful revenue from people using IA instead, but it might be a way to get better buy in if it were institutionalized.
tr_user 1y ago
I saw someone suggest in another post, if only one crawler was visiting and scraping and everyone else reused from that copy I think most websites would be ok with it. But the problem is every billionaire backed startup draining your resources with something similar to a DOS attack.

cj 1y ago

Doesn't o3 sort of already do this? Whenever I ask it something, it makes it look like it simultaneously opens 3-8 pages (something a human can't do). Seems like a reasonable stance would be something like "Following the no crawl directive is especially necessary when navigating websites faster than humans can." > What if it gets a bit smarter and tried to anticipate what you'll ask and does a bunch of crawling to gather information regularly to try to stay up to date on things (from your machine)? To be fair, Google Chrome already (somewhat) does this by preloading links it thinks you might click, before you click it. But your point is still valid. We tolerate it because as website owners, we want our sites to load fast for users. But if we're just serving pages to robots and the data is repackaged to users without citing the original source, then yea... let's rethink that.

Spivak 1y ago
You don't middle click a bunch of links when doing research? Of all the things to point to I wouldn't have thought "opens a bunch of tabs" to be one of the differentiating behaviors between browsing with Firefox and browsing with an LLM.
- daviddanielng 1y ago
  > simultaneously opens 3-8 pages (something a human can't do). Can't you read?
fauigerzigerk 1y ago
>Doesn't o3 sort of already do this? ChatGPT probably uses a cache though. Theoretically, the average load on the original sites could be far less than users accessing them directly.
skeezyboy 1y ago
how do you propose we do anything about this? any law you propose would have to be global

fxtentacle 1y ago

Maybe we should just institutionalize and explicitly legalize the Internet Archive and Archive Team. Then, I can download a complete and halfway current crawl of domain X from the IA and that way, no additional costs are incurred for domain X. But of course, most website publishers would hate that. Because they don't want people to access their content, they want people to look at the ads that pay them. That's why to them, the IA crawling their website is akin to stealing. Because it's taking away some of their ad impressions.

ivape 1y ago
Or websites can monetize their data via paid apis and downloadable archives. That's what makes Reddit the most valuable data trove for regular users.
- ccgreg 1y ago
  I don't think Reddit pays the people who voluntarily write Reddit content. Valuable to Reddit, I guess.
- p3rls 1y ago
  [flagged]
palmfacehn 1y ago
https://commoncrawl.org/ https://commoncrawl.org/ >Common Crawl maintains a free, open repository of web crawl data that can be used by anyone.
- fxtentacle 1y ago
  The problem is that many websites and domains are missing from it.
stanmancan 1y ago
I have mixed feelings on this. Many websites (especially the bigger ones) are just businesses. They pay people to produce content, hopefully make enough ad revenue to make a profit, and repeat. Anything that reproduces their content and steals their views has a direct effect on their income and their ability to stay in business. Maybe IA should have a way for websites to register to collect payment for lost views or something. I think it’s negligible now, there are likely no websites losing meaningful revenue from people using IA instead, but it might be a way to get better buy in if it were institutionalized.

tr_user 1y ago

I saw someone suggest in another post, if only one crawler was visiting and scraping and everyone else reused from that copy I think most websites would be ok with it. But the problem is every billionaire backed startup draining your resources with something similar to a DOS attack.