3 ms·

A fossilized snapshot will only get them so far, and sites are increasingly opting to block AI-related crawlers. Apparently about a quarter of the top 1000 site

A fossilized snapshot will only get them so far, and sites are increasingly opting to block AI-related crawlers. Apparently about a quarter of the top 1000 sites already block GPTBot: https://originality.ai/ai-bot-blocking https://originality.ai/ai-bot-blocking I guess they could be using Bing as their search backend, which would mostly get around the blocking issue (except for searching Reddit which blocks Bingbot now).

toomuchtodo 2y ago
Certainly, countermeasures against crawler blocking will be a necessary component of effective search corpus aggregation in the go forward. Otherwise, search will balkanize around who will pay the most for access to public content. Common Crawl is ~10PB, this is not insurmountable. Edit: I understand there is a freerider/economic issue here, unsure how to solve that as the balance between search engine/gen AI systems and content stores/providers becomes more adversarial.
- jsheard 2y ago
  AFAIK OpenAI currently respects robots.txt, so we'll have to see if they change that policy out of desperation at some point.
  - andrethegiant 2y ago
    > AFAIK OpenAI currently respects robots.txt I wonder to what degree -- for example, do they respect the Crawl-delay directive? For example, HN itself has a 30-second crawl-delay (https://news.ycombinator.com/robots.txt https://news.ycombinator.com/robots.txt), meaning that crawlers are supposed to wait 30 seconds before requesting the next page. I doubt ChatGPT will delay a user's search of HN by up to 30 seconds, even though that's what robots.txt instructs them to do.
    - echoangle 2y ago
      Would ChatGPT when live interacting with a user even have to respect robots.txt? I would think the robots.txt only applies to automatic crawling. When directed by a user, one could argue that ChatGPT is basically the user agent the user is using to view the web. If you wanted to write a browser extension that shows the reading time for all search results on google, would you respect robots.txt when prefetching all pages from the results? I probably wouldn’t, because that’s not really automated crawling to me.
  - claudiulodro 2y ago
    They do respect robots.txt (supposedly), but they also introduced a new user agent that nobody would yet have in their robots.txt as part of this feature[1], and looking at my server logs it's already crawled a bunch of sites. [1] https://platform.openai.com/docs/bots/overview-of-openai-crawlers https://platform.openai.com/docs/bots/overview-of-openai-cra...
StableAlkyne 2y ago
If it ends up anywhere near as popular as Google, those sites will have a financial incentive to allow the crawlers. The average person just does not discover content without the search engine recommending it.
- jsheard 2y ago
  The whole issue that site owners have with these AI search engines is that there isn't a financial incentive for them to cooperate, since the summarization largely replaces the need for users to click through to the site the information came from. No click-through, no ad impressions, no possibility of the user being converted into a recurring visitor or paid subscriber, just pure freeloading by the search engine.

toomuchtodo 2y ago

Certainly, countermeasures against crawler blocking will be a necessary component of effective search corpus aggregation in the go forward. Otherwise, search will balkanize around who will pay the most for access to public content. Common Crawl is ~10PB, this is not insurmountable. Edit: I understand there is a freerider/economic issue here, unsure how to solve that as the balance between search engine/gen AI systems and content stores/providers becomes more adversarial.

jsheard 2y ago
AFAIK OpenAI currently respects robots.txt, so we'll have to see if they change that policy out of desperation at some point.
- andrethegiant 2y ago
  > AFAIK OpenAI currently respects robots.txt I wonder to what degree -- for example, do they respect the Crawl-delay directive? For example, HN itself has a 30-second crawl-delay (https://news.ycombinator.com/robots.txt https://news.ycombinator.com/robots.txt), meaning that crawlers are supposed to wait 30 seconds before requesting the next page. I doubt ChatGPT will delay a user's search of HN by up to 30 seconds, even though that's what robots.txt instructs them to do.
  - echoangle 2y ago
    Would ChatGPT when live interacting with a user even have to respect robots.txt? I would think the robots.txt only applies to automatic crawling. When directed by a user, one could argue that ChatGPT is basically the user agent the user is using to view the web. If you wanted to write a browser extension that shows the reading time for all search results on google, would you respect robots.txt when prefetching all pages from the results? I probably wouldn’t, because that’s not really automated crawling to me.
- claudiulodro 2y ago
  They do respect robots.txt (supposedly), but they also introduced a new user agent that nobody would yet have in their robots.txt as part of this feature[1], and looking at my server logs it's already crawled a bunch of sites. [1] https://platform.openai.com/docs/bots/overview-of-openai-crawlers https://platform.openai.com/docs/bots/overview-of-openai-cra...

StableAlkyne 2y ago

If it ends up anywhere near as popular as Google, those sites will have a financial incentive to allow the crawlers. The average person just does not discover content without the search engine recommending it.

jsheard 2y ago
The whole issue that site owners have with these AI search engines is that there isn't a financial incentive for them to cooperate, since the summarization largely replaces the need for users to click through to the site the information came from. No click-through, no ad impressions, no possibility of the user being converted into a recurring visitor or paid subscriber, just pure freeloading by the search engine.