3 ms·

Feels like only a matter of time before these crawlers are blocked from large swathes of the internet. I understand that they’re already prohibited from Reddit

Feels like only a matter of time before these crawlers are blocked from large swathes of the internet. I understand that they’re already prohibited from Reddit and YouTube. If that spreads, this approach might be in trouble.

drcode 2y ago
I suppose there is an equilibrium, where sites that penalize these types of crawlers will also get less traffic from people reading ai citations, so for many sites the upsides of allowing it will be greater than the downsides.
bbor 2y ago
TBF OpenAI in particular bought access to Reddit. Otherwise yeah this is my main confusion with all of these products, Perplexity being the biggest -- how do you get around the status-quo of refusing access to bots? Just to start off with, there is no Google Search API, and they work hard to make sure headless browsers can't access the normal service. They do say "Currently, deep research can access the open web...", so maybe "open" there implies something significant. Like, "websites that have agreements with OpenAI and/or do not enforce norobot policies".
- wahnfrieden 2y ago
  Client-side browsers that crawl for users (and prompt for logins or captcha as needed) won't be as easily blockable
scarab92 2y ago
I doubt those crawler rules will be honoured for long. I wouldn’t even be surprised if a law is passed requiring sites to provide equal access to humans whether accessed directly or via these models. It’s too important an innovation to stall, especially considering the US’s competitors (China) won’t respect robots.txt either.
cj 2y ago
Anyone selling anything would want to remain crawlable if people use this to research something that could lead to a purchase.
- reaperman 2y ago
  Not necessarily. Southwest airlines doesnt allow itself on price comparison sites or Google Flights. Amazon listings are blocked from google shopping and other price comparison sites.
  - shlomo_z 2y ago
    Your point is completely valid, but... Southwest now has an arrangement with Google Flights to allow their listings there.
  - rsanek 2y ago
    they finally bowed mid last year https://www.nerdwallet.com/article/travel/southwest-google-flights https://www.nerdwallet.com/article/travel/southwest-google-f...
  - yencabulator 2y ago
    > Amazon listings are blocked from google shopping I see Amazon results there all the time. 3 of the visible 8 sponsored results are Amazon, in the non-sponsored results an Amazon listing is either first or second in every category.
crazylogger 2y ago
This is trivially bypassed by OpenAI asking the user to take control of their computer (or a sandboxed browser within it,) then for all intents and purposes it’s the user themselves accessing your site (with some productivity/accessibility aid from OAI.)
optimalsolver 2y ago
Big Tech Podcast listener?
felindev 2y ago
While people might attempt that, it's going to be an arms race, just like ads vs adblocks. There's already multiple crawlers that present fake user-agent when their original one is blocked. Temptation of more data is just to irresistible to them
sumedh 2y ago
> Feels like only a matter of time before these crawlers are blocked from large swathes of the internet. How would you know its a crawler?

drcode 2y ago

I suppose there is an equilibrium, where sites that penalize these types of crawlers will also get less traffic from people reading ai citations, so for many sites the upsides of allowing it will be greater than the downsides.

bbor 2y ago

TBF OpenAI in particular bought access to Reddit. Otherwise yeah this is my main confusion with all of these products, Perplexity being the biggest -- how do you get around the status-quo of refusing access to bots? Just to start off with, there is no Google Search API, and they work hard to make sure headless browsers can't access the normal service. They do say "Currently, deep research can access the open web...", so maybe "open" there implies something significant. Like, "websites that have agreements with OpenAI and/or do not enforce norobot policies".

wahnfrieden 2y ago
Client-side browsers that crawl for users (and prompt for logins or captcha as needed) won't be as easily blockable

scarab92 2y ago

I doubt those crawler rules will be honoured for long. I wouldn’t even be surprised if a law is passed requiring sites to provide equal access to humans whether accessed directly or via these models. It’s too important an innovation to stall, especially considering the US’s competitors (China) won’t respect robots.txt either.

cj 2y ago

Anyone selling anything would want to remain crawlable if people use this to research something that could lead to a purchase.

reaperman 2y ago
Not necessarily. Southwest airlines doesnt allow itself on price comparison sites or Google Flights. Amazon listings are blocked from google shopping and other price comparison sites.
- shlomo_z 2y ago
  Your point is completely valid, but... Southwest now has an arrangement with Google Flights to allow their listings there.
- rsanek 2y ago
  they finally bowed mid last year https://www.nerdwallet.com/article/travel/southwest-google-flights https://www.nerdwallet.com/article/travel/southwest-google-f...
- yencabulator 2y ago
  > Amazon listings are blocked from google shopping I see Amazon results there all the time. 3 of the visible 8 sponsored results are Amazon, in the non-sponsored results an Amazon listing is either first or second in every category.

crazylogger 2y ago

This is trivially bypassed by OpenAI asking the user to take control of their computer (or a sandboxed browser within it,) then for all intents and purposes it’s the user themselves accessing your site (with some productivity/accessibility aid from OAI.)

optimalsolver 2y ago

Big Tech Podcast listener?

felindev 2y ago

While people might attempt that, it's going to be an arms race, just like ads vs adblocks. There's already multiple crawlers that present fake user-agent when their original one is blocked. Temptation of more data is just to irresistible to them

sumedh 2y ago

> Feels like only a matter of time before these crawlers are blocked from large swathes of the internet. How would you know its a crawler?