5 ms·

I turned this on and it adjusts the robots.txt automatically; not sure what else it is doing. # NOTICE: The collection of content and other data on this # site

I turned this on and it adjusts the robots.txt automatically; not sure what else it is doing. # NOTICE: The collection of content and other data on this # site through automated means, including any device, tool, # or process designed to data mine or scrape content, is # prohibited except (1) for the purpose of search engine indexing or # artificial intelligence retrieval augmented generation or (2) with express # written permission from this site’s operator. # To request permission to license our intellectual # property and/or other materials, please contact this # site’s operator directly. # BEGIN Cloudflare Managed content User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: meta-externalagent Disallow: / # END Cloudflare Managed Content User-agent: * Disallow: /* Allow: /$

xyst 1y ago
So in addition to updating the robots.txt file, which really only blocks a small number of them. Seems CF has been gathering data and profiling these malicious agents. This post by CF elaborates a bit further: https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/ https://blog.cloudflare.com/declaring-your-aindependence-blo... Basically becomes a game of cat and mouse.
postalcoder 1y ago
This is interesting. The reasoning and response don't line up. > Cloudflare is making the change to protect original content on the internet, Mr. Prince said. If A.I. companies freely use data from various websites without permission or payment, people will be discouraged from creating new digital content, he said > prohibited except for the purpose of [..] artificial intelligence retrieval augmented generation This seems to be targeted at taxing training of language models, but why an exclusion for the RAG stuff? That seems like it has a much greater immediate impact for online content creators, for whom the bots are obviating a click.
- fennecfoxy 1y ago
  With that opinion, are you also suggesting that we ban ad blockers? Because it's better I not click & consume resources than click and not be served ads, basically just costing the host money. It means sense to allow for RAG in the same way that search engines provide a snippet of an important chunk of the page. A blog author could not complain that their blog is getting ragged when they're extremely liable to be Google/whatever searching all day and basically consuming others' content in exactly the same way that they're trying to disparage.
  - postalcoder 1y ago
    I don't think we should ban ad blockers, but I also think it's fair to suggest that the loss of organic traffic could be affecting the incentive to create new digital content, at least as much as the fear of having your content absorbed into an LLM's training data.
    - Boldened15 1y ago
      IMO the backlash against LLMs is more philosophical, a lot of people don’t like them or the idea of one learning from their content. Unless your website has some unique niche information unavailable anywhere else there’s no direct personal risk. RAG would be a more direct threat if anything.
      - toomuchtodo 1y ago
        It's really about who is getting the value from the work of the content. If content creators of all sorts have their work consumed by LLMs, and LLM orgs charge for it can capture all the value, why should people create to have their work vacuumed up for the robot's benefit? For exposure? You can't eat or pay rent with exposure. Humans must get paid, and LLMs (foundational models and output using RAG) cannot improve without a stream of works and data humans create. Whether you call it training or something else is irrelevant, it's really exploitation of human work and effort for AI shareholder returns and tech worker comp (if those who create aren't compensated). And the technocracy has not been, based on the evidence, great stewards of the power they obtain through this. Pay the humans for their work.
bee_rider 1y ago
I wonder… Google scrapes for indexing and for AI, right? I wonder if they will eventually say: ok, you can have me or not, if you don’t want to help train my AI you won’t get my searches either. That’s a tough deal but it is sort of self-consistent.
- giancarlostoro 1y ago
  "Embrace, Extend, Extinguish" Google's mantra. And yes, I know about Microsoft's history with that phrase ;) But Google has done this with email, browsers (Google has web apps that run fine on Firefox but request you use Chrome), Linux (Android), and I'm sure there's others I am forgetting about. So yeah, I too could see them doing this.
- mrweasel 1y ago
  Very few people seems to be complaining that Google crashes their sites. Google also publish their crawlers IP ranges, but you really don't need to rate-limit Google, they know how to back off and not overload sites.
  - Symbiote 1y ago
    In theory — in practise I've had to limit Google on two large sites at work. I currently have them limited to 10/s for non-cached requests.
    - progmetaldev 1y ago
      Curious if the content on those sites might have high value to Google? Such as if they have data that is new or unavailable elsewhere, or if they're just standard sites, and you've just been unlucky? I have had odd bot behavior from some major crawlers, but never from Google. I wonder if there is a correlation to usefulness of content, or if certain sites get stuck in a software bug (or some other strange behavior).
      - Symbiote 1y ago
1vuio0pswjnm7 1y ago
"User-agent: CCBot disallow: /" Is Common Crawl exclusively for "AI" CCBot was already in so many robots.txt prior to this How is CC supposed to know or control how people use the archive contents What if CC is relying on fair use # To request permission to license our intellectual # property andd/or other materials, please contact this # site's operator directly If the operator has no intellectual property rights in the material, then do they need permission from the rights holders to license such materials for use in creating LLMs and collect licensing fees Is it common for website terms and conditions to permit site operators to sublicense other peoples' ("users") work for use in creating LLMs for a fee Is this fee shared with the rights holders
- nemomarx 1y ago
  Read a tos and notice that you give the site operators unlimited license to reproduce or spread your works, almost on any site. it's required to host and show the content essentially
- ronsor 1y ago
  # To request permission to license our intellectual # property andd/or other materials, please contact this # site's operator directly Scrapers don't accept the terms of service. Ironically, I've only ever scraped sites that block CCBot, otherwise I'd rather go to Common Crawl for the data.
Bender 1y ago
For my silly hobby sites I just return status 444 close the connection for anything that has case-insentive "bot" in the UA requesting anything other than robots.txt, humans.txt, favicon.ico, etc... This would also drop search engines but I blackhole route most of their CIDR blocks. I'm probably the only one here that would do this.
- sneak 1y ago
  How does a bot scraping your silly hobby sites for any purpose harm or negatively affect you in any way?
  - pixl97 1y ago
    Depends if they hit a site enough to make it cost something. It's not hard for bots to flood servers.
  - Bender 1y ago
    Only if they push me over my bandwidth limits but they can't do that if I just drop them on the floor.
slenk 1y ago
I thought I saw cloudflare insert noindex links?
swyx 1y ago
what actually are the consequences of ignoring robots.txt (apart from DDOS)? have any of these cases ended up in court at all?
- v5v3 1y ago
  BBC recently served a cease and desist on perplexity to stop, and delete all existing. https://www.bbc.co.uk/news/articles/cy7ndgylzzmo https://www.bbc.co.uk/news/articles/cy7ndgylzzmo So an ai company can just be naughty till asked to stop, and then exclud that one company that has the financial resources to go legal.
lxgr 1y ago
That's at least a more reasonable default than that I've seen at least one newspaper do, which is to block both LLM scrapers and things like ChatGPT's search feature explicitly.

xyst 1y ago

So in addition to updating the robots.txt file, which really only blocks a small number of them. Seems CF has been gathering data and profiling these malicious agents. This post by CF elaborates a bit further: https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/ https://blog.cloudflare.com/declaring-your-aindependence-blo... Basically becomes a game of cat and mouse.

postalcoder 1y ago

This is interesting. The reasoning and response don't line up. > Cloudflare is making the change to protect original content on the internet, Mr. Prince said. If A.I. companies freely use data from various websites without permission or payment, people will be discouraged from creating new digital content, he said > prohibited except for the purpose of [..] artificial intelligence retrieval augmented generation This seems to be targeted at taxing training of language models, but why an exclusion for the RAG stuff? That seems like it has a much greater immediate impact for online content creators, for whom the bots are obviating a click.

fennecfoxy 1y ago
With that opinion, are you also suggesting that we ban ad blockers? Because it's better I not click & consume resources than click and not be served ads, basically just costing the host money. It means sense to allow for RAG in the same way that search engines provide a snippet of an important chunk of the page. A blog author could not complain that their blog is getting ragged when they're extremely liable to be Google/whatever searching all day and basically consuming others' content in exactly the same way that they're trying to disparage.
- postalcoder 1y ago
  I don't think we should ban ad blockers, but I also think it's fair to suggest that the loss of organic traffic could be affecting the incentive to create new digital content, at least as much as the fear of having your content absorbed into an LLM's training data.
  - Boldened15 1y ago
    IMO the backlash against LLMs is more philosophical, a lot of people don’t like them or the idea of one learning from their content. Unless your website has some unique niche information unavailable anywhere else there’s no direct personal risk. RAG would be a more direct threat if anything.
    - toomuchtodo 1y ago
      It's really about who is getting the value from the work of the content. If content creators of all sorts have their work consumed by LLMs, and LLM orgs charge for it can capture all the value, why should people create to have their work vacuumed up for the robot's benefit? For exposure? You can't eat or pay rent with exposure. Humans must get paid, and LLMs (foundational models and output using RAG) cannot improve without a stream of works and data humans create. Whether you call it training or something else is irrelevant, it's really exploitation of human work and effort for AI shareholder returns and tech worker comp (if those who create aren't compensated). And the technocracy has not been, based on the evidence, great stewards of the power they obtain through this. Pay the humans for their work.

bee_rider 1y ago

I wonder… Google scrapes for indexing and for AI, right? I wonder if they will eventually say: ok, you can have me or not, if you don’t want to help train my AI you won’t get my searches either. That’s a tough deal but it is sort of self-consistent.

giancarlostoro 1y ago
"Embrace, Extend, Extinguish" Google's mantra. And yes, I know about Microsoft's history with that phrase ;) But Google has done this with email, browsers (Google has web apps that run fine on Firefox but request you use Chrome), Linux (Android), and I'm sure there's others I am forgetting about. So yeah, I too could see them doing this.
mrweasel 1y ago
Very few people seems to be complaining that Google crashes their sites. Google also publish their crawlers IP ranges, but you really don't need to rate-limit Google, they know how to back off and not overload sites.
- Symbiote 1y ago
  In theory — in practise I've had to limit Google on two large sites at work. I currently have them limited to 10/s for non-cached requests.
  - progmetaldev 1y ago
    Curious if the content on those sites might have high value to Google? Such as if they have data that is new or unavailable elsewhere, or if they're just standard sites, and you've just been unlucky? I have had odd bot behavior from some major crawlers, but never from Google. I wonder if there is a correlation to usefulness of content, or if certain sites get stuck in a software bug (or some other strange behavior).
    - Symbiote 1y ago

1vuio0pswjnm7 1y ago

"User-agent: CCBot disallow: /" Is Common Crawl exclusively for "AI" CCBot was already in so many robots.txt prior to this How is CC supposed to know or control how people use the archive contents What if CC is relying on fair use # To request permission to license our intellectual # property andd/or other materials, please contact this # site's operator directly If the operator has no intellectual property rights in the material, then do they need permission from the rights holders to license such materials for use in creating LLMs and collect licensing fees Is it common for website terms and conditions to permit site operators to sublicense other peoples' ("users") work for use in creating LLMs for a fee Is this fee shared with the rights holders

nemomarx 1y ago
Read a tos and notice that you give the site operators unlimited license to reproduce or spread your works, almost on any site. it's required to host and show the content essentially
ronsor 1y ago
# To request permission to license our intellectual # property andd/or other materials, please contact this # site's operator directly Scrapers don't accept the terms of service. Ironically, I've only ever scraped sites that block CCBot, otherwise I'd rather go to Common Crawl for the data.

Bender 1y ago

For my silly hobby sites I just return status 444 close the connection for anything that has case-insentive "bot" in the UA requesting anything other than robots.txt, humans.txt, favicon.ico, etc... This would also drop search engines but I blackhole route most of their CIDR blocks. I'm probably the only one here that would do this.

sneak 1y ago
How does a bot scraping your silly hobby sites for any purpose harm or negatively affect you in any way?
- pixl97 1y ago
  Depends if they hit a site enough to make it cost something. It's not hard for bots to flood servers.
- Bender 1y ago
  Only if they push me over my bandwidth limits but they can't do that if I just drop them on the floor.

slenk 1y ago

I thought I saw cloudflare insert noindex links?

swyx 1y ago

what actually are the consequences of ignoring robots.txt (apart from DDOS)? have any of these cases ended up in court at all?

v5v3 1y ago
BBC recently served a cease and desist on perplexity to stop, and delete all existing. https://www.bbc.co.uk/news/articles/cy7ndgylzzmo https://www.bbc.co.uk/news/articles/cy7ndgylzzmo So an ai company can just be naughty till asked to stop, and then exclud that one company that has the financial resources to go legal.

lxgr 1y ago

That's at least a more reasonable default than that I've seen at least one newspaper do, which is to block both LLM scrapers and things like ChatGPT's search feature explicitly.