3 ms·

every major news org now blocks the parasitic "AI" crawlers examples: https://www.bbc.co.uk/robots.txt https://www.cnn.com/robots.txt https://www.

every major news org now blocks the parasitic "AI" crawlers examples: https://www.bbc.co.uk/robots.txt https://www.cnn.com/robots.txt https://www.nbcnews.com/robots.txt all they will be training on now is spam anyone that says "AI is the worst today it will ever be", no because that was before the world reacted to it

AstroBen 1y ago
They torrented a shitload of books illegally and trained on them.. but they're unable to get past The Great Wall of robots.txt?
- strangattractor 1y ago
  Great Firewall actually. Robots.txt depended on the integrity of the companies crawling. I think they have demonstrated how much integrity they actually have:)
- blibble 1y ago
  I doubt it's their only countermeasure plus it's a pretty dangerous game for them to play against large, powerful actors with legions of lawyers
  - selcuka 1y ago
    > plus it's a pretty dangerous game for them to play against large, powerful actors with legions of lawyers Like book publishers?
- Gigachad 1y ago
  If the AI crawlers circumvent the protection mechanisms it's a serious crime now rather than just "Well it was on the open internet for free". Wouldn't surprise me if the the news orgs are also looking at honeypot articles to see if the fake details slip in to LLMs.
  - crazygringo 1y ago
    It's not a serious crime, or any crime at all, to ignore robots.txt. It's entirely voluntary whether you want to follow it or not. If you don't, you're being a dick maybe, but that's not a crime.
    - Gigachad 1y ago
      It's not just robots.txt, if you've tried using a VPN lately, so many sites like reddit/youtube/etc block you from viewing content until you log in. Every major website is getting anti scraping tech in the last year. Even archive.org is getting blocked from more and more sites since it can be used for indirect scraping of sites.
- simianwords 1y ago
  robots.txt prevents real time search use for grounding and citations.
  - crazygringo 1y ago
    No it doesn't. It has zero legal force. Or any technical force either.
    - simianwords 1y ago
      Not an expert so I ask: no technical force either? Is it just a polite ask then?
      - crazygringo 1y ago
        Correct. Literally just a polite ask.
      - zdragnar 1y ago
        It's hardly even a polite ask. It's literally a text file. Automated http clients, such as search engine indexers (Google, yahoo, etc) are expected to use it to know what pages can be visited or not. That expectation is nothing more than a convention. If you are on a Mac or Linux computer, odds are it has a program called curl pre-installed. If you type in curl website address in a terminal, it'll fetch make a request and download the response. Robot.txt never gets involved. Same is true for AI agents and search engines that aren't polite.
      - euLh7SM5HDFY 1y ago
        Linkedin lost their anti-scrapping suit: https://www.forbes.com/sites/zacharysmith/2022/04/18/scraping-data-from-linkedin-profiles-is-legal-appeals-court-rules/ https://www.forbes.com/sites/zacharysmith/2022/04/18/scrapin... but it seems since then they were able to successfully appeal that decision. Regardless - requiring an account to read anything, even a "free" one, totally changes whole situation. Even when sites terms of service are limited by local law.
faangguyindia 1y ago
My take is when AGI comes into existence and breaks out of labs to become our masters. Those who opposed AI adoption will be the first who will be sent to labor camps. I want to be in good books of AGI masters, so i am helping apply AI everywhere.
- vict7 1y ago
  AKA Roko’s Basilisk
- JKCalhoun 1y ago
  > Those who opposed AI adoption will be the first who will be sent to labor camps. I know you're joking, but other people are serious about this. Why do they think that an AGI will be vengeful? So strange.
  - teitoklien 1y ago
    The desire to become superbeing entity cannot be separated from supremacist ideals
  - macintux 1y ago
    Reminds me of Bruce Wayne from the dreadful DC film. > He has the power to wipe out the entire human race, and if we believe there's even a one percent chance that he is our enemy we have to take it as an absolute certainty... and we have to destroy him.
- Analemma_ 1y ago
  This is pretty much word-for-word the reasoning behind Roko’s basilisk, which made its proponents an internet laughingstock for a decade, but is a surprisingly tricky thing to actually refute if you accept the premise that AGI is in fact coming.
  - AstroBen 1y ago
    It seems quite easy to refute—why would it punish anyone? We would pose 0 threat at that point to any super intelligence, and I highly doubt it would have anything like a human grudge. It's just a case of anthropomorphizing it
simonw 1y ago
They seem to still be allowing Google to crawl them, unsurprisingly. Advantage Gemini.
- strangattractor 1y ago
  The Google advantage is that you need to show up in their search results or else you are a nobody.
- selcuka 1y ago
  No, they explicitly block Gemini as well: User-agent: Google-Extended Disallow: / Gemini still uses the same user agent, but it has a different robots.txt entry (Google-Extended) [1]: > Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI. [1] https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers#google-extended https://developers.google.com/search/docs/crawling-indexing/...
  - simonw 1y ago
    Honestly I feel like "training" is a bit of a distraction at this point. For a lot of types of content RAG-style search is much more important. I imagine many of the orgs that are blocking "training" don't understand the difference between training and inference-time tool-based context extension (which really needs an agreed upon name, it's hard to talk about right now).
    - selcuka 1y ago
      My understanding is that it also affects RAG ("grounding" in Google terminology): > [...] and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI. So they seem to be blocking both training and RAG while still allowing search engine indexing.
klysm 1y ago
And this is stopping who exactly?
- ares623 1y ago
  I suppose there’s a non-zero chance that a future lawsuit can point to this explicit block and say “see judge, we explicitly don’t want them crawling our stuff. Remember that Linkedin case from a while back?”
simianwords 1y ago
This is also for real time "grounding". So it makes it even harder for AI to give factual answers.

AstroBen 1y ago

They torrented a shitload of books illegally and trained on them.. but they're unable to get past The Great Wall of robots.txt?

strangattractor 1y ago
Great Firewall actually. Robots.txt depended on the integrity of the companies crawling. I think they have demonstrated how much integrity they actually have:)
blibble 1y ago
I doubt it's their only countermeasure plus it's a pretty dangerous game for them to play against large, powerful actors with legions of lawyers
- selcuka 1y ago
  > plus it's a pretty dangerous game for them to play against large, powerful actors with legions of lawyers Like book publishers?
Gigachad 1y ago
If the AI crawlers circumvent the protection mechanisms it's a serious crime now rather than just "Well it was on the open internet for free". Wouldn't surprise me if the the news orgs are also looking at honeypot articles to see if the fake details slip in to LLMs.
- crazygringo 1y ago
  It's not a serious crime, or any crime at all, to ignore robots.txt. It's entirely voluntary whether you want to follow it or not. If you don't, you're being a dick maybe, but that's not a crime.
  - Gigachad 1y ago
    It's not just robots.txt, if you've tried using a VPN lately, so many sites like reddit/youtube/etc block you from viewing content until you log in. Every major website is getting anti scraping tech in the last year. Even archive.org is getting blocked from more and more sites since it can be used for indirect scraping of sites.
simianwords 1y ago
robots.txt prevents real time search use for grounding and citations.
- crazygringo 1y ago
  No it doesn't. It has zero legal force. Or any technical force either.
  - simianwords 1y ago
    Not an expert so I ask: no technical force either? Is it just a polite ask then?
    - crazygringo 1y ago
      Correct. Literally just a polite ask.
    - zdragnar 1y ago
      It's hardly even a polite ask. It's literally a text file. Automated http clients, such as search engine indexers (Google, yahoo, etc) are expected to use it to know what pages can be visited or not. That expectation is nothing more than a convention. If you are on a Mac or Linux computer, odds are it has a program called curl pre-installed. If you type in curl website address in a terminal, it'll fetch make a request and download the response. Robot.txt never gets involved. Same is true for AI agents and search engines that aren't polite.
    - euLh7SM5HDFY 1y ago
      Linkedin lost their anti-scrapping suit: https://www.forbes.com/sites/zacharysmith/2022/04/18/scraping-data-from-linkedin-profiles-is-legal-appeals-court-rules/ https://www.forbes.com/sites/zacharysmith/2022/04/18/scrapin... but it seems since then they were able to successfully appeal that decision. Regardless - requiring an account to read anything, even a "free" one, totally changes whole situation. Even when sites terms of service are limited by local law.

faangguyindia 1y ago

My take is when AGI comes into existence and breaks out of labs to become our masters. Those who opposed AI adoption will be the first who will be sent to labor camps. I want to be in good books of AGI masters, so i am helping apply AI everywhere.

vict7 1y ago
AKA Roko’s Basilisk
JKCalhoun 1y ago
> Those who opposed AI adoption will be the first who will be sent to labor camps. I know you're joking, but other people are serious about this. Why do they think that an AGI will be vengeful? So strange.
- teitoklien 1y ago
  The desire to become superbeing entity cannot be separated from supremacist ideals
- macintux 1y ago
  Reminds me of Bruce Wayne from the dreadful DC film. > He has the power to wipe out the entire human race, and if we believe there's even a one percent chance that he is our enemy we have to take it as an absolute certainty... and we have to destroy him.
Analemma_ 1y ago
This is pretty much word-for-word the reasoning behind Roko’s basilisk, which made its proponents an internet laughingstock for a decade, but is a surprisingly tricky thing to actually refute if you accept the premise that AGI is in fact coming.
- AstroBen 1y ago
  It seems quite easy to refute—why would it punish anyone? We would pose 0 threat at that point to any super intelligence, and I highly doubt it would have anything like a human grudge. It's just a case of anthropomorphizing it

simonw 1y ago

They seem to still be allowing Google to crawl them, unsurprisingly. Advantage Gemini.

strangattractor 1y ago
The Google advantage is that you need to show up in their search results or else you are a nobody.
selcuka 1y ago
No, they explicitly block Gemini as well: User-agent: Google-Extended Disallow: / Gemini still uses the same user agent, but it has a different robots.txt entry (Google-Extended) [1]: > Google-Extended is a standalone product token that web publishers can use to manage whether content Google crawls from their sites may be used for training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI. [1] https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers#google-extended https://developers.google.com/search/docs/crawling-indexing/...
- simonw 1y ago
  Honestly I feel like "training" is a bit of a distraction at this point. For a lot of types of content RAG-style search is much more important. I imagine many of the orgs that are blocking "training" don't understand the difference between training and inference-time tool-based context extension (which really needs an agreed upon name, it's hard to talk about right now).
  - selcuka 1y ago
    My understanding is that it also affects RAG ("grounding" in Google terminology): > [...] and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy) in Gemini Apps and Grounding with Google Search on Vertex AI. So they seem to be blocking both training and RAG while still allowing search engine indexing.

klysm 1y ago

And this is stopping who exactly?

ares623 1y ago
I suppose there’s a non-zero chance that a future lawsuit can point to this explicit block and say “see judge, we explicitly don’t want them crawling our stuff. Remember that Linkedin case from a while back?”

simianwords 1y ago

This is also for real time "grounding". So it makes it even harder for AI to give factual answers.