4 ms·

AI crawlers haven't learned to play nice with websites

77 points by belter 2y ago

shprd 2y ago
That's intentional. They take the data and deal with the consequences later because there's none so far. I hope data poisoning see more traction as a possible countermeasure.
- 01HNNWZ0MV43FF 2y ago
  I wonder what their data cleaning process looks like. If I put a lot of furry porn on my website, would it get excluded from the data sets?
  - mtvrs 2y ago
    Probably only those data sets which do not include furry porn but it's also possible that it could be used in negative data set.
Gerardo1 2y ago
Is there any indication that the owners of those crawlers intend for them to play nice? I think this form the article is telling: > The Register asked Schubert about this in early January. "Funnily enough, a few days after the post went viral, all crawling stopped," he responded at the time. "Not just on the Diaspora wiki, but on my entire infrastructure. I'm not entirely sure why, but here we are." So the owners of those crawlers certainly have the ability to stop the traffic, they're just choosing not too...until publicity makes their actions risky.
entropi 2y ago
The owners don't even care about copyright and their deep-pocketed owners. They will certainly not care about playing nice with random websites, or any conventions that may exist in any domain.
ohgr 2y ago
I just shut my entire web site down after 27 years this year. All traffic was AI crawlers. Fuck it!
- bayindirh 2y ago
  Why not install Nepenthes and let them bath in the tar?
  - ohgr 2y ago
    Spend more time and money? Nope. It's on CloudFront + S3 at the moment.
blueflow 2y ago
I set up a robots.txt with one entry: a disallow for a specific html file. The html file is not listed elsewhere and thus can only be discovered by the robots.txt. The file features random stuff and an <a href=""> element with the nofollow attribute and a link target set. That link has no text contents, no size, and thus doesn't occupy pixels that a user could click on. When clicked, that link would report the http request to me. At 2025-02-22 02:14:52, less than a day after i set up this honeypot, 66.249.68.37 (AS15169 GOOGLE) made a request on that invisible link, using user agent string "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.6943.53 Mobile Safari/537.36 (compatible; GoogleOther)" So now i have hard evidence that google runs bots that ignore the rules from the robots.txt, but still parses urls from there, queries these resources anyways and follows links marked as "nofollow".
- bayindirh 2y ago
  It's already well known that none of the search engines, crawlers, and companies honor robots.txt (anymore). It's just for us oldies who prefer the ethical web where people respected the RFCs they have collectively written.
  - pimlottc 2y ago
    It’s one thing to know they don’t honor it, it’s another to know they intentionally abuse it
    - johnea 2y ago
      Yes, it is, and now we know both of those things don't we. What will be the consequence? My risky prediction: Absolutely Nothing, for neither goggle nor any of the other miscreants running the bot nets. And just to take a poke at the title of the original El Reg article: Do you really think the problem is that they just haven't "learned" yet how to "play nice"? I'm sure the great oracle goggle, just needs someone to poperly splain it to them.
- extraduder_ire 2y ago
yodon 2y ago
As someone who has done a tiny bit of scraping, I've been surprised how difficult it is to be a good citizen. Are there packages you know of in the major language ecosystems that you can easily point at a both a robots.txt and an index.html (you need both to be compliant, most packages I've found only look at robots.txt) to get an answer on what you can and can't do at what rate? The only one I'm aware of is a C# package which still requires you to do a lot of the heavy lifting[0]. [0]https://github.com/TurnerSoftware/RobotsExclusionTools https://github.com/TurnerSoftware/RobotsExclusionTools
- cavisne 2y ago
  https://github.com/google/robotstxt https://github.com/google/robotstxt
  - yodon 2y ago
    You're proving my point. That's a reference C++ parser that only parses robots.txt files. It doesn't parse meta tags and doesn't help with writing crawlers running in any of the major language ecosystems used by crawler authors. It's much more difficult than it should be for scrapers who want to be compliant to to actually be compliant.
- mrweasel 2y ago
  You obviously should read the robots.txt and follow it, but it's just as much about not just absolutely hammering a site over and over again, 24-7. Your scraper needs to learn about update patterns and adjust it's scraping interval. It needs to back off if the respons times increases. People are bringing up robots.txt because they want to ban AI scrapers, but if those same scrapers didn't constantly pound sites into the ground, it would be less of an issue.
esskay 2y ago
I manage a few thousand client sites spread across a bunch of differing infrastructures but one common thing we have on all of them is all the specific robots rules to block ai crawlers. Not a single one of them respects that request anymore. Anthropic is the worst at the moment. It starts hitting the site with an anthropic useragent. You can litterally see it hit the robots file, then it changes it's useragent to a generic browser one and carries on. I say at the moment because they're all doing this kind of crap, they seem to take it in turns at ramping up hammering servers.
- onion2k 2y ago
  I say at the moment because they're all doing this kind of crap... Maybe they used AI to code their agent, and it's just not that good.
- pjc50 2y ago
  > Not a single one of them respects that request anymore AI companies have seen the prisoner's dilemma of good internet citizenship and slammed hard on the "defect" button. They plan to steal your content, put it in an attribution-removing blender, then sell it to other people. With the explicit purpose of replacing human contribution on the internet and in the workplace. After all, they represent a reality-distorting amount of capital, so why should any rules apply to them?
leonidasv 2y ago
This is going to drive websites to put even more Cloudflare captcha-walls before you can even see the content. Web will become less accessible. I have a domain parked with just a blank page that's getting 172,000 requests per month. Since it's on a free-tier static hosting, that costs me nothing. However, if that trend continues, I don't know how long we will have those free static hosting options available for everyone.
- deadbabe 2y ago
  Send a zip bomb
  - deleted 2y ago
    [deleted]
  - jonatron 2y ago
    It didn't seem to matter what response I sent to PerplexityBot, the requests kept coming. I guess it's not surprising that a badly written bot isn't monitored either.
    - reginald78 2y ago
      Can you use hashcash or some proof of work system somehow? I don't know anything about this space.
      - jonatron 2y ago
        They were only requesting the HTML, not using a headless browser.
    - Dwedit 2y ago
      There was the tarpit that sends out markov-chain-generated random words, loading very slowly . If the client has a limited number of concurrent requests, that could theoretically choke the client.
    - JohnFen 2y ago
discardedrefuse 2y ago
This headline sucks. It suggest that AI crawlers' bad behavior isn't intentional. Robots.txt has existed for ages, and crawler operators know exactly how to play nice. They simply choose not to. Media outlets need to stop attributing ignorance or incompetence to what is clearly willful action. These megacorps need to be held accountable, even on the little things.
- soco 2y ago
  Technically it's true - they have not learned, but because nobody taught them to. Because nobody slapped any over-reaching wrist, ever, and if somebody even just thinks about, then it's all "damn EU bureaucrats".
- mrweasel 2y ago
  > It suggest that AI crawlers' bad behaviour isn't intentional. The robots.txt is only part of the problem, simply overloading sites is another. Maybe you don't really mind that your site is getting scraped, but you do mind that it crashes due to poorly written scrapers. It was always understood that your scrappers should limit their load to something that a site can handle. Google has always excelled at this, I've never experienced the GoogleBot being an issue, nor the BingBot. The developers who work at the AI companies however are less talented and less careering. If their bot crashes a site, meeeh, try again later. I don't think they DDoS sites deliberately, they just don't have the skills to fix their shitty scrapper. If we want the problem to go away, ask AWS, Azure, GCP, Alibaba Cloud and others for a way to report bad behaviour. Make their hosting providers take action.
  - danaris 2y ago
    I will go one step further: It doesn't matter whether they have the skills or not, because they don't care. As long as they can siphon up enough data to increase their revenue or stock price by 0.001¢, they're perfectly content to leave small independent websites shattered and unusable under the weight of their aggressive, unrelenting scraper bots. I'm firmly of the opinion that they should all be considered hostile, predatory companies, regardless of what supposed "value" they bring. Train your river-guzzling models with data that's paid for or given voluntarily and then we can talk about your "value".
josefritzishere 2y ago
I just update my robots.txt to block AI crawlers just so I cna join the future inevitable class action lawsuit.
tempodox 2y ago
If you rip everyone off anyway, why would you bother doing it “the nice way”?
boredhacker3 2y ago
Why don’t they use an anti-bot solution like Cloudflare? It won’t block everything entirely, but it will at least stop low-quality bots
deleted 2y ago
[deleted]
deleted 2y ago
[deleted]
CraigRood 2y ago
I've personally noticed loads of crawls too, but what has been interesting is they have been crawling effectively the same pages, over and over, for months. Also, they don't seem to be caching DNS at all. I guess I'm in the minority that I don't care too much about crawlers, even the use of my work for AI, but I think the current hit rates just can't be justified.
elpocko 2y ago
>SourceHut says it's getting DDoSed by LLM bots How do they know it's LLM "bots"? Did they provide any proof? Is it crawlers or automated user agents? The guy behind SH hates all "AI" and is vocal about it; it would be reassuring to see some evidence, all I've seen is them saying "it's those pesky AI bots!"
dmje 2y ago
Compared to others in the comments (with "a few thousand sites"!!) we're obviously small beer, with maybe 50 or so - but we are being botted to death and it's driving us fucking nuts :-/
EdiX 2y ago
If search engines were invented today they wouldn't stand a chance. People would complain that they can return immoral results. Different sets of people would complain that they can return links to various isms that offend them. The owners would be worried that they return results critical of themselves or their friends. Copyright owners would sue them for linking to piracy. Website owners would block them for reading their precious content.

77 points by belter 2y ago

That's intentional. They take the data and deal with the consequences later because there's none so far. I hope data poisoning see more traction as a possible countermeasure.

01HNNWZ0MV43FF 2y ago
I wonder what their data cleaning process looks like. If I put a lot of furry porn on my website, would it get excluded from the data sets?
- mtvrs 2y ago
  Probably only those data sets which do not include furry porn but it's also possible that it could be used in negative data set.

Gerardo1 2y ago

Is there any indication that the owners of those crawlers intend for them to play nice? I think this form the article is telling: > The Register asked Schubert about this in early January. "Funnily enough, a few days after the post went viral, all crawling stopped," he responded at the time. "Not just on the Diaspora wiki, but on my entire infrastructure. I'm not entirely sure why, but here we are." So the owners of those crawlers certainly have the ability to stop the traffic, they're just choosing not too...until publicity makes their actions risky.

entropi 2y ago

The owners don't even care about copyright and their deep-pocketed owners. They will certainly not care about playing nice with random websites, or any conventions that may exist in any domain.

ohgr 2y ago

I just shut my entire web site down after 27 years this year. All traffic was AI crawlers. Fuck it!

bayindirh 2y ago
Why not install Nepenthes and let them bath in the tar?
- ohgr 2y ago
  Spend more time and money? Nope. It's on CloudFront + S3 at the moment.

blueflow 2y ago

I set up a robots.txt with one entry: a disallow for a specific html file. The html file is not listed elsewhere and thus can only be discovered by the robots.txt. The file features random stuff and an <a href=""> element with the nofollow attribute and a link target set. That link has no text contents, no size, and thus doesn't occupy pixels that a user could click on. When clicked, that link would report the http request to me. At 2025-02-22 02:14:52, less than a day after i set up this honeypot, 66.249.68.37 (AS15169 GOOGLE) made a request on that invisible link, using user agent string "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.6943.53 Mobile Safari/537.36 (compatible; GoogleOther)" So now i have hard evidence that google runs bots that ignore the rules from the robots.txt, but still parses urls from there, queries these resources anyways and follows links marked as "nofollow".

bayindirh 2y ago
It's already well known that none of the search engines, crawlers, and companies honor robots.txt (anymore). It's just for us oldies who prefer the ethical web where people respected the RFCs they have collectively written.
- pimlottc 2y ago
  It’s one thing to know they don’t honor it, it’s another to know they intentionally abuse it
  - johnea 2y ago
    Yes, it is, and now we know both of those things don't we. What will be the consequence? My risky prediction: Absolutely Nothing, for neither goggle nor any of the other miscreants running the bot nets. And just to take a poke at the title of the original El Reg article: Do you really think the problem is that they just haven't "learned" yet how to "play nice"? I'm sure the great oracle goggle, just needs someone to poperly splain it to them.
extraduder_ire 2y ago

yodon 2y ago

As someone who has done a tiny bit of scraping, I've been surprised how difficult it is to be a good citizen. Are there packages you know of in the major language ecosystems that you can easily point at a both a robots.txt and an index.html (you need both to be compliant, most packages I've found only look at robots.txt) to get an answer on what you can and can't do at what rate? The only one I'm aware of is a C# package which still requires you to do a lot of the heavy lifting[0]. [0]https://github.com/TurnerSoftware/RobotsExclusionTools https://github.com/TurnerSoftware/RobotsExclusionTools

cavisne 2y ago
https://github.com/google/robotstxt https://github.com/google/robotstxt
- yodon 2y ago
  You're proving my point. That's a reference C++ parser that only parses robots.txt files. It doesn't parse meta tags and doesn't help with writing crawlers running in any of the major language ecosystems used by crawler authors. It's much more difficult than it should be for scrapers who want to be compliant to to actually be compliant.
mrweasel 2y ago
You obviously should read the robots.txt and follow it, but it's just as much about not just absolutely hammering a site over and over again, 24-7. Your scraper needs to learn about update patterns and adjust it's scraping interval. It needs to back off if the respons times increases. People are bringing up robots.txt because they want to ban AI scrapers, but if those same scrapers didn't constantly pound sites into the ground, it would be less of an issue.

esskay 2y ago

I manage a few thousand client sites spread across a bunch of differing infrastructures but one common thing we have on all of them is all the specific robots rules to block ai crawlers. Not a single one of them respects that request anymore. Anthropic is the worst at the moment. It starts hitting the site with an anthropic useragent. You can litterally see it hit the robots file, then it changes it's useragent to a generic browser one and carries on. I say at the moment because they're all doing this kind of crap, they seem to take it in turns at ramping up hammering servers.

onion2k 2y ago
I say at the moment because they're all doing this kind of crap... Maybe they used AI to code their agent, and it's just not that good.
pjc50 2y ago
> Not a single one of them respects that request anymore AI companies have seen the prisoner's dilemma of good internet citizenship and slammed hard on the "defect" button. They plan to steal your content, put it in an attribution-removing blender, then sell it to other people. With the explicit purpose of replacing human contribution on the internet and in the workplace. After all, they represent a reality-distorting amount of capital, so why should any rules apply to them?

leonidasv 2y ago

This is going to drive websites to put even more Cloudflare captcha-walls before you can even see the content. Web will become less accessible. I have a domain parked with just a blank page that's getting 172,000 requests per month. Since it's on a free-tier static hosting, that costs me nothing. However, if that trend continues, I don't know how long we will have those free static hosting options available for everyone.

deadbabe 2y ago
Send a zip bomb
- deleted 2y ago
  [deleted]
- jonatron 2y ago
  It didn't seem to matter what response I sent to PerplexityBot, the requests kept coming. I guess it's not surprising that a badly written bot isn't monitored either.
  - reginald78 2y ago
    Can you use hashcash or some proof of work system somehow? I don't know anything about this space.
    - jonatron 2y ago
      They were only requesting the HTML, not using a headless browser.
  - Dwedit 2y ago
    There was the tarpit that sends out markov-chain-generated random words, loading very slowly . If the client has a limited number of concurrent requests, that could theoretically choke the client.
  - JohnFen 2y ago

discardedrefuse 2y ago

This headline sucks. It suggest that AI crawlers' bad behavior isn't intentional. Robots.txt has existed for ages, and crawler operators know exactly how to play nice. They simply choose not to. Media outlets need to stop attributing ignorance or incompetence to what is clearly willful action. These megacorps need to be held accountable, even on the little things.

soco 2y ago
Technically it's true - they have not learned, but because nobody taught them to. Because nobody slapped any over-reaching wrist, ever, and if somebody even just thinks about, then it's all "damn EU bureaucrats".
mrweasel 2y ago
> It suggest that AI crawlers' bad behaviour isn't intentional. The robots.txt is only part of the problem, simply overloading sites is another. Maybe you don't really mind that your site is getting scraped, but you do mind that it crashes due to poorly written scrapers. It was always understood that your scrappers should limit their load to something that a site can handle. Google has always excelled at this, I've never experienced the GoogleBot being an issue, nor the BingBot. The developers who work at the AI companies however are less talented and less careering. If their bot crashes a site, meeeh, try again later. I don't think they DDoS sites deliberately, they just don't have the skills to fix their shitty scrapper. If we want the problem to go away, ask AWS, Azure, GCP, Alibaba Cloud and others for a way to report bad behaviour. Make their hosting providers take action.
- danaris 2y ago
  I will go one step further: It doesn't matter whether they have the skills or not, because they don't care. As long as they can siphon up enough data to increase their revenue or stock price by 0.001¢, they're perfectly content to leave small independent websites shattered and unusable under the weight of their aggressive, unrelenting scraper bots. I'm firmly of the opinion that they should all be considered hostile, predatory companies, regardless of what supposed "value" they bring. Train your river-guzzling models with data that's paid for or given voluntarily and then we can talk about your "value".

josefritzishere 2y ago

I just update my robots.txt to block AI crawlers just so I cna join the future inevitable class action lawsuit.

tempodox 2y ago

If you rip everyone off anyway, why would you bother doing it “the nice way”?

boredhacker3 2y ago

Why don’t they use an anti-bot solution like Cloudflare? It won’t block everything entirely, but it will at least stop low-quality bots

deleted 2y ago

[deleted]

deleted 2y ago

[deleted]

CraigRood 2y ago

I've personally noticed loads of crawls too, but what has been interesting is they have been crawling effectively the same pages, over and over, for months. Also, they don't seem to be caching DNS at all. I guess I'm in the minority that I don't care too much about crawlers, even the use of my work for AI, but I think the current hit rates just can't be justified.

elpocko 2y ago

>SourceHut says it's getting DDoSed by LLM bots How do they know it's LLM "bots"? Did they provide any proof? Is it crawlers or automated user agents? The guy behind SH hates all "AI" and is vocal about it; it would be reassuring to see some evidence, all I've seen is them saying "it's those pesky AI bots!"

dmje 2y ago

Compared to others in the comments (with "a few thousand sites"!!) we're obviously small beer, with maybe 50 or so - but we are being botted to death and it's driving us fucking nuts :-/

EdiX 2y ago

If search engines were invented today they wouldn't stand a chance. People would complain that they can return immoral results. Different sets of people would complain that they can return links to various isms that offend them. The owners would be worried that they return results critical of themselves or their friends. Copyright owners would sue them for linking to piracy. Website owners would block them for reading their precious content.