10 ms·

AI web crawlers are destroying websites in their never-ending content hunger

213 points by CrankyBear 1y ago

k310 1y ago
> Cloud services company Fastly agrees. It reports that 80% of all AI bot traffic comes from AI data fetcher bots. No kidding. An increasing number of sites are putting up CAPTCHA's. Problem? CAPTCHAS are annoying, they're a 50 times a day eye exam, and > Google's reCAPTCHA is not only useless, it's also basically spyware [0] > reCAPTCHA v3's checkbox test doesn't stop bots and tracks user data [0] https://www.techspot.com/news/106717-google-recaptcha-not-only-useless-also-basically-spyware.html https://www.techspot.com/news/106717-google-recaptcha-not-on...
- superkuh 1y ago
  And because companies like Fastly only measure things via javascript execution and assume everything that doesn't execute JS correctly is a bot, that 80% contains a whole bunch of human persons.
- ronsor 1y ago
  I've just started clicking away from pages that are full of CAPTCHAs. Ironically this has resulted in me using AI more.
- marginalia_nu 1y ago
  Webmasters are really kinda stuck between a rock and a hard place with this one. At least with what I'm doing poorly configured or outright malicious bots consume about 5000x the resources than human visitors, so having no bot mitigation means I've basically given up and decided I should try to make it as a vegetable farmer instead of doing stuff online. Bot mitigation in practice is a tradeoff between what's enough of an obstacle to keep most of the bots out, while at the same time not annoying the users so much they leave. I think right now Anubis is one of the less bad options. Some users are annoyed by it (and it is annoying), but it's less annoying than clicking fire hydrants 35 times and as long as you configure right it seems to keep most of the bots out, or at least drives them to behave in a more identifiable manner. Probably won't last forever, but I don't know what would besides like going full anacap special needs kid and doing crypto microtransactions for each page request. Would unfortunately drive off not only the bots, but the human visitors as well.
  - timpera 1y ago
    Anubis is extremely slow on low-end devices, it often takes >30 seconds to complete. Users deserve better, but I guess it's still a better experience than reCaptcha or Cloudflare.
    - danudey 1y ago
      Well, >30 seconds to complete anubis is still better than >30 seconds to complete every single page load because AI bots are overloading the servers.
- ccgreg 1y ago
  The Fastly report[1] has a couple of great quotes that mention Common Crawl's CCBot: > Our observations also highlight the vital role of open data initiatives like Common Crawl. Unlike commercial crawlers, Common Crawl makes its data freely available to the public, helping create a more inclusive ecosystem for AI research and development. With coverage across 63% of the unique websites crawled by AI bots, substantially higher than most commercial alternatives, it plays a pivotal role in democratizing access to large-scale web data. This open-access model empowers a broader community of researchers and developers to train and improve AI models, fostering more diverse and widespread innovation in the field. ... > What’s notable is that the top four crawlers (Meta, Google, OpenAI and Claude) seem to prefer Commerce websites. Common Crawl’s CCBot, whose open data set is widely used, has a balanced preference for Commerce, Media & Entertainment and High Tech sectors. Its commercial equivalents Timpibot and Diffbot seem to have a high preference for Media & Entertainment, perhaps to complement what’s available through Common Crawl. And also there's one final number that isn't in the Fastly report but is in the EL Reg article[2]: > The Common Crawl Project, which slurps websites to include in a free public dataset designed to prevent duplication of effort and traffic multiplication at the heart of the crawler problem, was a surprisingly-low 0.21 percent. 1: https://learn.fastly.com/rs/025-XKO-469/images/Fastly-Threat-Insights-Report.pdf https://learn.fastly.com/rs/025-XKO-469/images/Fastly-Threat... 2: https://www.theregister.com/2025/08/21/ai_crawler_traffic/ https://www.theregister.com/2025/08/21/ai_crawler_traffic/
- benjiro 1y ago
  Ironic part ... LLM are very good as solving CAPTCHA's. So the only people bothered by those same CAPTCHA's are the actual site visitors. What sites need to do is temp block repeat request from the same IPs. Sure, some agents use 10.000's of IP's but if they are really so aggressive as people state, your going to run into the same IP's way more often then normal users. That will kick out the over aggressive guys. I have done web scraping and limited it to around 1r/s. You never run into any blocking or detection that way because you hardly show up. But when you have some *** that send 1000's off parallel request down a website, because they never figured out query builders for large page hits. And do not know how to build checks to see from last-update pages. One of the main issues i see, is some people simply write the most basic of basic scrapers. See link, follow, spawn process, scrap, see 100 more links ... Updates? Just rescrap website, repeat, repeat... Because it takes time to make a scrap template for each website, that knows where to check for updated. So some never bother.
  - k310 1y ago
    I often use a VPN or iCloud private relay. Some sites gripe “too many accesses (downloads) from your IP address today.” The devil’s in the details. I (a non-bot) sometimes resort to VPN-flipping. I suppose that some bots try this, just a wild guess.
nextworddev 1y ago
“I drink your milkshake” type sh
onetokeoverthe 1y ago
captchas have eliminated 25 years of browsing speed progress.
- sidewndr46 1y ago
  And so the mantle has been passed from the Javascript developer, to the Turing test author.
throw_m239339 1y ago
My cousin manages a dozens of mid-sized informational websites and communities, his former hosting provider kicked him out because he refused to pay the insane bills as a result of literally AI bots DDoS-ing his sites... He unfortunately had no choice to put most of the content behind a login-wall (you can only see parts of the articles/forum posts when logged out) but he is strongly considering just hard pay-walling some content at that point... We're talking about someone who in good faith provided partial data dumps of content freely available for these companies to download, but, caching / etags? none of these AI companies, hiring "the best and the brightest" have ever heard of that, rate limiting? LOL what is that? This is nuts, these AI companies are ruining the web.
pluc 1y ago
People who didnt respect basic ethics, legal copyrights and common sense aren't gonna stop because they're a nuisance. They'll keep at it until they've ruined what birthed them so they may replace it. Fuck AI.
krunck 1y ago
I just block them by User Agent string[1]. The rest that fake the UA get clobbered by rate limiting[2] on the web server. Not perfect, but our site is not getting hammered any more. [1] https://perishablepress.com/ultimate-ai-block-list/ https://perishablepress.com/ultimate-ai-block-list/ [2] https://github.com/jzdziarski/mod_evasive https://github.com/jzdziarski/mod_evasive
- braden_e 1y ago
  There is a very large scale crawler that uses random valid user agents and a staggeringly large pool of ips. I first noticed it because a lot of traffic was coming from Brazil and "HostRoyale" (asn 203020). They send only a few requests a day from each ip so rate limiting is not useful. I run a honeypot that generates urls with the source IP so I am pretty confident it is all one bot, in the past 48 hours I have had over 200,000 ips hit the honeypot. I am pretty sure this is Bytedance, they occasionally hit these tagged honeypot urls with their normal user agent and their usual .sg datacenter.
  - kjkjadksj 1y ago
    I wonder if you could implement a dummy rate limit? Half the time you are rate limited randomly. A real user will think nothing of it and refresh the page.
    - ronsor 1y ago
      That will irritate real users half the time while the bots won't care.
      - kjkjadksj 1y ago
        If they are a real user going on your site in 2025 then they have no alternative they are even interested in. They will blame their ISP and wait. Meanwhile rate limiting the llm could potentially cost a lot of money in time and compute to people who don’t have our best interests at heart. Seems like a win to me.
  - 1y ago
lysace 1y ago
From my pov (saas, 1k domains): the most destructive/DDoS/idiotic brute force crawling peaked around half a year ago.
giancarlostoro 1y ago
I'm not sure why they don't just cache the websites and avoid going back for at least 24 hours, especially in the case of most sites. I swear its like we're re-learning software engineering basics with LLMs / AI and it kills me.
- kpw94 1y ago
  Yeah the landscpe when there were many more Search engines must have been exactly the same... I think the eng teams behind those were just more competent / more frugal on their processing. And since there wasn't any AWS equivalent, they had to be better citizens since well-known IP range ban for the crawled websites was trivial.
  - ccgreg 1y ago
    The blekko search engine index was only 1 billion pages, compared to Common Crawl Foundation's crawl of 3 billion webpages per month.
  - acdha 1y ago
    Bandwidth cost more then, so the early search engines had an inventive not to massively increase their own costs if nothing else.
  - danudey 1y ago
    It's worth noting that search engines back then (and now? except the AI ones) generally tended to follow robots.txt, which meant that if there were heavy areas of your site that you didn't want them to index you could filter them out and let them just follow static pages. You could block off all of /cgi-bin/ for example and then they would never be hitting your CGI scripts - useful if your guestbook software wrote out static files to be served, for example. The search engines were also limited in resources, so they were judicious about what they fetched, when, and how often; optimizing their own crawlers saved them money, and in return it also saved the websites too. Even with a hundred crawlers actively indexing your site, they weren't going to index it more than, say, once a day, and 100 requests in a day isn't really that much even back then. Now, companies are pumping billions of dollars into AI; budgets are infinite, limits are bypassed, and norms are ignored. If the company thinks it can benefit from indexing your site 30 times a minute then it will, but even if it doesn't benefit from it there's no reason for them to stop it from doing so because it doesn't cost them anything. They cannot risk being anything other than up-to-date, because if users are coming to you asking about current events and why space force is moving to Alabama and your AI doesn't know but someone else's does, then you're behind the times. So in the interests of maximizing short-term profit above all else - which is the only thing AI companies are doing in any way shape or form - they may as well scrape every URL on your site once per second, because it doesn't cost them anything and they don't care if you go bankrupt and shut down.
ChrisArchitect 1y ago
Blogspam just linking to a bunch of prior reports and posts from earlier in the year. Some ongoing recent discussion: Cloudflare Radar: AI Insights https://news.ycombinator.com/item?id=45093090 https://news.ycombinator.com/item?id=45093090 The age of agents: cryptographically recognizing agent traffic https://news.ycombinator.com/item?id=45055452 https://news.ycombinator.com/item?id=45055452 That Perplexity one: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives https://news.ycombinator.com/item?id=44785636 https://news.ycombinator.com/item?id=44785636 AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders https://news.ycombinator.com/item?id=44971487 https://news.ycombinator.com/item?id=44971487
verdverm 1y ago
At the same time, a lot of HN push back for new solutions like Signed Agents by CF https://news.ycombinator.com/item?id=45066258 https://news.ycombinator.com/item?id=45066258
- fruitworks 1y ago
  Because it's a bad solution. The core problem is that the internet is vulnerable to DDoS attacks and the web has no native sybil resistance mechanism. Cloudflare's solution to every problem is to allow them to control more of the internet. What happens when they have enough control to do whatever they want? They could charge any price they want.
  - add-sub-mul-div 1y ago
    I'm more afraid of the orgs that are gaining enough control of knowledge, cognition, and creativity that they'll be able to charge any price for them once they've trained us out of practicing them ourselves.
    - fruitworks 1y ago
      There Is No Moat
      - wongarsu 1y ago
        Until crawling without being on people's whitelist becomes sufficiently difficult
  - marginalia_nu 1y ago
    The idea itself has merit, even if the implementation is questionable. Giving bots a cryptographic identity would allow good bots to meaningfully have skin in the game and crawl with their reputation at stake. It's not a complete solution, but could be part of one. Though you can likely get the good parts from HTTP request signing alone, Cloudflare's additions to that seem fairly extraneous. I honestly don't know what is a good solution. The status quo is certainly completely untenable. If we keep going like we are now, there won't be a web left to protect in a few years. It's worth keeping in mind that there's an opportunity cost, and even a bad solution may be preferrable to no solution at all. ... I say operating an independent web crawler.
tmaly 1y ago
What if content providers reduced the 30k word page for a recipe down to just the actual recipe, would this reduce the amount of data these bots are pulling down? I don't see this slowing down. If websites don't adapt to the AI deep search reality, the bot will just go somewhere else. People don't want to read these massive long form pages geared at outdated Google SEO techniques.
- Fade_Dance 1y ago
  You're painting this as a problem that is somehow related to overly long form text based web pages. It isn't. If you host a local cleaning company site, or a game walkthrough site, or a roleplaying forum, the bots will flood the gates all the same. You are right that it doesn't look like it is slowing down, but the developing result of this will not be people posting a shorter recipe, it will be a further contraction of the public facing, open internet.
  - hombre_fatal 1y ago
    Kinda funny to see someone casually mention a roleplaying forum. That's what I run and it got 10x traffic overnight from AI bots. Made it when I was a teenager and got stuck running it the rest of my life. Of course, the bots go super deep into the site and bust your cache.
    - lawlessone 1y ago
      you'd be within your right to serve the bots fake gibberish data. Maybe they'll crawl less when it starts damaging models.
      - Analemma_ 1y ago
        That's only a stopgap measure, eventually they'll realize what's happening and use distributed IPs and fake user agents to look like normal users. The Tencent and Bytedance scrapers are already doing this.
- 1y ago
NickJLange 1y ago
Coming soon to a future ShowHN: https://www.open-embeddings.org/ https://www.open-embeddings.org/ The hope is to flip the incentives and feed the bots without drowning content publishers
bdefore 1y ago
I created and maintain ProtonDB, a popular Linux gaming resource. I don't do ads, just pay the bills from some Patreon donations. It's a statically generated React site I deploy on Netlify. About ten days ago I started incurring 30GB of data per day from user agents indicating they're using Prerender. At this pace almost all of that will push me past the 1TB allotted for my plan, so I'm looking at an extra ~$500USD a month for the extra bandwdith boosters. I'm gonna try the robots.txt options, but I'm doubtful this will be effective in the long run. Many other options aren't available if I want to continue using a SaaS like Netlify. My initial thoughts are to either move to Cloudflare Pages/Workers where bandwidth is unlimited, or make an edge function that parses the user agent and hope it's effective enough. That'd be about $60 in edge function invocations. I've got so many better things to do than play whack-a-mole on user agents and, when failing, pay this scraping ransom. Can I just say fuck all y'all AI harvesters? This is a popular free service that helps get people off of their Microsoft dependency and live their lives on a libre operating system. You wanna leech on that? Fine, download the data dumps I already offer on an ODbL license instead of making me wonder why I fucking bother.
- gjsman-1000 1y ago
  Your mistake is openly suggesting on HN that you're going to use Cloudflare, increasing the centralization of the internet and contributing to their attestation schemes, while society forces you to be a victim of the tragedy of the commons.
  - bdefore 1y ago
    Please believe me that it is not a step I want to take.
    - gjsman-1000 1y ago
      You don't need to apologize - HN needs to get their heads out of the sand that not everything is a tragedy of the commons, there's a reason why centralization exists, and the decentralized internet as it is now comes with serious drawbacks. We're never going to overcome the popularity of big tech if we can't be honest with the problems they solve. Also, sue me, the cathedral has defeated the bazaar. This was predictable, as the bazaar is a bunch of stonecutters competing with each other to sell the best stone for building the cathedral with. We reinvented the farmer's market, and thought that if all the farmers united, they could take down Walmart. It's never happening.
thaumaturgy 1y ago
People outside of a really small sysadmin niche really don't grasp the scale of this problem. I run a small-but-growing boutique hosting infrastructure for agency clients. The AI bot crawler problem recently got severe enough that I couldn't just ignore it anymore. I'm stuck between, on one end, crawlers from companies that absolutely have the engineering talent and resources to do things right but still aren't, and on the other end, resource-heavy WordPress installations where the client was told it was a build-it-and-forget-it kind of thing. I can't police their robots.txt files; meanwhile, each page load can take a full 1s round trip (most of that spent in MySQL), there are about 6 different pretty aggressive AI bots, and occasionally they'll get stuck on some site's product variants or categories pages and start hitting it at a 1r/s rate. There's an invisible caching layer that does a pretty nice job with images and the like, so it's not really a bandwidth problem. The bots aren't even requesting images and other page resources very often; they're just doing tons and tons of page requests, and each of those is tying up a DB somewhere. Cumulatively, it is close to having a site get Slashdotted every single day. I finally started filtering out most bot and crawler traffic at nginx, before it gets passed off to a WP container. I spent a fair bit of time sampling traffic from logs, and at a rough guess, I'd say maybe 5% of web traffic is currently coming from actual humans. It's insane. I've just wrapped up the first round of work for this problem, but that's just buying a little time. Now, I've gotta put together an IP intelligence system, because clearly these companies aren't gonna take "403" for an answer.
- gjsman-1000 1y ago
  I might write a blog post on this, but I seriously believe we collectively need to rethink The Cathedral and the Bazaar. The Cathedral won. Full stop. Everyone, more or less, is just a stonecutter, competing to sell the best stone (i.e. content, libraries, source code, tooling) for building the cathedrals with. If the world is a farmer's market, we're shocked that the farmer's market is not defeating Walmart, and never will. People want Cathedrals; not Bazaars. Being a Bazaar vendor is a race to the bottom. This is not the Cathedral exploiting a "tragedy of the commons," it's intrinsic to decentralization as a whole. The Bazaar feeds the Cathedral, just as the farmers feed Walmart, just as independent websites feed Claude, a food chain and not an aberration.
dizlexic 1y ago
I was sysadmining a virtual art gallery thousands of "exhibits" including sound, video, and images. We had never had any issue before and suddenly we get taken down 3 times in as many days. When I investigated it was all claude. They were just pounding every route regardless of timeouts with no throttle. It was nasty. They give web scrapers a bad rep.
- dylan604 1y ago
  Web scrapers earned their bad rep all on their own thank you very much. This is nothing new. Scrapers have no concern about whether a site is mostly static with stale text vs constantly updated. Most sites are not FB/Twitt..er,X/etc. Even retail sites not Amazon don't have new products being listed every minute. But that would involve someone on the scraper's side to pay attention and instead just let the computer run even if it is reading the same data every time. Even if sites offered their content in a single downloadable file for bots, the bot creators would not trust it is not stale and out of date so they'd still continue to scrape ignoring the easy method.
RebeccaTheDev 1y ago
I'll add my voice to others here that this is a huge problem especially for small hobbyist websites. I help administer a somewhat popular railroading forum. We've had some of these AI crawlers hammering the site to the point that it became unusable to actual human beings. You design your architecture around certain assumptions, and one of those was definitely not "traffic quintuples." We've ended up blocking lots of them, but it's a neverending game of whack-a-mole.
- benjiro 1y ago
  > one of those was definitely not "traffic quintuples." O, it was... People warned about the mass usage of WordPress because of its performance issues. The internet usage kept growing, even without LLM scraping in mass. Everybody wants more and more up to date info, recent price checks, and so many other features. This trend has been going on for over 10+ years. Its just now, that bot scraping for LLMs has pushed some sites over the edge. > We've ended up blocking lots of them, but it's a neverending game of whack-a-mole. And unless you block every IP, you can not stop them. Its really easy to hide scrapers, especially if you use a slow scrap rate. The issue comes when you have like one of the posters here, a setup where a DB call takes up to 1s for some product pages that are not in cache. Those sites already lived on borrowed time. Ironically, better software on their site (like not using WP), will allow them to handle easily 1000x the volume for the same resources. And do not get me started in how badly configured a lot of sites are in the backend. People are kind of blaming the wrong issue. Our needs for up to date, data, has been growing for over the last 10 years. Its just that people considered website that took 400ms to generate a webpage as ok. (when in reality they are wasting tons of resource or are limited in the backend)
0cf8612b2e1e 1y ago
This has been widely reported for months now. Anthropic just reported another $13B in funding. Clearly, the companies just do not care to invest any effort to improving their behavior.
idle_zealot 1y ago
This is something I have a hard time understanding. What is the point of this aggressive crawling? Gathering training data? Don't we already have massive repos of scraped web data being used for search indexing? Is this a coordination issue, each little AI startup having to scrape its own data because nobody is willing to share their stuff as regular dumps? For Wikipedia we have the official offline downloads, for books we have books3, but there's not an equivalent for the rest of the web? Could this be solved by some system where website operators submit text copies of their sites to a big database? Then in robots.txt or similar add a line that points to that database with a deep link to their site's mirrored content? The obvious issues are: a) who would pay to host that database. b) Sites not participating because they don't want their content accessible by LLMs for training (so scraping will still provide an advantage over using the database). c) The people implementing these scrapers are unscrupulous and just won't bother respecting sites that direct them to an existing dumped version of their content. d) Strong opponents to AI will try poisoning the database with fake submissions... Or does this proposed database basically already exist between Cloudflare and the Internet Archive, and we already know that the scrapers are some combination of dumb and belligerent and refuse to use anything but the live site?
- drozycki 1y ago
  I asked Google AI Mode “does Google ai mode make tens of site requests for a single prompt” and it showed “Looking at 69 sites” before giving a response about query fan-out. Cloudflare has some large part of the web cached, IA takes too long to respond and couldn’t handle the load. Google/OpenAI and co could cache these pages but apparently don’t do it aggressively enough or at all
  - ccgreg 1y ago
    I don't think you're correct about Google. Caching webpages is bread-and-butter for search engines, that's how they show snippets.
    - danudey 1y ago
      They might cache it, but what if it changed in the last 30 seconds and now their information is out of date? Better make another request just in case.
dlcarrier 1y ago
It's really bad for anyone using anything other than Chrome to browse the web, or any accessability tools or privacy software, because a bunch of sites will now block you, assuming you're a web crawler.
deleted 1y ago
[deleted]
mat_b 1y ago
Is this data being collected for training sets? That seems problematic. I can't be the only one who's noticed that the web is quickly filling up with AI generated clickbait (which has made using a search engine more difficult).
- rasz 1y ago
  Today most is probably live crawling during AI query resolution.
  - asddubs 1y ago
    no, you get tons of infinite link web following
1vuio0pswjnm7 1y ago
"It used to be when search indexing crawler, Googlebot, came calling, I could always hope that some story on my site would land on the magical first page of someone's search results so they'd visit me, they'd read the story, and two or three times out of a hundred visits, they'd click on an ad, and I'd get a few pennies of income." Perhaps the AI crawlers can "click on some ads"
neoromantique 1y ago
I think we are just reaping the delayed storm of the insanely inefficient web we have created over the past decades. There is absolutely no need for vast majority of websites to use databases and SSR, most of the web can be statically rendered and cost peanuts to host, but alas WP is the most popular "framework"
badgersnake 1y ago
Hope we’re including this in the energy usage for LLMs.
TheServitor 1y ago
My site isn't cool enough to get slammed by crawlers. Except for one Chinese bot that just will not give up. Petal bot? Something like that. I have Cloudflare's anti-bot thing turned on and OpenAI and Anthropic appear to either respect my rule or be stopped by it.
altcognito 1y ago
Can I ask a stupid question? Why is this so much worse than what they were doing to gather articles for traditional search engines? I assume that they are gathering pretty much the same data? It is the same articles, no? — I just realized these are callouts from the LLM on behalf of the client. I can see how this is problematic but it does seem like there should be a way to cache that
- splitbrain 1y ago
  No, the traffic is not caused by client requests (like when your chat gpt session does a search and checks some sources). They are caused by training runs. The difference is that AI companies are not storing the data they scrape. They let the model ingest the data, then throw it away. When they train the next model, they scrape the entire Internet again. At least that's how I understand it.
- zipy124 1y ago
  There's many factors but the largest are that it comes down to the fact there weren't many search companies, and they weren't that well capitalised. This meant there wasn't really competition for "freshness" in your results. There are many many many AI companies, and even more AI data companies providing the data to those doing the actual training. Finally search engines don't actually cache all the text, but do something akin to calculating embeddings/keywords and stuff like pagerank which just uses links. AI companies however want ALL the text/image/video data, and it's too expensive to store this all. It is however cheap to download it every time you need it. (Data ingress is usually free, as opposed to data egress)
yuriks 1y ago
The world today is just crooks all the way down (or up, depending on how you look at it).
senecaso 1y ago
Why don't sites just start publishing a dump of their site that crawlers could pull instead? I realize that won't work for dynamic content, but surely a lot of these "small" sites that are out there which are currently getting hammered, are not purely dynamic content? Maybe we could just publish a dump, in a standard format (WARC?), at a well-known address, and have the crawlers check there? The content could be regularly updated, and use an etag/etc so that crawlers know when its been updated. I suspect that even some dynamic sites could essentially snapshot themselves periodically, maybe once every few hours, and put it up for download to satiate these crawlers while keeping the bulk of the serving capacity for actual humans.
- krapp 1y ago
  Because crawlers aren't concerned about the bandwidth of the sites they crawl and will simply continue to take everything, everywhere, all the time regardless of what sites do. Also it's unfair to expect every small site to put in the time and effort to, in essence, pay the Danegeld to AI companies just for the privilege of their continued existence. It shouldn't be the case that the web only exists to feed AI, or that everyone must design their sites around feeding AI.
theshrike79 1y ago
Can we get an AI-powered tarpit for these crawlers? Add a hidden link, put it in robots.txt A crawler hits that link, a light-on-resources language model produces infinite amounts of plausible-looking gibberish for them to crawl with links and everything.

213 points by CrankyBear 1y ago

k310 1y ago

> Cloud services company Fastly agrees. It reports that 80% of all AI bot traffic comes from AI data fetcher bots. No kidding. An increasing number of sites are putting up CAPTCHA's. Problem? CAPTCHAS are annoying, they're a 50 times a day eye exam, and > Google's reCAPTCHA is not only useless, it's also basically spyware [0] > reCAPTCHA v3's checkbox test doesn't stop bots and tracks user data [0] https://www.techspot.com/news/106717-google-recaptcha-not-only-useless-also-basically-spyware.html https://www.techspot.com/news/106717-google-recaptcha-not-on...

superkuh 1y ago
And because companies like Fastly only measure things via javascript execution and assume everything that doesn't execute JS correctly is a bot, that 80% contains a whole bunch of human persons.
ronsor 1y ago
I've just started clicking away from pages that are full of CAPTCHAs. Ironically this has resulted in me using AI more.
marginalia_nu 1y ago
Webmasters are really kinda stuck between a rock and a hard place with this one. At least with what I'm doing poorly configured or outright malicious bots consume about 5000x the resources than human visitors, so having no bot mitigation means I've basically given up and decided I should try to make it as a vegetable farmer instead of doing stuff online. Bot mitigation in practice is a tradeoff between what's enough of an obstacle to keep most of the bots out, while at the same time not annoying the users so much they leave. I think right now Anubis is one of the less bad options. Some users are annoyed by it (and it is annoying), but it's less annoying than clicking fire hydrants 35 times and as long as you configure right it seems to keep most of the bots out, or at least drives them to behave in a more identifiable manner. Probably won't last forever, but I don't know what would besides like going full anacap special needs kid and doing crypto microtransactions for each page request. Would unfortunately drive off not only the bots, but the human visitors as well.
- timpera 1y ago
  Anubis is extremely slow on low-end devices, it often takes >30 seconds to complete. Users deserve better, but I guess it's still a better experience than reCaptcha or Cloudflare.
  - danudey 1y ago
    Well, >30 seconds to complete anubis is still better than >30 seconds to complete every single page load because AI bots are overloading the servers.
ccgreg 1y ago
The Fastly report[1] has a couple of great quotes that mention Common Crawl's CCBot: > Our observations also highlight the vital role of open data initiatives like Common Crawl. Unlike commercial crawlers, Common Crawl makes its data freely available to the public, helping create a more inclusive ecosystem for AI research and development. With coverage across 63% of the unique websites crawled by AI bots, substantially higher than most commercial alternatives, it plays a pivotal role in democratizing access to large-scale web data. This open-access model empowers a broader community of researchers and developers to train and improve AI models, fostering more diverse and widespread innovation in the field. ... > What’s notable is that the top four crawlers (Meta, Google, OpenAI and Claude) seem to prefer Commerce websites. Common Crawl’s CCBot, whose open data set is widely used, has a balanced preference for Commerce, Media & Entertainment and High Tech sectors. Its commercial equivalents Timpibot and Diffbot seem to have a high preference for Media & Entertainment, perhaps to complement what’s available through Common Crawl. And also there's one final number that isn't in the Fastly report but is in the EL Reg article[2]: > The Common Crawl Project, which slurps websites to include in a free public dataset designed to prevent duplication of effort and traffic multiplication at the heart of the crawler problem, was a surprisingly-low 0.21 percent. 1: https://learn.fastly.com/rs/025-XKO-469/images/Fastly-Threat-Insights-Report.pdf https://learn.fastly.com/rs/025-XKO-469/images/Fastly-Threat... 2: https://www.theregister.com/2025/08/21/ai_crawler_traffic/ https://www.theregister.com/2025/08/21/ai_crawler_traffic/
benjiro 1y ago
Ironic part ... LLM are very good as solving CAPTCHA's. So the only people bothered by those same CAPTCHA's are the actual site visitors. What sites need to do is temp block repeat request from the same IPs. Sure, some agents use 10.000's of IP's but if they are really so aggressive as people state, your going to run into the same IP's way more often then normal users. That will kick out the over aggressive guys. I have done web scraping and limited it to around 1r/s. You never run into any blocking or detection that way because you hardly show up. But when you have some *** that send 1000's off parallel request down a website, because they never figured out query builders for large page hits. And do not know how to build checks to see from last-update pages. One of the main issues i see, is some people simply write the most basic of basic scrapers. See link, follow, spawn process, scrap, see 100 more links ... Updates? Just rescrap website, repeat, repeat... Because it takes time to make a scrap template for each website, that knows where to check for updated. So some never bother.
- k310 1y ago
  I often use a VPN or iCloud private relay. Some sites gripe “too many accesses (downloads) from your IP address today.” The devil’s in the details. I (a non-bot) sometimes resort to VPN-flipping. I suppose that some bots try this, just a wild guess.

nextworddev 1y ago

“I drink your milkshake” type sh

onetokeoverthe 1y ago

captchas have eliminated 25 years of browsing speed progress.

sidewndr46 1y ago
And so the mantle has been passed from the Javascript developer, to the Turing test author.

throw_m239339 1y ago

My cousin manages a dozens of mid-sized informational websites and communities, his former hosting provider kicked him out because he refused to pay the insane bills as a result of literally AI bots DDoS-ing his sites... He unfortunately had no choice to put most of the content behind a login-wall (you can only see parts of the articles/forum posts when logged out) but he is strongly considering just hard pay-walling some content at that point... We're talking about someone who in good faith provided partial data dumps of content freely available for these companies to download, but, caching / etags? none of these AI companies, hiring "the best and the brightest" have ever heard of that, rate limiting? LOL what is that? This is nuts, these AI companies are ruining the web.

pluc 1y ago

People who didnt respect basic ethics, legal copyrights and common sense aren't gonna stop because they're a nuisance. They'll keep at it until they've ruined what birthed them so they may replace it. Fuck AI.

krunck 1y ago

I just block them by User Agent string[1]. The rest that fake the UA get clobbered by rate limiting[2] on the web server. Not perfect, but our site is not getting hammered any more. [1] https://perishablepress.com/ultimate-ai-block-list/ https://perishablepress.com/ultimate-ai-block-list/ [2] https://github.com/jzdziarski/mod_evasive https://github.com/jzdziarski/mod_evasive

braden_e 1y ago
There is a very large scale crawler that uses random valid user agents and a staggeringly large pool of ips. I first noticed it because a lot of traffic was coming from Brazil and "HostRoyale" (asn 203020). They send only a few requests a day from each ip so rate limiting is not useful. I run a honeypot that generates urls with the source IP so I am pretty confident it is all one bot, in the past 48 hours I have had over 200,000 ips hit the honeypot. I am pretty sure this is Bytedance, they occasionally hit these tagged honeypot urls with their normal user agent and their usual .sg datacenter.
- kjkjadksj 1y ago
  I wonder if you could implement a dummy rate limit? Half the time you are rate limited randomly. A real user will think nothing of it and refresh the page.
  - ronsor 1y ago
    That will irritate real users half the time while the bots won't care.
    - kjkjadksj 1y ago
      If they are a real user going on your site in 2025 then they have no alternative they are even interested in. They will blame their ISP and wait. Meanwhile rate limiting the llm could potentially cost a lot of money in time and compute to people who don’t have our best interests at heart. Seems like a win to me.
- 1y ago

lysace 1y ago

From my pov (saas, 1k domains): the most destructive/DDoS/idiotic brute force crawling peaked around half a year ago.

giancarlostoro 1y ago

I'm not sure why they don't just cache the websites and avoid going back for at least 24 hours, especially in the case of most sites. I swear its like we're re-learning software engineering basics with LLMs / AI and it kills me.

kpw94 1y ago
Yeah the landscpe when there were many more Search engines must have been exactly the same... I think the eng teams behind those were just more competent / more frugal on their processing. And since there wasn't any AWS equivalent, they had to be better citizens since well-known IP range ban for the crawled websites was trivial.
- ccgreg 1y ago
  The blekko search engine index was only 1 billion pages, compared to Common Crawl Foundation's crawl of 3 billion webpages per month.
- acdha 1y ago
  Bandwidth cost more then, so the early search engines had an inventive not to massively increase their own costs if nothing else.
- danudey 1y ago
  It's worth noting that search engines back then (and now? except the AI ones) generally tended to follow robots.txt, which meant that if there were heavy areas of your site that you didn't want them to index you could filter them out and let them just follow static pages. You could block off all of /cgi-bin/ for example and then they would never be hitting your CGI scripts - useful if your guestbook software wrote out static files to be served, for example. The search engines were also limited in resources, so they were judicious about what they fetched, when, and how often; optimizing their own crawlers saved them money, and in return it also saved the websites too. Even with a hundred crawlers actively indexing your site, they weren't going to index it more than, say, once a day, and 100 requests in a day isn't really that much even back then. Now, companies are pumping billions of dollars into AI; budgets are infinite, limits are bypassed, and norms are ignored. If the company thinks it can benefit from indexing your site 30 times a minute then it will, but even if it doesn't benefit from it there's no reason for them to stop it from doing so because it doesn't cost them anything. They cannot risk being anything other than up-to-date, because if users are coming to you asking about current events and why space force is moving to Alabama and your AI doesn't know but someone else's does, then you're behind the times. So in the interests of maximizing short-term profit above all else - which is the only thing AI companies are doing in any way shape or form - they may as well scrape every URL on your site once per second, because it doesn't cost them anything and they don't care if you go bankrupt and shut down.

ChrisArchitect 1y ago

Blogspam just linking to a bunch of prior reports and posts from earlier in the year. Some ongoing recent discussion: Cloudflare Radar: AI Insights https://news.ycombinator.com/item?id=45093090 https://news.ycombinator.com/item?id=45093090 The age of agents: cryptographically recognizing agent traffic https://news.ycombinator.com/item?id=45055452 https://news.ycombinator.com/item?id=45055452 That Perplexity one: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives https://news.ycombinator.com/item?id=44785636 https://news.ycombinator.com/item?id=44785636 AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders https://news.ycombinator.com/item?id=44971487 https://news.ycombinator.com/item?id=44971487

verdverm 1y ago

At the same time, a lot of HN push back for new solutions like Signed Agents by CF https://news.ycombinator.com/item?id=45066258 https://news.ycombinator.com/item?id=45066258

fruitworks 1y ago
Because it's a bad solution. The core problem is that the internet is vulnerable to DDoS attacks and the web has no native sybil resistance mechanism. Cloudflare's solution to every problem is to allow them to control more of the internet. What happens when they have enough control to do whatever they want? They could charge any price they want.
- add-sub-mul-div 1y ago
  I'm more afraid of the orgs that are gaining enough control of knowledge, cognition, and creativity that they'll be able to charge any price for them once they've trained us out of practicing them ourselves.
  - fruitworks 1y ago
    There Is No Moat
    - wongarsu 1y ago
      Until crawling without being on people's whitelist becomes sufficiently difficult
- marginalia_nu 1y ago
  The idea itself has merit, even if the implementation is questionable. Giving bots a cryptographic identity would allow good bots to meaningfully have skin in the game and crawl with their reputation at stake. It's not a complete solution, but could be part of one. Though you can likely get the good parts from HTTP request signing alone, Cloudflare's additions to that seem fairly extraneous. I honestly don't know what is a good solution. The status quo is certainly completely untenable. If we keep going like we are now, there won't be a web left to protect in a few years. It's worth keeping in mind that there's an opportunity cost, and even a bad solution may be preferrable to no solution at all. ... I say operating an independent web crawler.

tmaly 1y ago

What if content providers reduced the 30k word page for a recipe down to just the actual recipe, would this reduce the amount of data these bots are pulling down? I don't see this slowing down. If websites don't adapt to the AI deep search reality, the bot will just go somewhere else. People don't want to read these massive long form pages geared at outdated Google SEO techniques.

Fade_Dance 1y ago
You're painting this as a problem that is somehow related to overly long form text based web pages. It isn't. If you host a local cleaning company site, or a game walkthrough site, or a roleplaying forum, the bots will flood the gates all the same. You are right that it doesn't look like it is slowing down, but the developing result of this will not be people posting a shorter recipe, it will be a further contraction of the public facing, open internet.
- hombre_fatal 1y ago
  Kinda funny to see someone casually mention a roleplaying forum. That's what I run and it got 10x traffic overnight from AI bots. Made it when I was a teenager and got stuck running it the rest of my life. Of course, the bots go super deep into the site and bust your cache.
  - lawlessone 1y ago
    you'd be within your right to serve the bots fake gibberish data. Maybe they'll crawl less when it starts damaging models.
    - Analemma_ 1y ago
      That's only a stopgap measure, eventually they'll realize what's happening and use distributed IPs and fake user agents to look like normal users. The Tencent and Bytedance scrapers are already doing this.
1y ago

NickJLange 1y ago

Coming soon to a future ShowHN: https://www.open-embeddings.org/ https://www.open-embeddings.org/ The hope is to flip the incentives and feed the bots without drowning content publishers

bdefore 1y ago

I created and maintain ProtonDB, a popular Linux gaming resource. I don't do ads, just pay the bills from some Patreon donations. It's a statically generated React site I deploy on Netlify. About ten days ago I started incurring 30GB of data per day from user agents indicating they're using Prerender. At this pace almost all of that will push me past the 1TB allotted for my plan, so I'm looking at an extra ~$500USD a month for the extra bandwdith boosters. I'm gonna try the robots.txt options, but I'm doubtful this will be effective in the long run. Many other options aren't available if I want to continue using a SaaS like Netlify. My initial thoughts are to either move to Cloudflare Pages/Workers where bandwidth is unlimited, or make an edge function that parses the user agent and hope it's effective enough. That'd be about $60 in edge function invocations. I've got so many better things to do than play whack-a-mole on user agents and, when failing, pay this scraping ransom. Can I just say fuck all y'all AI harvesters? This is a popular free service that helps get people off of their Microsoft dependency and live their lives on a libre operating system. You wanna leech on that? Fine, download the data dumps I already offer on an ODbL license instead of making me wonder why I fucking bother.

gjsman-1000 1y ago
Your mistake is openly suggesting on HN that you're going to use Cloudflare, increasing the centralization of the internet and contributing to their attestation schemes, while society forces you to be a victim of the tragedy of the commons.
- bdefore 1y ago
  Please believe me that it is not a step I want to take.
  - gjsman-1000 1y ago
    You don't need to apologize - HN needs to get their heads out of the sand that not everything is a tragedy of the commons, there's a reason why centralization exists, and the decentralized internet as it is now comes with serious drawbacks. We're never going to overcome the popularity of big tech if we can't be honest with the problems they solve. Also, sue me, the cathedral has defeated the bazaar. This was predictable, as the bazaar is a bunch of stonecutters competing with each other to sell the best stone for building the cathedral with. We reinvented the farmer's market, and thought that if all the farmers united, they could take down Walmart. It's never happening.

thaumaturgy 1y ago

People outside of a really small sysadmin niche really don't grasp the scale of this problem. I run a small-but-growing boutique hosting infrastructure for agency clients. The AI bot crawler problem recently got severe enough that I couldn't just ignore it anymore. I'm stuck between, on one end, crawlers from companies that absolutely have the engineering talent and resources to do things right but still aren't, and on the other end, resource-heavy WordPress installations where the client was told it was a build-it-and-forget-it kind of thing. I can't police their robots.txt files; meanwhile, each page load can take a full 1s round trip (most of that spent in MySQL), there are about 6 different pretty aggressive AI bots, and occasionally they'll get stuck on some site's product variants or categories pages and start hitting it at a 1r/s rate. There's an invisible caching layer that does a pretty nice job with images and the like, so it's not really a bandwidth problem. The bots aren't even requesting images and other page resources very often; they're just doing tons and tons of page requests, and each of those is tying up a DB somewhere. Cumulatively, it is close to having a site get Slashdotted every single day. I finally started filtering out most bot and crawler traffic at nginx, before it gets passed off to a WP container. I spent a fair bit of time sampling traffic from logs, and at a rough guess, I'd say maybe 5% of web traffic is currently coming from actual humans. It's insane. I've just wrapped up the first round of work for this problem, but that's just buying a little time. Now, I've gotta put together an IP intelligence system, because clearly these companies aren't gonna take "403" for an answer.

gjsman-1000 1y ago
I might write a blog post on this, but I seriously believe we collectively need to rethink The Cathedral and the Bazaar. The Cathedral won. Full stop. Everyone, more or less, is just a stonecutter, competing to sell the best stone (i.e. content, libraries, source code, tooling) for building the cathedrals with. If the world is a farmer's market, we're shocked that the farmer's market is not defeating Walmart, and never will. People want Cathedrals; not Bazaars. Being a Bazaar vendor is a race to the bottom. This is not the Cathedral exploiting a "tragedy of the commons," it's intrinsic to decentralization as a whole. The Bazaar feeds the Cathedral, just as the farmers feed Walmart, just as independent websites feed Claude, a food chain and not an aberration.

dizlexic 1y ago

I was sysadmining a virtual art gallery thousands of "exhibits" including sound, video, and images. We had never had any issue before and suddenly we get taken down 3 times in as many days. When I investigated it was all claude. They were just pounding every route regardless of timeouts with no throttle. It was nasty. They give web scrapers a bad rep.

dylan604 1y ago
Web scrapers earned their bad rep all on their own thank you very much. This is nothing new. Scrapers have no concern about whether a site is mostly static with stale text vs constantly updated. Most sites are not FB/Twitt..er,X/etc. Even retail sites not Amazon don't have new products being listed every minute. But that would involve someone on the scraper's side to pay attention and instead just let the computer run even if it is reading the same data every time. Even if sites offered their content in a single downloadable file for bots, the bot creators would not trust it is not stale and out of date so they'd still continue to scrape ignoring the easy method.

RebeccaTheDev 1y ago

I'll add my voice to others here that this is a huge problem especially for small hobbyist websites. I help administer a somewhat popular railroading forum. We've had some of these AI crawlers hammering the site to the point that it became unusable to actual human beings. You design your architecture around certain assumptions, and one of those was definitely not "traffic quintuples." We've ended up blocking lots of them, but it's a neverending game of whack-a-mole.

benjiro 1y ago
> one of those was definitely not "traffic quintuples." O, it was... People warned about the mass usage of WordPress because of its performance issues. The internet usage kept growing, even without LLM scraping in mass. Everybody wants more and more up to date info, recent price checks, and so many other features. This trend has been going on for over 10+ years. Its just now, that bot scraping for LLMs has pushed some sites over the edge. > We've ended up blocking lots of them, but it's a neverending game of whack-a-mole. And unless you block every IP, you can not stop them. Its really easy to hide scrapers, especially if you use a slow scrap rate. The issue comes when you have like one of the posters here, a setup where a DB call takes up to 1s for some product pages that are not in cache. Those sites already lived on borrowed time. Ironically, better software on their site (like not using WP), will allow them to handle easily 1000x the volume for the same resources. And do not get me started in how badly configured a lot of sites are in the backend. People are kind of blaming the wrong issue. Our needs for up to date, data, has been growing for over the last 10 years. Its just that people considered website that took 400ms to generate a webpage as ok. (when in reality they are wasting tons of resource or are limited in the backend)

0cf8612b2e1e 1y ago

This has been widely reported for months now. Anthropic just reported another $13B in funding. Clearly, the companies just do not care to invest any effort to improving their behavior.

idle_zealot 1y ago

This is something I have a hard time understanding. What is the point of this aggressive crawling? Gathering training data? Don't we already have massive repos of scraped web data being used for search indexing? Is this a coordination issue, each little AI startup having to scrape its own data because nobody is willing to share their stuff as regular dumps? For Wikipedia we have the official offline downloads, for books we have books3, but there's not an equivalent for the rest of the web? Could this be solved by some system where website operators submit text copies of their sites to a big database? Then in robots.txt or similar add a line that points to that database with a deep link to their site's mirrored content? The obvious issues are: a) who would pay to host that database. b) Sites not participating because they don't want their content accessible by LLMs for training (so scraping will still provide an advantage over using the database). c) The people implementing these scrapers are unscrupulous and just won't bother respecting sites that direct them to an existing dumped version of their content. d) Strong opponents to AI will try poisoning the database with fake submissions... Or does this proposed database basically already exist between Cloudflare and the Internet Archive, and we already know that the scrapers are some combination of dumb and belligerent and refuse to use anything but the live site?

drozycki 1y ago
I asked Google AI Mode “does Google ai mode make tens of site requests for a single prompt” and it showed “Looking at 69 sites” before giving a response about query fan-out. Cloudflare has some large part of the web cached, IA takes too long to respond and couldn’t handle the load. Google/OpenAI and co could cache these pages but apparently don’t do it aggressively enough or at all
- ccgreg 1y ago
  I don't think you're correct about Google. Caching webpages is bread-and-butter for search engines, that's how they show snippets.
  - danudey 1y ago
    They might cache it, but what if it changed in the last 30 seconds and now their information is out of date? Better make another request just in case.

dlcarrier 1y ago

It's really bad for anyone using anything other than Chrome to browse the web, or any accessability tools or privacy software, because a bunch of sites will now block you, assuming you're a web crawler.

deleted 1y ago

[deleted]

mat_b 1y ago

Is this data being collected for training sets? That seems problematic. I can't be the only one who's noticed that the web is quickly filling up with AI generated clickbait (which has made using a search engine more difficult).

rasz 1y ago
Today most is probably live crawling during AI query resolution.
- asddubs 1y ago
  no, you get tons of infinite link web following

1vuio0pswjnm7 1y ago

"It used to be when search indexing crawler, Googlebot, came calling, I could always hope that some story on my site would land on the magical first page of someone's search results so they'd visit me, they'd read the story, and two or three times out of a hundred visits, they'd click on an ad, and I'd get a few pennies of income." Perhaps the AI crawlers can "click on some ads"

neoromantique 1y ago

I think we are just reaping the delayed storm of the insanely inefficient web we have created over the past decades. There is absolutely no need for vast majority of websites to use databases and SSR, most of the web can be statically rendered and cost peanuts to host, but alas WP is the most popular "framework"

badgersnake 1y ago

Hope we’re including this in the energy usage for LLMs.

TheServitor 1y ago

My site isn't cool enough to get slammed by crawlers. Except for one Chinese bot that just will not give up. Petal bot? Something like that. I have Cloudflare's anti-bot thing turned on and OpenAI and Anthropic appear to either respect my rule or be stopped by it.

altcognito 1y ago

Can I ask a stupid question? Why is this so much worse than what they were doing to gather articles for traditional search engines? I assume that they are gathering pretty much the same data? It is the same articles, no? — I just realized these are callouts from the LLM on behalf of the client. I can see how this is problematic but it does seem like there should be a way to cache that

splitbrain 1y ago
No, the traffic is not caused by client requests (like when your chat gpt session does a search and checks some sources). They are caused by training runs. The difference is that AI companies are not storing the data they scrape. They let the model ingest the data, then throw it away. When they train the next model, they scrape the entire Internet again. At least that's how I understand it.
zipy124 1y ago
There's many factors but the largest are that it comes down to the fact there weren't many search companies, and they weren't that well capitalised. This meant there wasn't really competition for "freshness" in your results. There are many many many AI companies, and even more AI data companies providing the data to those doing the actual training. Finally search engines don't actually cache all the text, but do something akin to calculating embeddings/keywords and stuff like pagerank which just uses links. AI companies however want ALL the text/image/video data, and it's too expensive to store this all. It is however cheap to download it every time you need it. (Data ingress is usually free, as opposed to data egress)

yuriks 1y ago

The world today is just crooks all the way down (or up, depending on how you look at it).

senecaso 1y ago

Why don't sites just start publishing a dump of their site that crawlers could pull instead? I realize that won't work for dynamic content, but surely a lot of these "small" sites that are out there which are currently getting hammered, are not purely dynamic content? Maybe we could just publish a dump, in a standard format (WARC?), at a well-known address, and have the crawlers check there? The content could be regularly updated, and use an etag/etc so that crawlers know when its been updated. I suspect that even some dynamic sites could essentially snapshot themselves periodically, maybe once every few hours, and put it up for download to satiate these crawlers while keeping the bulk of the serving capacity for actual humans.

krapp 1y ago
Because crawlers aren't concerned about the bandwidth of the sites they crawl and will simply continue to take everything, everywhere, all the time regardless of what sites do. Also it's unfair to expect every small site to put in the time and effort to, in essence, pay the Danegeld to AI companies just for the privilege of their continued existence. It shouldn't be the case that the web only exists to feed AI, or that everyone must design their sites around feeding AI.

theshrike79 1y ago

Can we get an AI-powered tarpit for these crawlers? Add a hidden link, put it in robots.txt A crawler hits that link, a light-on-resources language model produces infinite amounts of plausible-looking gibberish for them to crawl with links and everything.