5 ms·

> we all know what happens if you just open your site up without these, especially with AI bots which hammer servers and are in effect a legalized DDoS system

by jwr 5mo ago

> we all know what happens if you just open your site up without these, especially with AI bots which hammer servers and are in effect a legalized DDoS system No, we don't know. I honestly do not understand the problem. I run websites, both static and non-static. Granted, my sites aren't exactly the most popular internet go-to destinations, but I should be seeing this DDoS too, right? I do see lots of requests. Nothing that any modern system can't handle. Computers are stupid fast these days. Unless you are doing something unreasonable, it's really hard to even notice this "extra load". I understand there are sites for whom this causes problems, but I think these are rare and could be optimized not to do unreasonable things. I think too many people are annoyed by AI companies (arguably understandable position), look at their logs and speak of "hammering", "DDoS" and "extra load", while in reality it doesn't matter much.

ethin 5mo ago
Has anyone pointed an AI scraper at your server at all? Unless your website appears in search engine listings I don't think the AI scrapers will slam it. My server has never been hit by them but my server is also practically unknown. All of this said, I'm not going to claim that server loads can handle it because many sysadmins have claimed otherwise, and I would like to think that their claims are reliable.
- redox99 5mo ago
  As soon as you get your TLS certificate you get bombarded with scraping. You don't need someone to "point a scraper at you". What matters most is usually how much there is to scrape. If you have like 5 pages that's nothing. For forum like websites where each thread, each user profile, etc. gets scraped that's when traffic increases. I just let them have at it with no issues though, computers are fast.
  - ethin 5mo ago
    That's really weird. My experience is quite different: I have several subdomains and all of them have TLS certs and I haven't (yet) seen this (thankfully). Either that, or my server is masking it. The weird thing is that my server is an OVH dedicated box that doesn't exactly have top-tier specs, so I have no idea what's going on there. Very weird indeed.
    - redox99 5mo ago
      Probably you don't have much to scrape?
      - ethin 5mo ago
        I mean... It may be that most of the things I run aren't really scrape-able. I run Matrix (which requires authentication), an XWiki instance, Zulip, Terraria, Forgejo, Nextcloud, a Mastodon server... Most of those require auth behind my Kanidm instance to actually do anything. Well and most of them have APIs that are much better than "scrape the universe".
  - GoblinSlayer 5mo ago
    If you run the site on a custom port, scrapers won't find it?
- userbinator 5mo ago
  Also, how do we even know they're really "AI scrapers", or just a deliberate DDoS to push sites into using CF or other "anti-bot" providers?
  - danielheath 5mo ago
    They showed up when the AI money did. The evidence is circumstantial, but… some of them are remarkably well engineered (from a “how difficult is it to identify this traffic” perspective, in a way that never existed before (I have been running a quite sizeable site for 8 years, over 200k registered users, and you don’t need to register to use 99% of it).
    - userbinator 5mo ago
      Yes, circumstantial is exactly the point; it's easy to use AI as a scapegoat because it's something popular to hate on.
      - danielheath 5mo ago
        It's circumstantial evidence, but Occam's Razor also applies. It's not a hostile DOS in the traditional sense (I've mitigated a few of those) - no "pay us to make it stop", no pattern to the requests other than "fetch every unique URL a few times". It wasn't happening until financial incentives to gather large datasets for AI training appeared. Bad actors (using residential proxies & claiming to be a real browser) mostly showed up after folk started blocking ones that identified themselves as AI scrapers. It's obvious to blame AI training because there's a shortage of better explanations. Who else would be paying for these (expensive) residential botnets, only to use them to (eg) web-scrape wikipedia (which offers free downloads of its content in a structured format)? The simplest explanation of the technical behavior is "a bot coded to follow every link it sees & save the results", and the simplest explanation of the motive to run such a bot is "to train a large language model".
        userbinator 5mo ago
redox99 5mo ago
You get downvoted for these opinions but I agree. Most people that complain that their servers get hammered by AI bots are those that run very unoptimized servers that can only handle like 100 rps. I've never had any issues with any of my moderately optimized websites. A $10 VPS can handle sooo much traffic.
- CodeBytes 5mo ago
  I think people get annoyed when it's suggested they spend time optimising or even re-writing their websites to handle high traffic loads just to cater to AI bots ripping their content. It's also not always easy to do. I run a small wiki which is fairly optimised, nearly every page manages at least ~3k rps on a small VPS. The only exception is the diff page which is ~150 rps. Optimising that while still giving good output isn't that easy, but the wiki doesn't have many users so that would be fine if it wasn't for the AI bots. The AI bots ignore robots.txt and were initially hitting the site with ~1k rps crawling every combination. Even that would be manageable as there's currently ~150,000 combinations, except they kept re-crawling the whole lot each day. The server could manage it but it was a massive waste of resources. They were using residential IPs and only sending 1 request from each IP making it impossible to block. In the end I gave up and put a Cloudflare challenge in front of it. I don't want to use Cloudflare but the alternative is forcing users to login to view diffs or remove them entirely.
  - canyp 5mo ago
    Curious, but how do the bots figure out the combinations? Or do you have links to the diffs from other sites? I assume the diff takes two files in query parameters or something.
    - CodeBytes 5mo ago
      I'm not 100% sure but I think links. There's a bunch on the history and revision pages. Yeah, the diff URL has two revision ID's as parameters. I did try removing some of the links without success. I guess once they have them they just keep checking.
  - 5mo ago
dr_um 5mo ago
A small, single EU country focused non-static e-commerce, with proper robots.txt instructions that worked perfectly well in the search & co bots -only "era" with rate limiting for nginx/php-fpm setup - is kinda struggling without CF to handle 15000 requests per 15 minutes, coming from Chrome "users" from IPv6. Best so far was an avg. server load in htop = 40 on an 8-core server x_x
- PunchyHamster 5mo ago
  > handle 15000 requests per 15 minutes, that's just ~17 req/sec That's "cheap VPS running wordpress" level of traffic
  - BenjiWiebe 5mo ago
    Maybe a plain WordPress install. Run something like WooCommerce and install a bunch of plugins to get the functionality that WordPress and WooCommerce should have built-in, and suddenly a cheap VPS can only handle 2 or 3 requests per second. It's phenomenal how inefficient the WordPress/WooCommerce stack is. Though the main issue I'm seeing is credit card testing, not scraping. And I'm ideologically opposed to using a CDN (because it shouldn't be needed for such a small site!) so it's somewhat a self-inflicted problem...
    - PunchyHamster 5mo ago
      "Security" plugins are also HUGE problem here, most of them turns "few cached DB SELECTs" (or static file read if you use caching plugin) into now a bunch of inserts, just to log/analyze "offender" IP and maybe block it, in many cases turning "blocking offender" to be more costly that would be serving the page without the security plugin
- canyp 5mo ago
  Block out IPv6 and see if that helps.
  - lxgr 5mo ago
matt_heimer 5mo ago
It might depend on the tech stack. I run a small niche website but it has PHP and a database (MediaWiki/PHPBB) and without Cloudflare I'd estimate I'd need to spend several hundred dollars a month to handle the traffic. Traffic used to be tens of thousands of requests a day. AI has increased that to between 400k and 3M requests per day but it's not a smooth distribution. This is with bot fight mode on that greatly reduces traffic. I adopted Cloudflare because it was getting DDoSed by the AI crawlers. I'm pretty sure all of them are vibe coding their crawlers and don't bother adding rate limiting as a requirement.
- jwr 5mo ago
  That was my point. I was trying to be gentle by mentioning "unreasonable" things, but seriously — how did we get to the point where less than 6 requests per second (that's 500k requests per day) is considered a DDoS? I've spent some effort on optimizing my sites, but most of the effort was focused on avoiding unreasonable (stupid) work. Do I need a session for every request? No, I don't! Do I need a database fetch for every access to my homepage? No, I don't! Is it a problem to actually load all of my static content in all supported languages (24) into memory and serve it from memory? No, it isn't! I use Clojure behind nginx on the server for my sites. Oh, and I also pre-compress all static assets to Brotli, so anything that handles brotli gets a static file served directly from nginx. I also use immutable assets with unlimited caching semantics. Really — the problem is that we've grown lax and our software has become bloated, slow, and with unreasonable code paths. If every page fetch does 12 database accesses and runs through a slow interpreter, that is surely going to be a problem.
  - matt_heimer 5mo ago
    That's the traffic after rate limiting controls and bot fight mode. It's 3-4 million requests per day without bot fight mode and just rate limits. And as I said it's not a smooth distribution. Plus the requests are almost never for pages in cache. It's always stuff like loading all the message threads from the year 2000 or loading up the details of every page edit ever made to a wiki page. If it was more static content it'd be easier, it's really the db being a bottle neck in a dynamic site. Yes, the software could be better optimized but then I'd have to own the development of it. There is no reason a niche website should be getting millions of requests per day.
canyp 5mo ago
I second this. My website exposes a cgit and 99% of the traffic now is AI scraping the sources, but the load is nowhere near DoS territory. And this is running on the cheapest VPS I could find. Not saying I'm not annoyed by the scraping; I am looking to block them, but I'm also not going to put the site behind the gatekeeper. If anything, Cloudflare must love AI scraping now for the same reason AV companies love malware. Now, if you are running a PHP stack...yeah, maybe that's the problem right there.
- lxgr 5mo ago
  Is there actually any plausible theory why "AI" would repeatedly scrape the same sites? Are there that many competing, completely independent AI labs? Is it cheaper to repeatedly scrape than to buffer the scraped data locally? (I find it very hard to imagine that it's easier to deal with changing/disappearing content than it is to stand up such a cache.)
  - jack_pp 5mo ago
    If you ask an agent to check sources / function definitions of open source packages it will wget / curl it
  - GoblinSlayer 5mo ago
    It's an AI generated scraper that scrapes nonstop.
- account42 5mo ago
  The PHP stack isn't even the problem, it's having unauthenticated requests getting past the cache in the first place, something that most sites should be able to prevent.
- ern_ave 5mo ago
  > 99% of the traffic now is AI scraping the sources I wonder if we should stop fighting this and instead create an API specifically for this purpose? Or, a central repository that you could send your data to and say to anyone wanting to scrape, "safe yourself some time and just get my data from this other place"
acdha 5mo ago
We do know, just ask anyone who runs a more popular site or does anything where abuse can be monetized (shopping, reviews, etc.). Avoiding that due to obscurity isn’t an answer because it’s saying you’re safe until something, possibly outside of your control, causes the bots to descend and give you an extra 500M requests with no chance of revenue. I’m with OP: I don’t like this but the alternatives all look like the death of the open web.
- handoflixue 5mo ago
  > just ask anyone who runs a more popular site The person you're responding to already said they ran a modestly sized site. What actual scale opens one up to abuse? If only the top 1% of sites need it, then it seems silly to say "everyone" needs it.
  - daishi55 5mo ago
    So everyone is paying cloudflare… why?
    - fragmede 5mo ago
      Most likely not. Their free tier is fairly generous.
    - tardedmeme 5mo ago
      Because paying with MITM is far less visible than paying with money
    - LgWoodenBadger 5mo ago
      Because charging for bandwidth/traffic is still a thing, unfortunately
  - ceejayoz 5mo ago
    It’s not just scale. Do you accept user generated content? If so, more of a target.
piker 5mo ago
Same. Tritium and the blog have done stents on the front page here and high traffic subreddits and that plus bots has never been a problem. UX could be improved through a CDN but even that isn’t worth the trade-off for us at the moment.
JohnTHaller 5mo ago
If you're in any way semi-popular and a decent size, you're gonna get hammered. PortableApps.com was partially offline for weeks due to China-based AI scrapers. You block the useragent, they start hitting you with another one from the same IP in the same way. You block the IP, they switch to another. You block the subnet, they use another. At one point it was nearly a thousand different IPs from around China hammering away. For all intents and purposes, a DDoS. This wasn't a little "extra load", this was load that was thousands of times beyond what our legitimate userbase was using. And if you're thinking about blocking all of China, while this particular AI bot didn't use them, a bunch of other ones I've encountered use VPNs and hacked clients worldwide.
RHSeeger 5mo ago
> I understand there are sites for whom this causes problems, but I think these are rare and could be optimized not to do unreasonable things. There are. They're not. They can't (without significant effort)
hombre_fatal 5mo ago
Consider yourself lucky. But don't let yourself fall into the trap of thinking it's a nonissue for everyone else until it happens to you. People shouldn't have to be experts or provision a larger server to run a UGC service that can withstand the sort of 30x more traffic I'm seeing from AI bots. Or rather, you didn't render the argument for why they should have to do that if they can just use CloudFlare's free tier. Either way, it's easy to have all the answers when you've never had the problem.

by jwr 5mo ago

ethin 5mo ago

Has anyone pointed an AI scraper at your server at all? Unless your website appears in search engine listings I don't think the AI scrapers will slam it. My server has never been hit by them but my server is also practically unknown. All of this said, I'm not going to claim that server loads can handle it because many sysadmins have claimed otherwise, and I would like to think that their claims are reliable.

redox99 5mo ago
As soon as you get your TLS certificate you get bombarded with scraping. You don't need someone to "point a scraper at you". What matters most is usually how much there is to scrape. If you have like 5 pages that's nothing. For forum like websites where each thread, each user profile, etc. gets scraped that's when traffic increases. I just let them have at it with no issues though, computers are fast.
- ethin 5mo ago
  That's really weird. My experience is quite different: I have several subdomains and all of them have TLS certs and I haven't (yet) seen this (thankfully). Either that, or my server is masking it. The weird thing is that my server is an OVH dedicated box that doesn't exactly have top-tier specs, so I have no idea what's going on there. Very weird indeed.
  - redox99 5mo ago
    Probably you don't have much to scrape?
    - ethin 5mo ago
      I mean... It may be that most of the things I run aren't really scrape-able. I run Matrix (which requires authentication), an XWiki instance, Zulip, Terraria, Forgejo, Nextcloud, a Mastodon server... Most of those require auth behind my Kanidm instance to actually do anything. Well and most of them have APIs that are much better than "scrape the universe".
- GoblinSlayer 5mo ago
  If you run the site on a custom port, scrapers won't find it?
userbinator 5mo ago
Also, how do we even know they're really "AI scrapers", or just a deliberate DDoS to push sites into using CF or other "anti-bot" providers?
- danielheath 5mo ago
  They showed up when the AI money did. The evidence is circumstantial, but… some of them are remarkably well engineered (from a “how difficult is it to identify this traffic” perspective, in a way that never existed before (I have been running a quite sizeable site for 8 years, over 200k registered users, and you don’t need to register to use 99% of it).
  - userbinator 5mo ago
    Yes, circumstantial is exactly the point; it's easy to use AI as a scapegoat because it's something popular to hate on.
    - danielheath 5mo ago
      It's circumstantial evidence, but Occam's Razor also applies. It's not a hostile DOS in the traditional sense (I've mitigated a few of those) - no "pay us to make it stop", no pattern to the requests other than "fetch every unique URL a few times". It wasn't happening until financial incentives to gather large datasets for AI training appeared. Bad actors (using residential proxies & claiming to be a real browser) mostly showed up after folk started blocking ones that identified themselves as AI scrapers. It's obvious to blame AI training because there's a shortage of better explanations. Who else would be paying for these (expensive) residential botnets, only to use them to (eg) web-scrape wikipedia (which offers free downloads of its content in a structured format)? The simplest explanation of the technical behavior is "a bot coded to follow every link it sees & save the results", and the simplest explanation of the motive to run such a bot is "to train a large language model".
      - userbinator 5mo ago

redox99 5mo ago

You get downvoted for these opinions but I agree. Most people that complain that their servers get hammered by AI bots are those that run very unoptimized servers that can only handle like 100 rps. I've never had any issues with any of my moderately optimized websites. A $10 VPS can handle sooo much traffic.

CodeBytes 5mo ago
I think people get annoyed when it's suggested they spend time optimising or even re-writing their websites to handle high traffic loads just to cater to AI bots ripping their content. It's also not always easy to do. I run a small wiki which is fairly optimised, nearly every page manages at least ~3k rps on a small VPS. The only exception is the diff page which is ~150 rps. Optimising that while still giving good output isn't that easy, but the wiki doesn't have many users so that would be fine if it wasn't for the AI bots. The AI bots ignore robots.txt and were initially hitting the site with ~1k rps crawling every combination. Even that would be manageable as there's currently ~150,000 combinations, except they kept re-crawling the whole lot each day. The server could manage it but it was a massive waste of resources. They were using residential IPs and only sending 1 request from each IP making it impossible to block. In the end I gave up and put a Cloudflare challenge in front of it. I don't want to use Cloudflare but the alternative is forcing users to login to view diffs or remove them entirely.
- canyp 5mo ago
  Curious, but how do the bots figure out the combinations? Or do you have links to the diffs from other sites? I assume the diff takes two files in query parameters or something.
  - CodeBytes 5mo ago
    I'm not 100% sure but I think links. There's a bunch on the history and revision pages. Yeah, the diff URL has two revision ID's as parameters. I did try removing some of the links without success. I guess once they have them they just keep checking.
- 5mo ago

dr_um 5mo ago

A small, single EU country focused non-static e-commerce, with proper robots.txt instructions that worked perfectly well in the search & co bots -only "era" with rate limiting for nginx/php-fpm setup - is kinda struggling without CF to handle 15000 requests per 15 minutes, coming from Chrome "users" from IPv6. Best so far was an avg. server load in htop = 40 on an 8-core server x_x

PunchyHamster 5mo ago
> handle 15000 requests per 15 minutes, that's just ~17 req/sec That's "cheap VPS running wordpress" level of traffic
- BenjiWiebe 5mo ago
  Maybe a plain WordPress install. Run something like WooCommerce and install a bunch of plugins to get the functionality that WordPress and WooCommerce should have built-in, and suddenly a cheap VPS can only handle 2 or 3 requests per second. It's phenomenal how inefficient the WordPress/WooCommerce stack is. Though the main issue I'm seeing is credit card testing, not scraping. And I'm ideologically opposed to using a CDN (because it shouldn't be needed for such a small site!) so it's somewhat a self-inflicted problem...
  - PunchyHamster 5mo ago
    "Security" plugins are also HUGE problem here, most of them turns "few cached DB SELECTs" (or static file read if you use caching plugin) into now a bunch of inserts, just to log/analyze "offender" IP and maybe block it, in many cases turning "blocking offender" to be more costly that would be serving the page without the security plugin
canyp 5mo ago
Block out IPv6 and see if that helps.
- lxgr 5mo ago

matt_heimer 5mo ago

It might depend on the tech stack. I run a small niche website but it has PHP and a database (MediaWiki/PHPBB) and without Cloudflare I'd estimate I'd need to spend several hundred dollars a month to handle the traffic. Traffic used to be tens of thousands of requests a day. AI has increased that to between 400k and 3M requests per day but it's not a smooth distribution. This is with bot fight mode on that greatly reduces traffic. I adopted Cloudflare because it was getting DDoSed by the AI crawlers. I'm pretty sure all of them are vibe coding their crawlers and don't bother adding rate limiting as a requirement.

jwr 5mo ago
That was my point. I was trying to be gentle by mentioning "unreasonable" things, but seriously — how did we get to the point where less than 6 requests per second (that's 500k requests per day) is considered a DDoS? I've spent some effort on optimizing my sites, but most of the effort was focused on avoiding unreasonable (stupid) work. Do I need a session for every request? No, I don't! Do I need a database fetch for every access to my homepage? No, I don't! Is it a problem to actually load all of my static content in all supported languages (24) into memory and serve it from memory? No, it isn't! I use Clojure behind nginx on the server for my sites. Oh, and I also pre-compress all static assets to Brotli, so anything that handles brotli gets a static file served directly from nginx. I also use immutable assets with unlimited caching semantics. Really — the problem is that we've grown lax and our software has become bloated, slow, and with unreasonable code paths. If every page fetch does 12 database accesses and runs through a slow interpreter, that is surely going to be a problem.
- matt_heimer 5mo ago
  That's the traffic after rate limiting controls and bot fight mode. It's 3-4 million requests per day without bot fight mode and just rate limits. And as I said it's not a smooth distribution. Plus the requests are almost never for pages in cache. It's always stuff like loading all the message threads from the year 2000 or loading up the details of every page edit ever made to a wiki page. If it was more static content it'd be easier, it's really the db being a bottle neck in a dynamic site. Yes, the software could be better optimized but then I'd have to own the development of it. There is no reason a niche website should be getting millions of requests per day.

canyp 5mo ago

I second this. My website exposes a cgit and 99% of the traffic now is AI scraping the sources, but the load is nowhere near DoS territory. And this is running on the cheapest VPS I could find. Not saying I'm not annoyed by the scraping; I am looking to block them, but I'm also not going to put the site behind the gatekeeper. If anything, Cloudflare must love AI scraping now for the same reason AV companies love malware. Now, if you are running a PHP stack...yeah, maybe that's the problem right there.

lxgr 5mo ago
Is there actually any plausible theory why "AI" would repeatedly scrape the same sites? Are there that many competing, completely independent AI labs? Is it cheaper to repeatedly scrape than to buffer the scraped data locally? (I find it very hard to imagine that it's easier to deal with changing/disappearing content than it is to stand up such a cache.)
- jack_pp 5mo ago
  If you ask an agent to check sources / function definitions of open source packages it will wget / curl it
- GoblinSlayer 5mo ago
  It's an AI generated scraper that scrapes nonstop.
account42 5mo ago
The PHP stack isn't even the problem, it's having unauthenticated requests getting past the cache in the first place, something that most sites should be able to prevent.
ern_ave 5mo ago
> 99% of the traffic now is AI scraping the sources I wonder if we should stop fighting this and instead create an API specifically for this purpose? Or, a central repository that you could send your data to and say to anyone wanting to scrape, "safe yourself some time and just get my data from this other place"

acdha 5mo ago

We do know, just ask anyone who runs a more popular site or does anything where abuse can be monetized (shopping, reviews, etc.). Avoiding that due to obscurity isn’t an answer because it’s saying you’re safe until something, possibly outside of your control, causes the bots to descend and give you an extra 500M requests with no chance of revenue. I’m with OP: I don’t like this but the alternatives all look like the death of the open web.

handoflixue 5mo ago
> just ask anyone who runs a more popular site The person you're responding to already said they ran a modestly sized site. What actual scale opens one up to abuse? If only the top 1% of sites need it, then it seems silly to say "everyone" needs it.
- daishi55 5mo ago
  So everyone is paying cloudflare… why?
  - fragmede 5mo ago
    Most likely not. Their free tier is fairly generous.
  - tardedmeme 5mo ago
    Because paying with MITM is far less visible than paying with money
  - LgWoodenBadger 5mo ago
    Because charging for bandwidth/traffic is still a thing, unfortunately
- ceejayoz 5mo ago
  It’s not just scale. Do you accept user generated content? If so, more of a target.

piker 5mo ago

Same. Tritium and the blog have done stents on the front page here and high traffic subreddits and that plus bots has never been a problem. UX could be improved through a CDN but even that isn’t worth the trade-off for us at the moment.

JohnTHaller 5mo ago

If you're in any way semi-popular and a decent size, you're gonna get hammered. PortableApps.com was partially offline for weeks due to China-based AI scrapers. You block the useragent, they start hitting you with another one from the same IP in the same way. You block the IP, they switch to another. You block the subnet, they use another. At one point it was nearly a thousand different IPs from around China hammering away. For all intents and purposes, a DDoS. This wasn't a little "extra load", this was load that was thousands of times beyond what our legitimate userbase was using. And if you're thinking about blocking all of China, while this particular AI bot didn't use them, a bunch of other ones I've encountered use VPNs and hacked clients worldwide.

RHSeeger 5mo ago

> I understand there are sites for whom this causes problems, but I think these are rare and could be optimized not to do unreasonable things. There are. They're not. They can't (without significant effort)

hombre_fatal 5mo ago

Consider yourself lucky. But don't let yourself fall into the trap of thinking it's a nonissue for everyone else until it happens to you. People shouldn't have to be experts or provision a larger server to run a UGC service that can withstand the sort of 30x more traffic I'm seeing from AI bots. Or rather, you didn't render the argument for why they should have to do that if they can just use CloudFlare's free tier. Either way, it's easy to have all the answers when you've never had the problem.