4 ms·

People outside of a really small sysadmin niche really don't grasp the scale of this problem. I run a small-but-growing boutique hosting infrastructure for age

People outside of a really small sysadmin niche really don't grasp the scale of this problem. I run a small-but-growing boutique hosting infrastructure for agency clients. The AI bot crawler problem recently got severe enough that I couldn't just ignore it anymore. I'm stuck between, on one end, crawlers from companies that absolutely have the engineering talent and resources to do things right but still aren't, and on the other end, resource-heavy WordPress installations where the client was told it was a build-it-and-forget-it kind of thing. I can't police their robots.txt files; meanwhile, each page load can take a full 1s round trip (most of that spent in MySQL), there are about 6 different pretty aggressive AI bots, and occasionally they'll get stuck on some site's product variants or categories pages and start hitting it at a 1r/s rate. There's an invisible caching layer that does a pretty nice job with images and the like, so it's not really a bandwidth problem. The bots aren't even requesting images and other page resources very often; they're just doing tons and tons of page requests, and each of those is tying up a DB somewhere. Cumulatively, it is close to having a site get Slashdotted every single day. I finally started filtering out most bot and crawler traffic at nginx, before it gets passed off to a WP container. I spent a fair bit of time sampling traffic from logs, and at a rough guess, I'd say maybe 5% of web traffic is currently coming from actual humans. It's insane. I've just wrapped up the first round of work for this problem, but that's just buying a little time. Now, I've gotta put together an IP intelligence system, because clearly these companies aren't gonna take "403" for an answer.

gjsman-1000 1y ago
I might write a blog post on this, but I seriously believe we collectively need to rethink The Cathedral and the Bazaar. The Cathedral won. Full stop. Everyone, more or less, is just a stonecutter, competing to sell the best stone (i.e. content, libraries, source code, tooling) for building the cathedrals with. If the world is a farmer's market, we're shocked that the farmer's market is not defeating Walmart, and never will. People want Cathedrals; not Bazaars. Being a Bazaar vendor is a race to the bottom. This is not the Cathedral exploiting a "tragedy of the commons," it's intrinsic to decentralization as a whole. The Bazaar feeds the Cathedral, just as the farmers feed Walmart, just as independent websites feed Claude, a food chain and not an aberration.
- thaumaturgy 1y ago
  The Cathedral and the Bazaar meets The Tragedy of the Commons. Let's say there's two competing options in some market. One option is fully commercialized, the other option holds to open-source ideals (whatever those are). The commercial option attracts investors, because investors like money. The money attracts engineers, because at some point "hacker" came to mean "comfortable lifestyle in a high COL area". The commercial option gets all the resources, it gets a marketing team, and it captures 75% of the market because most people will happily pay a few dollars for something they don't have to understand. The open source option attracts a few enthusiasts (maybe; or, often, just one), who labor at it in whatever spare time they can scrape together. Because it's free, other commercial entities use and rely on the open source thing, as long it continues to be maintained in something that, if you squint, resembles slave labor. The open source option is always a bit harder to use, with fewer features, but it appeals to the 25% of the market that cares about things like privacy or ownership or self-determination. So, one conclusion is "people want Cathedrals", but another conclusion could be that all of our society's incentives are aligned towards Cathedrals. It would be insane, after all, to not pursue wealth just because of some personal ideals.
  - gjsman-1000 1y ago
    The answer is quite simply that where complexity exceeds the regular person's interest, there will be a cathedral. It's not about capitalism or incentives. Humans have cognitive limits and technology is very low on the list for most. They want someone else to handle complexity so they can focus on their lives. Medieval guilds, religious hierarchies, tribal councils, your distribution's package repository, it's all cathedrals. Humans have always delegated complexity to trusted authorities. The 25% who 'care about privacy or ownership' mostly just say they care. When actually faced with configuring their own email server or compiling their own kernel, 24% of that 25% immediately choose the cathedral. You know the type, the people who attend FOSDEM carrying MacBooks. The incentives don't create the demand for cathedrals, but respond to it. Even in a post-scarcity commune, someone would emerge to handle the complex stuff while everyone else gratefully lets them. The bazaar doesn't lose because of capitalism. It loses because most humans, given the choice between understanding something complex or trusting someone else to handle it, will choose trust every time. Not just trust, but CYA (I'm not responsible for something I don't fully understand) every time. Why do you think AI is successful? I'd rather even trust a blathering robot than myself. It turns out, people like being told what to do on things they don't care about.
jazzyjackson 1y ago
Couldn't it be addressed in front of the application with a fail2ban rule, some kind of 429 Too Many Requests quota on a per session basis? Or are the crawlers anonymizing themselves / coming from different IP addresses?
- sc68cal 1y ago
  They are spreading themselves across lots of different IP blocks
- thaumaturgy 1y ago
  Yeah, that's where IP intelligence comes in. They're using pretty big IP pools, so, either you're manually adding individual IPs to a list all day (and updating that list as ASNs get continuously shuffled around), or you've got a process in the background that essentially does whois lookups (and caches them, so you aren't also being abusive), parses the metadata returned, and decides whether that request is "okay" or not. The classic 80/20 rule applies. You can catch about 80% of lazy crawler activity pretty easily with something like this, but the remaining 20% will require a lot more effort. You start encountering edge cases, like crawlers that use AWS for their crawling activity, but also one of your customers somewhere is syncing their WooCommerce orders to their in-house ERP system via a process that also runs on AWS.
  - asddubs 1y ago
    I've had crawlers get stuck in a loop before on a search page where you basically could just keep adding things, even if there are no results. I filtered requests that are bots for sure (requests which are specified long past the point of any results). It was over a million unique IPs, most of which only doing 1 or 2 requests on their own (from many different ip blocks)
- loloquwowndueo 1y ago
  Its called Anubis.
  - dylan604 1y ago
jay_kyburz 1y ago
This is probably a dumb question, but at what point do we put a simple CAPTCHA in front of every new user that arrives at a site, then give them a cookie and start tracking requests per second from that user? I guess its a kind of soft login required for every session? update: you could bake it into the cookie approval dialog (joke!)
- thaumaturgy 1y ago
  The post-AI web is already a huge mess. I'd prefer solutions that don't make it worse. I myself browse with cookies off, sort of, most of the time, and the number of times per day that I have to click a Cloudflare checkbox or help Google classify objects from its datasets is nuts.
  - dragonwriter 1y ago
    > The post-AI web is already a huge mess. You mean the peri-AI web? Or is AI already done and over and no longer exerting an influence?
AnthonyMouse 1y ago
> meanwhile, each page load can take a full 1s round trip (most of that spent in MySQL) Can't these responses still be cached by a reverse proxy as long as the user isn't logged in, which the bots presumably aren't?
- thaumaturgy 1y ago
  That would be nice! This doesn't work reliably enough for WP sites. Whether it's devs making changes and testing them in prod, or dynamic content loaded in identical URLs, my past attempts to cache html have caused questions and complaints. The current caching strategy hits a nice balance and hasn't bothered anyone, with the significant downside that it's vulnerable to bot traffic. (If you choose to read this as, "WordPress is awful, don't use WordPress", I won't argue with you.)
- everforward 1y ago
  They're presumably not crawling the same page repeatedly, and caching the pages long enough to persist between crawls would require careful thinking and consultation with clients (e.g. if they want their blog posts to show up quickly, or an "on sale" banner or etc). It'd probably be easier to come at it from the other side and throw more resources at the DB or clean it up. I can't imagine what's going on that it's spending a full second on DB queries, but I also don't really use WP.
  - benjiro 1y ago
    Its been a few years when i last worked with WP. But the performance issue is because they store a ton of the data in a key value store, instead of table with fixed columns. This can result in a ton of individual row hits on your database, for what in any normal system is a single 0.1ms (often faster) DB request. Any web scraper that is scraping SEQUENCIALLY at 1r/s is actually a well behaved and non-intrusive scraper. Its just that the WP is in general ** for performance. If you want to see what a bad scraper does with parallel requests with little limits, yea, WP is going down without putting up any struggle. But everybody wanted to use WP, and now those ducks are coming home to roost when there is a bit more pressure.
TheServitor 1y ago
There are some ASN-based DROP list collections on GitHub if that would help.
- thaumaturgy 1y ago
  Oh! That didn't even occur to me. Yeah, I could pump that into ipset. Got one in particular that you think is reliable?
  - TheServitor 1y ago
    I think Spamhaus runs the big one.

gjsman-1000 1y ago

I might write a blog post on this, but I seriously believe we collectively need to rethink The Cathedral and the Bazaar. The Cathedral won. Full stop. Everyone, more or less, is just a stonecutter, competing to sell the best stone (i.e. content, libraries, source code, tooling) for building the cathedrals with. If the world is a farmer's market, we're shocked that the farmer's market is not defeating Walmart, and never will. People want Cathedrals; not Bazaars. Being a Bazaar vendor is a race to the bottom. This is not the Cathedral exploiting a "tragedy of the commons," it's intrinsic to decentralization as a whole. The Bazaar feeds the Cathedral, just as the farmers feed Walmart, just as independent websites feed Claude, a food chain and not an aberration.

thaumaturgy 1y ago
The Cathedral and the Bazaar meets The Tragedy of the Commons. Let's say there's two competing options in some market. One option is fully commercialized, the other option holds to open-source ideals (whatever those are). The commercial option attracts investors, because investors like money. The money attracts engineers, because at some point "hacker" came to mean "comfortable lifestyle in a high COL area". The commercial option gets all the resources, it gets a marketing team, and it captures 75% of the market because most people will happily pay a few dollars for something they don't have to understand. The open source option attracts a few enthusiasts (maybe; or, often, just one), who labor at it in whatever spare time they can scrape together. Because it's free, other commercial entities use and rely on the open source thing, as long it continues to be maintained in something that, if you squint, resembles slave labor. The open source option is always a bit harder to use, with fewer features, but it appeals to the 25% of the market that cares about things like privacy or ownership or self-determination. So, one conclusion is "people want Cathedrals", but another conclusion could be that all of our society's incentives are aligned towards Cathedrals. It would be insane, after all, to not pursue wealth just because of some personal ideals.
- gjsman-1000 1y ago
  The answer is quite simply that where complexity exceeds the regular person's interest, there will be a cathedral. It's not about capitalism or incentives. Humans have cognitive limits and technology is very low on the list for most. They want someone else to handle complexity so they can focus on their lives. Medieval guilds, religious hierarchies, tribal councils, your distribution's package repository, it's all cathedrals. Humans have always delegated complexity to trusted authorities. The 25% who 'care about privacy or ownership' mostly just say they care. When actually faced with configuring their own email server or compiling their own kernel, 24% of that 25% immediately choose the cathedral. You know the type, the people who attend FOSDEM carrying MacBooks. The incentives don't create the demand for cathedrals, but respond to it. Even in a post-scarcity commune, someone would emerge to handle the complex stuff while everyone else gratefully lets them. The bazaar doesn't lose because of capitalism. It loses because most humans, given the choice between understanding something complex or trusting someone else to handle it, will choose trust every time. Not just trust, but CYA (I'm not responsible for something I don't fully understand) every time. Why do you think AI is successful? I'd rather even trust a blathering robot than myself. It turns out, people like being told what to do on things they don't care about.

jazzyjackson 1y ago

Couldn't it be addressed in front of the application with a fail2ban rule, some kind of 429 Too Many Requests quota on a per session basis? Or are the crawlers anonymizing themselves / coming from different IP addresses?

sc68cal 1y ago
They are spreading themselves across lots of different IP blocks
thaumaturgy 1y ago
Yeah, that's where IP intelligence comes in. They're using pretty big IP pools, so, either you're manually adding individual IPs to a list all day (and updating that list as ASNs get continuously shuffled around), or you've got a process in the background that essentially does whois lookups (and caches them, so you aren't also being abusive), parses the metadata returned, and decides whether that request is "okay" or not. The classic 80/20 rule applies. You can catch about 80% of lazy crawler activity pretty easily with something like this, but the remaining 20% will require a lot more effort. You start encountering edge cases, like crawlers that use AWS for their crawling activity, but also one of your customers somewhere is syncing their WooCommerce orders to their in-house ERP system via a process that also runs on AWS.
- asddubs 1y ago
  I've had crawlers get stuck in a loop before on a search page where you basically could just keep adding things, even if there are no results. I filtered requests that are bots for sure (requests which are specified long past the point of any results). It was over a million unique IPs, most of which only doing 1 or 2 requests on their own (from many different ip blocks)
loloquwowndueo 1y ago
Its called Anubis.
- dylan604 1y ago

jay_kyburz 1y ago

This is probably a dumb question, but at what point do we put a simple CAPTCHA in front of every new user that arrives at a site, then give them a cookie and start tracking requests per second from that user? I guess its a kind of soft login required for every session? update: you could bake it into the cookie approval dialog (joke!)

thaumaturgy 1y ago
The post-AI web is already a huge mess. I'd prefer solutions that don't make it worse. I myself browse with cookies off, sort of, most of the time, and the number of times per day that I have to click a Cloudflare checkbox or help Google classify objects from its datasets is nuts.
- dragonwriter 1y ago
  > The post-AI web is already a huge mess. You mean the peri-AI web? Or is AI already done and over and no longer exerting an influence?

AnthonyMouse 1y ago

> meanwhile, each page load can take a full 1s round trip (most of that spent in MySQL) Can't these responses still be cached by a reverse proxy as long as the user isn't logged in, which the bots presumably aren't?

thaumaturgy 1y ago
That would be nice! This doesn't work reliably enough for WP sites. Whether it's devs making changes and testing them in prod, or dynamic content loaded in identical URLs, my past attempts to cache html have caused questions and complaints. The current caching strategy hits a nice balance and hasn't bothered anyone, with the significant downside that it's vulnerable to bot traffic. (If you choose to read this as, "WordPress is awful, don't use WordPress", I won't argue with you.)
everforward 1y ago
They're presumably not crawling the same page repeatedly, and caching the pages long enough to persist between crawls would require careful thinking and consultation with clients (e.g. if they want their blog posts to show up quickly, or an "on sale" banner or etc). It'd probably be easier to come at it from the other side and throw more resources at the DB or clean it up. I can't imagine what's going on that it's spending a full second on DB queries, but I also don't really use WP.
- benjiro 1y ago
  Its been a few years when i last worked with WP. But the performance issue is because they store a ton of the data in a key value store, instead of table with fixed columns. This can result in a ton of individual row hits on your database, for what in any normal system is a single 0.1ms (often faster) DB request. Any web scraper that is scraping SEQUENCIALLY at 1r/s is actually a well behaved and non-intrusive scraper. Its just that the WP is in general ** for performance. If you want to see what a bad scraper does with parallel requests with little limits, yea, WP is going down without putting up any struggle. But everybody wanted to use WP, and now those ducks are coming home to roost when there is a bit more pressure.

TheServitor 1y ago

There are some ASN-based DROP list collections on GitHub if that would help.

thaumaturgy 1y ago
Oh! That didn't even occur to me. Yeah, I could pump that into ipset. Got one in particular that you think is reliable?
- TheServitor 1y ago
  I think Spamhaus runs the big one.