I just had to put my small hobby OS development site behind cloudflare after 23 years without a CDN. Between occasional ip bans, mod_evasive, rate limiting, us

I just had to put my small hobby OS development site behind cloudflare after 23 years without a CDN. Between occasional ip bans, mod_evasive, rate limiting, user agent blocks, etc I've been self hosting on a single box without much problem until last week. We went from hundreds of thousands of requests per day to 5 million per day. Traffic was web scraping bots based on the obscure URLs. The URLs were valid (mediawiki history links, etc) and not attempts to hack the site. Banning IPs did not help, the traffic would move to new subnets. Mostly IPv4, some IPv6. The user agent was popular Chrome agent strings so I'm guessing it was masked puppeteer. It was a DDoS in practice but I get the feeling it's an immature web crawler. I think people are likely building a new generation of crawlers to feed LLMs as fast as possible. The caching aspect of Cloudflare helped a lot. Putting specific url patterns behind Cloudflares dynamic JavaScript challenge also helped. It was surprisingly easy to setup. And I know what some of you will say, Cloudflare is bad. I've personally been annoyed with them for making specific sites more difficult to use while on VPN. But it's not a hard choice when it's either taking your site offline or using their free tier offering.