3 ms·
> There has been upwards of 15 queries per second from bots. There is just no way to deal with that sort of traffic, barely even to reject it. If the queries a
by Aachen 4y ago
> There has been upwards of 15 queries per second from bots. There is just no way to deal with that sort of traffic, barely even to reject it.
If the queries are not a megabit each, you're doing way too much processing before applying rate limiting. Rejecting traffic ought not to take more than 1-2 milliseconds, even if you need to look up an api key or IP address in the database.
I, too, host services on a residential connection: 50 mbps shared with other users. My domains must host hundreds of separate scripts, a few of which have a database attached (I can think of six off the top of my head, but there's over a hundred databases in mariadb so I'm sure there's more that I've forgotten about). This is a ten-year-old laptop with a regular "apt install mariadb", no special configs.
Yes, most traffic is bots, and yes sometimes they submit more than 1 q/s. But it comes nowhere near to exhausting resources to a noticeable extent. Load average is about 0.15, main peaks come from my own cronjobs. If you're having this much trouble rejecting traffic, you might want to spend some time looking at bottlenecks. You'll also notice the bots knock it off if it's unsuccessful.
- marginalia_nu 4y agoThat's queries per second (as searches), not requests per second. You know how blogs sometimes can't handle being on the hacker news front page from all the traffic? Well my search engine has survived that. That was 1-2 QPS. Unmitigated bot-traffic is roughly 10x the traffic of a hacker news death hug, as a sustained load.