4 ms·

My team runs a quite popular website, #1 or #2 in the market depending on the region. Several million visits per day. Around June/July we got a 10x boost out o

by whstl 1mo ago

My team runs a quite popular website, #1 or #2 in the market depending on the region. Several million visits per day. Around June/July we got a 10x boost out of nowhere, and it started affecting performance for users, increased hosting costs, and random bursts would bring the website down. We spent some time trying out solutions, from Cloudflare and Anubis to AWS, but it ended up affecting real users, and we got complaints on Reddit from day one, pretty much. At the same time, it got a bit expensive having a WAF. In the end the only solution that worked was blocking every single Chinese IP. Traffic instantly got down to 10x less, same as it was before. Same conversion percentage as before.

robotmay 1mo ago
In an odd sequence of events, I've sorta got a weird tiny following in China due to having a vintage camera stall in an antiques market here in Wales that gets shared on Xiaohongshu (https://en.wikipedia.org/wiki/Xiaohongshu https://en.wikipedia.org/wiki/Xiaohongshu) sometimes when I have Chinese customers. So I'd feel bad blocking the entire country myself, though thankfully I'm not being inundated with traffic from there yet - the most egregious bots I've caught so far are actually ones from Meta which ignore robots.txt.
- whstl 1mo ago
  Oh trust me, we felt really really bad blocking the whole country, and it took weeks of attempts avoiding it. Meta bots are also problematic, as they seem to be quite buggy. But the volume was nowhere near the hundreds of bots from China :/
  - robotmay 1mo ago
    Yeah Meta's bots are well and truly stuck in the blackhole on this site at the moment. I have no idea what they're supposed to be doing, but in case they are using them for AI work I've just added a token multipler based on subnet totals so they can have some more fun. I might come up with some other fun response types for them too.
leohart 1mo ago
I am surprised POW Anubis doesn't work for you. Are these bots bypassing POW somehow?
- account42 1mo ago
  POW never worked. The only bots its stops are the simple ones that don't support java script or cookies at all.
rdtsc 1mo ago
The same trick worked for me 15 years ago to drop a lot of log noise out of constant connection attempts. Felt a bit bad but in the end it turned out fairly well.

by whstl 1mo ago

robotmay 1mo ago

In an odd sequence of events, I've sorta got a weird tiny following in China due to having a vintage camera stall in an antiques market here in Wales that gets shared on Xiaohongshu (https://en.wikipedia.org/wiki/Xiaohongshu https://en.wikipedia.org/wiki/Xiaohongshu) sometimes when I have Chinese customers. So I'd feel bad blocking the entire country myself, though thankfully I'm not being inundated with traffic from there yet - the most egregious bots I've caught so far are actually ones from Meta which ignore robots.txt.

whstl 1mo ago
Oh trust me, we felt really really bad blocking the whole country, and it took weeks of attempts avoiding it. Meta bots are also problematic, as they seem to be quite buggy. But the volume was nowhere near the hundreds of bots from China :/
- robotmay 1mo ago
  Yeah Meta's bots are well and truly stuck in the blackhole on this site at the moment. I have no idea what they're supposed to be doing, but in case they are using them for AI work I've just added a token multipler based on subnet totals so they can have some more fun. I might come up with some other fun response types for them too.

leohart 1mo ago

I am surprised POW Anubis doesn't work for you. Are these bots bypassing POW somehow?

account42 1mo ago
POW never worked. The only bots its stops are the simple ones that don't support java script or cookies at all.

rdtsc 1mo ago

The same trick worked for me 15 years ago to drop a lot of log noise out of constant connection attempts. Felt a bit bad but in the end it turned out fairly well.