3 ms·
At least their bots accurately identify themselves in the User-Agent field even when they're ignoring robots.txt, so serverside blocking is on the table for now
by jsheard 2y ago
At least their bots accurately identify themselves in the User-Agent field even when they're ignoring robots.txt, so serverside blocking is on the table for now at least.
Bytedances crawler (Bytespider) is another one which disregards robots.txt but still identifies itself, and you probably should block it because it's very aggressive.
It's going to get annoying fast when they inevitably go full blackhat and start masquerading as normal browser traffic.
- 0x000xca0xfe 2y agoThey should be easy to fingerprint if they are this aggressive.
- znpy 2y agopeople should probably heavily rate-limit (or block block their) whole ASN-advertised subnets, just in case.