5 ms·

I manage a network of several hundred websites. About 2 years ago we started blocking bot/crawler user-agents. Some websites see thousands of these requests e

• supriyo-biswas 3y ago
  You're better off using a combination of IP blocklisting and ratelimiting, both of which use some sort of binary tree to speed up lookups, rather than using a mega-regex which your solution proposes, which, with tons of backtracking, is very likely slower than serving the request would have been. Not to mention that user agents are easily spoofable.
• kube-system 3y ago
  If a bot's user agent doesn't look identical to a normal user, either the bot developer is dumb, or they are relatively well behaving. Anyone malicious who knows what they're doing will just send you a statistically common user agent string. Under most http libraries this is a one-liner: headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.0.0'}
• _lvbh 3y ago
  Try JA3 fingerprinting. It’s much more difficult to spoof than user agents