5 ms·

It isn't that hard to set up naive rate limiting per ip address. It's a few lines in haproxy, and there is documentation on how to do it. There are a couple of

by thayne 3y ago

It isn't that hard to set up naive rate limiting per ip address. It's a few lines in haproxy, and there is documentation on how to do it. There are a couple of problems that make it more complicated though. The first is that with NATs, you can have a lot of users behind a single IP address, which can result in legitimate requests getting blocked. The second is that, while it can help against a DoS, it doesn't help that much against a DDoS, because the requests are coming from a large number of distinct IP addreesses.

crazygringo 3y ago
But it's precisely those complicated parts though that seem like they should be solved by now. The solution is both of the problems is for when traffic gets unexpectedly high either behind a NAT or globally, everybody gets a CATPCHA when gives you a token and then you continue to rate-limit each token. (If users log in then each login already does this, no CAPTCHA needed.) This strategy is basically what e.g. CloudFlare does with the CAPTCHAs. Obviously if someone is attempting a large-scale DDoS your servers can't handle it and you'll be using CloudFlare for its scale. But otherwise, for basic protection against greedy spiders who are even trying to evade detection across a ton of VPN/cloud IP's, this strategy works fine. It's exactly the kind of thing that I would expect any large website to implement. If there isn't an open-source tool that does this, I wonder why not. Or if there is, I wonder why IA isn't using something like it. But heck, IA wasn't even using a simple version -- it was just 64 IP addresses where basic rate-limiting would have worked fine.
nektro 3y ago
ipv6 would solve these issues because they remove the need for NATs
- daoman 3y ago
  And create a new problem: banning by IPv6 is a nightmare https://adam-p.ca/blog/2022/02/ipv6-rate-limiting/ https://adam-p.ca/blog/2022/02/ipv6-rate-limiting/

by thayne 3y ago

crazygringo 3y ago

But it's precisely those complicated parts though that seem like they should be solved by now. The solution is both of the problems is for when traffic gets unexpectedly high either behind a NAT or globally, everybody gets a CATPCHA when gives you a token and then you continue to rate-limit each token. (If users log in then each login already does this, no CAPTCHA needed.) This strategy is basically what e.g. CloudFlare does with the CAPTCHAs. Obviously if someone is attempting a large-scale DDoS your servers can't handle it and you'll be using CloudFlare for its scale. But otherwise, for basic protection against greedy spiders who are even trying to evade detection across a ton of VPN/cloud IP's, this strategy works fine. It's exactly the kind of thing that I would expect any large website to implement. If there isn't an open-source tool that does this, I wonder why not. Or if there is, I wonder why IA isn't using something like it. But heck, IA wasn't even using a simple version -- it was just 64 IP addresses where basic rate-limiting would have worked fine.

nektro 3y ago

ipv6 would solve these issues because they remove the need for NATs

daoman 3y ago
And create a new problem: banning by IPv6 is a nightmare https://adam-p.ca/blog/2022/02/ipv6-rate-limiting/ https://adam-p.ca/blog/2022/02/ipv6-rate-limiting/