3 ms·

These attackers don't have inexhaustible resources.

logifail 3y ago
Indeed, but neither do you. Be very careful that fail2ban isn't actually exhausting your own resources faster than you believe you're exhausting the attackers' resources... https://www.google.com/search?q=fail2ban+resource+usage https://www.google.com/search?q=fail2ban+resource+usage
quesera 3y ago
I've recently logged credential stuffing attacks coming from over 100K distinct IPv4 addresses to a small backend API.
- vntok 3y ago
  Rate-limit after x failed logins on either source IP, username and password. Just provide a realworld sidechannel escape hatch for legitimate users (ex: phone or email). Barely anyone will actually use it.
  - quesera 3y ago
    We had to go further. The attacks were from arbitrary IPs, and were working through a huge list of leaked username/password combinations. The attacker even went to the trouble of spoofing some custom headers in our API client. Our eventual solution was to a) attack our own auth credentials first, identify any users with leaked creds (from other services) and force a password reset for them. b) disallow users from setting common leaked passwords. c) make the auth checking request as low-cost as possible, and scalable separately from the main application. d) when an attack request is detected, bypass the relatively-expensive real cred check but return the same failure response (including timing) as a real failure. e) build a secondary requirement in the auth flow that can be transparently enabled when under high volume attack. This works, so far. It sheds the volume to the application, and has low-to-zero impact on legit users. This took a couple weeks away from feature development though!
- Aachen 3y ago
  This interests me a lot as we do security but don't run a big service ourselves and so don't have data on what motivated attackers' behavior is exactly. How many active users (to an order of magnitude; no need for precise numbers of course) does this service have? 100k IPs sounds pretty costly to burn, so I'm curious how important one needs to be before that's considered worth it. And could you say what type of IP addresses those were? Did it look residential such as from compromises computers (botnet), do they rent lots of IP addresses temporarily from aws/netcup/alibaba, or is it a mix such that neither category has the overwhelming majority? If it's all server IP ranges for a service where end users normally log in, you could apply entirely different rate limits to those than to residential IPs, for example. Hence I'm wondering how these cred stuffing attacks are set up
  - quesera 3y ago
    It was a mix. Lots of apparent botnets (residential service scattered across many providers and networks), but also healthy chunks of colo/SaaS netblocks as well. And of course a ton of open proxies.
    - Aachen 3y ago
      Thanks!

Indeed, but neither do you. Be very careful that fail2ban isn't actually exhausting your own resources faster than you believe you're exhausting the attackers' resources... https://www.google.com/search?q=fail2ban+resource+usage https://www.google.com/search?q=fail2ban+resource+usage

quesera 3y ago

I've recently logged credential stuffing attacks coming from over 100K distinct IPv4 addresses to a small backend API.

vntok 3y ago
Rate-limit after x failed logins on either source IP, username and password. Just provide a realworld sidechannel escape hatch for legitimate users (ex: phone or email). Barely anyone will actually use it.
- quesera 3y ago
  We had to go further. The attacks were from arbitrary IPs, and were working through a huge list of leaked username/password combinations. The attacker even went to the trouble of spoofing some custom headers in our API client. Our eventual solution was to a) attack our own auth credentials first, identify any users with leaked creds (from other services) and force a password reset for them. b) disallow users from setting common leaked passwords. c) make the auth checking request as low-cost as possible, and scalable separately from the main application. d) when an attack request is detected, bypass the relatively-expensive real cred check but return the same failure response (including timing) as a real failure. e) build a secondary requirement in the auth flow that can be transparently enabled when under high volume attack. This works, so far. It sheds the volume to the application, and has low-to-zero impact on legit users. This took a couple weeks away from feature development though!
Aachen 3y ago
This interests me a lot as we do security but don't run a big service ourselves and so don't have data on what motivated attackers' behavior is exactly. How many active users (to an order of magnitude; no need for precise numbers of course) does this service have? 100k IPs sounds pretty costly to burn, so I'm curious how important one needs to be before that's considered worth it. And could you say what type of IP addresses those were? Did it look residential such as from compromises computers (botnet), do they rent lots of IP addresses temporarily from aws/netcup/alibaba, or is it a mix such that neither category has the overwhelming majority? If it's all server IP ranges for a service where end users normally log in, you could apply entirely different rate limits to those than to residential IPs, for example. Hence I'm wondering how these cred stuffing attacks are set up
- quesera 3y ago
  It was a mix. Lots of apparent botnets (residential service scattered across many providers and networks), but also healthy chunks of colo/SaaS netblocks as well. And of course a ton of open proxies.
  - Aachen 3y ago
    Thanks!