5 ms·

> at one point tried to load-balance the traffic across _30_ > machines efficient blocking rules, I was saturating their > connections. I have a slightly mor

by bArray 7y ago

> at one point tried to load-balance the traffic across _30_ > machines efficient blocking rules, I was saturating their > connections. I have a slightly more complex "load-balancing" method, as long as the network bandwidth itself can hold out (hosted in the cloud) I've been able to deal with all attacks so far. My current method is to have a "whitelist" and "blacklist" look up table to quickly begin filtering incoming connections. If you end up on the blacklist (reasons include too many connections over a time period, bad requests, too many requests without authentication, etc, etc) then you instantly get killed - not a single byte gets sent. Being on the whitelist (reasons include being an already auth'd user, low traffic density, unique requests, etc, etc) then you shortcut some additional checks. Connections not in either list (potentially either attackers or new users) go through an additional per-generated (the check and answer is statically generated, so it's not added to connection overhead) check to "test for humanness". Both lists are decayed over time and connections on both lists can be re-assigned if they start behaving badly. The additional checks are mostly only activated under high load. After that there is a resource management layer that is essentially "service temporarily unavailable" on anything too heavy (large file GET/POST, large database read/write, non-essential database writes are dropped, etc, etc). This is all then run through a high-level network simulation to test where the bottlenecks will be. Each "release" goes through the test before making it into production. I can't give too many details, but there is also a slightly malicious protection layer - if we detect somebody using a known/obvious bot (which is harmful) we have a few methods to crash them (based on known vulnerabilities). Some of these also act against aggressive search bots. I have a beautiful graph sitting somewhere showing an attack ramping up and then mostly disappearing when the defense was triggered. It was quite worrying at the time because we just assumed our system went down or we had accidentally started blocking real users. > That said, I strongly suspect cloudflare probably works > hand-in-hand with the NSA or some shit, and it's a dangerous > trend. I also highly suspect this. > But I just don't think there exists a viable alternative, unless > you're a multi-million dollar internet company I agree and that is a massive problem.

tpetry 7y ago
Your defense mechanism is actually the exact thing described as protection against a simple DDOS attack. With a REAL ddos attack you will get so many incoming traffic that your bandwidth is saturated before any software will come in effect. A real ddos is a hardware problem, and it cant therefore be solved by software.
- cannonedhamster 7y ago
  Not to mention his solution really does nothing against ping, UDP, and TCP traffic. It's really only effective against http traffic. Which means his upstream or provider is probably going to cut him loose for the duration of the attack. A single person cannot effectively block a concerted DDOS.
  - bArray 7y ago
    > Not to mention his solution really does nothing against > ping, UDP, and TCP traffic. Please bare in mind that the description is heavily simplified. If you're talking about bandwidth, of course this is the limit. > It's really only effective against http traffic. How so? > Which means his upstream or provider is probably going to > cut him loose for the duration of the attack. There's nothing we can do about that, but being in the cloud offers at least some protection. DDoS'ing some random Pi at home is a little different from DDoS'ing an AWS server. > A single person cannot effectively block a concerted DDOS. Of course not, but that doesn't mean you have to make it easy.
- bArray 7y ago
  > Your defense mechanism is actually the exact thing > described as protection against a simple DDOS attack. With > a REAL ddos attack [..] I'm not entirely sure what is really classified as a "simple" vs "real" DDoS attack. > With a REAL ddos attack you will get so many incoming > traffic that your bandwidth is saturated before any > software will come in effect. I specifically said: "as long as the network bandwidth itself can hold out". If the network is saturated, it's saturated - it's game over, for anybody. But you'll find that most services will die way before this limit is reached. On a WordPress site for example you'll usually find the database will give out long before the bandwidth is saturated. Even with legitimate users coming through some great filter like CloudFlare, it doesn't prevent a hug-of-death if you don't have some "smart" application level handling of high numbers of users. Like Linux OOM, when you hit your limits, the only choice left is to try and quickly and intelligently start killing without affecting the well behaved.
namibj 7y ago
The issue is that TLS handshakes are actually quite expensive. Assume a limit at about 3MHz of a modern non-hyperhtreaded core for each rps you want to do tls handshakes for. This is <40kbit/s in each direction. That's a measly 2.5 Gbit/s symmetric on one of AMD's 64 core flagship processors. They can normally push out ~1Tbit/s http unencrypted if the SSDs keep up, or about 20~50% of that if they have to encrypt it. These numbers are all rough, but they are all for the absolute core/unflexible C-code that barely complies with the RFCs.
- bArray 7y ago
  > The issue is that TLS handshakes are actually quite > expensive. Agreed and a very good point. IPs that end up on the "blacklist" are rejected before any handshake occurs. The description I gave was really quite simplified.

by bArray 7y ago

tpetry 7y ago

Your defense mechanism is actually the exact thing described as protection against a simple DDOS attack. With a REAL ddos attack you will get so many incoming traffic that your bandwidth is saturated before any software will come in effect. A real ddos is a hardware problem, and it cant therefore be solved by software.

cannonedhamster 7y ago
Not to mention his solution really does nothing against ping, UDP, and TCP traffic. It's really only effective against http traffic. Which means his upstream or provider is probably going to cut him loose for the duration of the attack. A single person cannot effectively block a concerted DDOS.
- bArray 7y ago
  > Not to mention his solution really does nothing against > ping, UDP, and TCP traffic. Please bare in mind that the description is heavily simplified. If you're talking about bandwidth, of course this is the limit. > It's really only effective against http traffic. How so? > Which means his upstream or provider is probably going to > cut him loose for the duration of the attack. There's nothing we can do about that, but being in the cloud offers at least some protection. DDoS'ing some random Pi at home is a little different from DDoS'ing an AWS server. > A single person cannot effectively block a concerted DDOS. Of course not, but that doesn't mean you have to make it easy.
bArray 7y ago
> Your defense mechanism is actually the exact thing > described as protection against a simple DDOS attack. With > a REAL ddos attack [..] I'm not entirely sure what is really classified as a "simple" vs "real" DDoS attack. > With a REAL ddos attack you will get so many incoming > traffic that your bandwidth is saturated before any > software will come in effect. I specifically said: "as long as the network bandwidth itself can hold out". If the network is saturated, it's saturated - it's game over, for anybody. But you'll find that most services will die way before this limit is reached. On a WordPress site for example you'll usually find the database will give out long before the bandwidth is saturated. Even with legitimate users coming through some great filter like CloudFlare, it doesn't prevent a hug-of-death if you don't have some "smart" application level handling of high numbers of users. Like Linux OOM, when you hit your limits, the only choice left is to try and quickly and intelligently start killing without affecting the well behaved.

namibj 7y ago

The issue is that TLS handshakes are actually quite expensive. Assume a limit at about 3MHz of a modern non-hyperhtreaded core for each rps you want to do tls handshakes for. This is <40kbit/s in each direction. That's a measly 2.5 Gbit/s symmetric on one of AMD's 64 core flagship processors. They can normally push out ~1Tbit/s http unencrypted if the SSDs keep up, or about 20~50% of that if they have to encrypt it. These numbers are all rough, but they are all for the absolute core/unflexible C-code that barely complies with the RFCs.

bArray 7y ago
> The issue is that TLS handshakes are actually quite > expensive. Agreed and a very good point. IPs that end up on the "blacklist" are rejected before any handshake occurs. The description I gave was really quite simplified.