6 ms·

Cloudflare Adaptive DDoS Protection

61 points by lanakei 4y ago

type_Ben_struct 4y ago
This is very similar to Google’s Cloud Armour Adaptive Protection[1] which recently fended off a 46 million RPS DDoS attack[2]. [1] https://cloud.google.com/armor/docs/adaptive-protection-overview https://cloud.google.com/armor/docs/adaptive-protection-over... [2] https://cloud.google.com/blog/products/identity-security/how-google-cloud-blocked-largest-layer-7-ddos-attack-at-46-million-rps https://cloud.google.com/blog/products/identity-security/how...
- majke 4y ago
  > Although terminating the encryption was necessary to inspect the traffic and effectively mitigate the attack, the use of HTTP Pipelining required Google to complete relatively few TLS handshakes. Ouch. I mean, congrats for allowing attack traffic to reach the servers, but you know what... once a request is recognized as "bad", then there are hight chances that other requests on the same http connection will also be bad perhaps? This is exactly, why a more appropriate mitigation is to issue connection-close (in http1.1 world) or GOAWAY frame (in http2 world). With that in hand, it's possible to bounce the connection back to the bot. Sure, the bot will retry which will put pressure on TLS. But then, if a source IP is trying very hard to establish many connections it's another strong sign of malicious traffic. In TCP you can rate limit SYN packets and _not_ allow more than X new connections per second, quite easily. Another mitigation is to allow the bot to send http requests, but just not answer them. Allow them hang on the connection, burning the bot's resources (memory, concurrent connetions). Anyway, the point is: requests-per-second is not a perfect metric for L7 / application attacks. A smart application shoudl be albe to push mitigation up to lower levels (like syn rate limiting, or TLS client-hello signature rate limiting), and _not_ intake large amounts of traffic. A better metric is bot count (or number of unique IP's seen), which was shared in the article and is in mid-range. To recap, a well designed system should _never_ need to handle 47MRPS. It should be able to packpressure it.
  - packetslave 4y ago
    Google doesn't terminate TLS on "the servers". It gets terminated on the frontend load balancers (GFE), which is presumably where the attack mitigation happened.
yjftsjthsd-h 4y ago
Oh good, a new way to break things; sure hope nobody using this expands into new markets or gets a surge of traffic outside of their usual viewers!
- jgrahamc 4y ago
  Sigh. You don't have to use this. Large companies that want super tailored DDoS protection can. Everyone else can just use our standard DDoS protection which is included with every plan and works: https://blog.cloudflare.com/26m-rps-ddos/ https://blog.cloudflare.com/26m-rps-ddos/
  - skyyler 4y ago
    Isn't it frustrating when users complain about new features that they aren't expected to use? Why is it like that? Is it like this in other industries? Do people that use Coach bags complain when Coach introduces a wallet that they won't use? I simply do not understand it.
    - ehPReth 4y ago
      Sort of, at least in fashion. This store https://www.bottegaveneta.com/en-ca/large-pillow-pouch-dark-green-717921V2G003009.html https://www.bottegaveneta.com/en-ca/large-pillow-pouch-dark-... caused a lot of mocking for their multi-thousand dollar normal looking shirts for example Don’t think it’s deserved in Cloudflare’s case though here.
    - charcircuit 4y ago
      Because when they are personally detected due to a false positive it is a poor experience.
    - ctoth 4y ago
      Not commenting on this particular case, but I know in smaller dev shops that time spent working on one feature is directly taken away from time spent on other things, and so if the new feature isn't useful to you, it has also taken fixing and dev time away from ones that might be. I also suspect that many people have seen the trajectory where apps get more and more features and don't focus on their core competency. Again, I don't believe this is the case here, but it might generally explain the phenomena/misplaced snark.
AcerbicZero 4y ago
Oh, I just figured Adaptive meant that they could adapt to not protecting a site faster, in case of backlash.
alberth 4y ago
FYI - this is Cloudflare "GA week" [0]. So we're going to see a bunch of beta/early-adopter service made GA this week. Other services made GA today, at the time of me writing this post: - Cloudflare One [1] - Cloudflare WAF [2] - Domain Scoped Roles [3] [0] https://blog.cloudflare.com/welcome-to-ga-week/ https://blog.cloudflare.com/welcome-to-ga-week/ [1] https://blog.cloudflare.com/cloudforce-one-is-now-ga/ https://blog.cloudflare.com/cloudforce-one-is-now-ga/ [2] https://blog.cloudflare.com/account-waf/ https://blog.cloudflare.com/account-waf/ [3] https://blog.cloudflare.com/domain-scoped-roles-ga/ https://blog.cloudflare.com/domain-scoped-roles-ga/
lxchase 4y ago
I used CF Magic Transit via one of their enterprise hosts to secure a game server from high 10-40 gbit/s attacks- worked great... however, it caused random dropped/slow UDP packets for legitimate users for a second every once in a while, and we can never figure out how to solve (my guess was something with Argo). Switched away and the problem went away. This adaptive program sounds similar to Corero SmartWall rules, but for their network vs an appliance.
- jgrahamc 4y ago
  Did you work with someone at Cloudflare on this?
  - lxchase 4y ago
    Sadly no, would have loved to, but had to go through the server host which makes sense as they are the direct customer (albeit maybe not as well invested in the specifics or diligent). I've used MT with 2 different server hosts (one an early partner of yours that moved to Path, and more recently one that still uses MT). Early days, MT had a few leaks (IoT device attacks), which I'm happy to say seems to be patched up so kudos to your team. This intermittent random UDP issue is more recent while I was trying the second host, but couldn't notice a pattern or replicate, so I ended up rolling my own with tc, port knocking, and the largest pipe we could find. Edit: For others, our problem was probably an edge case, and our use is probably not common. MT seems good compared to the alternative solutions I know so looking forward for an opp to try this if I have ever a need.
    - jgrahamc 4y ago
      Pity. I love debugging weird network things.
Gigachad 4y ago
These days instead of just spamming data at a site to ddos it, you just spam the twitter account of cloudflare and they drop your site.
- metadat 4y ago
  What? I don't follow, how do you spam the CloudFlare Twitter account?
  - Gigachad 4y ago
    The current playbook for when you want to take a site down is to sign up to it and post something illegal like a bomb threat. Then immediately screenshot it and post to Twitter. Have all of your followers retweet the screenshot to the cloudflare twitter account and the company will take down the site quickly. Doesn’t matter how quickly the sites moderators take your post down. Small site owners are expected to react within seconds while big tech is allowed days.
- seized 4y ago
  Useful contribution, defending a cesspit of hate like Kiwifarm. Or Stormfront. Cloudflares stance is extremely admirable but there should always be a limit. Things like Kiwifarm don't need protection.
  - Gigachad 4y ago
    Infrastructure platforms shouldn't be the deciders of what gets hosted online. Sure, those sites aren't nice places. But I'd much rather have governments take action on them if they are illegal than have CEOs in mega corps decide in combination with the court of twitter. Services like domain name providers and ddos protection are so essential for putting content online that they should be regulated as utilities. It would be absurd for a power or water company to cut service because they decided something legal but undesirable was happening at a building just because enough people tweeted at them. There are some services that should be allowed to pick and choose what to provide. A forum like HN or reddit for example is perfectly justified removing whatever they want. If you don't like it, go somewhere else. But once you get to the point where your domain name and phone number are getting canceled, its no longer possible.

61 points by lanakei 4y ago

This is very similar to Google’s Cloud Armour Adaptive Protection[1] which recently fended off a 46 million RPS DDoS attack[2]. [1] https://cloud.google.com/armor/docs/adaptive-protection-overview https://cloud.google.com/armor/docs/adaptive-protection-over... [2] https://cloud.google.com/blog/products/identity-security/how-google-cloud-blocked-largest-layer-7-ddos-attack-at-46-million-rps https://cloud.google.com/blog/products/identity-security/how...

majke 4y ago
> Although terminating the encryption was necessary to inspect the traffic and effectively mitigate the attack, the use of HTTP Pipelining required Google to complete relatively few TLS handshakes. Ouch. I mean, congrats for allowing attack traffic to reach the servers, but you know what... once a request is recognized as "bad", then there are hight chances that other requests on the same http connection will also be bad perhaps? This is exactly, why a more appropriate mitigation is to issue connection-close (in http1.1 world) or GOAWAY frame (in http2 world). With that in hand, it's possible to bounce the connection back to the bot. Sure, the bot will retry which will put pressure on TLS. But then, if a source IP is trying very hard to establish many connections it's another strong sign of malicious traffic. In TCP you can rate limit SYN packets and _not_ allow more than X new connections per second, quite easily. Another mitigation is to allow the bot to send http requests, but just not answer them. Allow them hang on the connection, burning the bot's resources (memory, concurrent connetions). Anyway, the point is: requests-per-second is not a perfect metric for L7 / application attacks. A smart application shoudl be albe to push mitigation up to lower levels (like syn rate limiting, or TLS client-hello signature rate limiting), and _not_ intake large amounts of traffic. A better metric is bot count (or number of unique IP's seen), which was shared in the article and is in mid-range. To recap, a well designed system should _never_ need to handle 47MRPS. It should be able to packpressure it.
- packetslave 4y ago
  Google doesn't terminate TLS on "the servers". It gets terminated on the frontend load balancers (GFE), which is presumably where the attack mitigation happened.

yjftsjthsd-h 4y ago

Oh good, a new way to break things; sure hope nobody using this expands into new markets or gets a surge of traffic outside of their usual viewers!

jgrahamc 4y ago
Sigh. You don't have to use this. Large companies that want super tailored DDoS protection can. Everyone else can just use our standard DDoS protection which is included with every plan and works: https://blog.cloudflare.com/26m-rps-ddos/ https://blog.cloudflare.com/26m-rps-ddos/
- skyyler 4y ago
  Isn't it frustrating when users complain about new features that they aren't expected to use? Why is it like that? Is it like this in other industries? Do people that use Coach bags complain when Coach introduces a wallet that they won't use? I simply do not understand it.
  - ehPReth 4y ago
    Sort of, at least in fashion. This store https://www.bottegaveneta.com/en-ca/large-pillow-pouch-dark-green-717921V2G003009.html https://www.bottegaveneta.com/en-ca/large-pillow-pouch-dark-... caused a lot of mocking for their multi-thousand dollar normal looking shirts for example Don’t think it’s deserved in Cloudflare’s case though here.
  - charcircuit 4y ago
    Because when they are personally detected due to a false positive it is a poor experience.
  - ctoth 4y ago
    Not commenting on this particular case, but I know in smaller dev shops that time spent working on one feature is directly taken away from time spent on other things, and so if the new feature isn't useful to you, it has also taken fixing and dev time away from ones that might be. I also suspect that many people have seen the trajectory where apps get more and more features and don't focus on their core competency. Again, I don't believe this is the case here, but it might generally explain the phenomena/misplaced snark.

AcerbicZero 4y ago

Oh, I just figured Adaptive meant that they could adapt to not protecting a site faster, in case of backlash.

alberth 4y ago

FYI - this is Cloudflare "GA week" [0]. So we're going to see a bunch of beta/early-adopter service made GA this week. Other services made GA today, at the time of me writing this post: - Cloudflare One [1] - Cloudflare WAF [2] - Domain Scoped Roles [3] [0] https://blog.cloudflare.com/welcome-to-ga-week/ https://blog.cloudflare.com/welcome-to-ga-week/ [1] https://blog.cloudflare.com/cloudforce-one-is-now-ga/ https://blog.cloudflare.com/cloudforce-one-is-now-ga/ [2] https://blog.cloudflare.com/account-waf/ https://blog.cloudflare.com/account-waf/ [3] https://blog.cloudflare.com/domain-scoped-roles-ga/ https://blog.cloudflare.com/domain-scoped-roles-ga/

lxchase 4y ago

I used CF Magic Transit via one of their enterprise hosts to secure a game server from high 10-40 gbit/s attacks- worked great... however, it caused random dropped/slow UDP packets for legitimate users for a second every once in a while, and we can never figure out how to solve (my guess was something with Argo). Switched away and the problem went away. This adaptive program sounds similar to Corero SmartWall rules, but for their network vs an appliance.

jgrahamc 4y ago
Did you work with someone at Cloudflare on this?
- lxchase 4y ago
  Sadly no, would have loved to, but had to go through the server host which makes sense as they are the direct customer (albeit maybe not as well invested in the specifics or diligent). I've used MT with 2 different server hosts (one an early partner of yours that moved to Path, and more recently one that still uses MT). Early days, MT had a few leaks (IoT device attacks), which I'm happy to say seems to be patched up so kudos to your team. This intermittent random UDP issue is more recent while I was trying the second host, but couldn't notice a pattern or replicate, so I ended up rolling my own with tc, port knocking, and the largest pipe we could find. Edit: For others, our problem was probably an edge case, and our use is probably not common. MT seems good compared to the alternative solutions I know so looking forward for an opp to try this if I have ever a need.
  - jgrahamc 4y ago
    Pity. I love debugging weird network things.

Gigachad 4y ago

These days instead of just spamming data at a site to ddos it, you just spam the twitter account of cloudflare and they drop your site.

metadat 4y ago
What? I don't follow, how do you spam the CloudFlare Twitter account?
- Gigachad 4y ago
  The current playbook for when you want to take a site down is to sign up to it and post something illegal like a bomb threat. Then immediately screenshot it and post to Twitter. Have all of your followers retweet the screenshot to the cloudflare twitter account and the company will take down the site quickly. Doesn’t matter how quickly the sites moderators take your post down. Small site owners are expected to react within seconds while big tech is allowed days.
seized 4y ago
Useful contribution, defending a cesspit of hate like Kiwifarm. Or Stormfront. Cloudflares stance is extremely admirable but there should always be a limit. Things like Kiwifarm don't need protection.
- Gigachad 4y ago
  Infrastructure platforms shouldn't be the deciders of what gets hosted online. Sure, those sites aren't nice places. But I'd much rather have governments take action on them if they are illegal than have CEOs in mega corps decide in combination with the court of twitter. Services like domain name providers and ddos protection are so essential for putting content online that they should be regulated as utilities. It would be absurd for a power or water company to cut service because they decided something legal but undesirable was happening at a building just because enough people tweeted at them. There are some services that should be allowed to pick and choose what to provide. A forum like HN or reddit for example is perfectly justified removing whatever they want. If you don't like it, go somewhere else. But once you get to the point where your domain name and phone number are getting canceled, its no longer possible.