11 ms·

Cloudflare mitigates record-breaking 71M request-per-second DDoS attack

204 points by neom 4y ago

atominte 4y ago
[flagged]
- bloodyplonker22 4y ago
  If you actually took the time to read the article, it says the following: >Over the past few days, Cloudflare, as well as other sources, have observed healthcare organizations targeted by a pro-Russian hacktivist group claiming to be Killnet.
  - nielsbot 4y ago
    You quoted the wrong article. This article says > Is this related to the Super Bowl or Killnet? > No. This campaign of attacks arrives less than two weeks after the Killnet DDoS campaign that targeted healthcare websites.
  - deleted 4y ago
    [deleted]
silisili 4y ago
This just feels like a typical sales team fluff piece. I worked in this space, and constantly read articles of different providers claiming to have blocked the biggest attack ever. Some go by total bandwidth, some by pps, some by number of actors, and apparently some by rps. Not that it isn't impressive, still, but hardly seems newsworthy.
- blibble 4y ago
  indeed I suspect a single machine could put out 71M HTTP requests/second without too much difficulty (but cloudflare never misses an opportunity for a blog post)
  - tmikaeld 4y ago
    Smallest packet flood size is like ~60 bytes, so ~16M req/s on a 1Gbit connection or ~166M req/s on 10 Gbit. But I'm sure most providers or ISPs would block such output - thus, why it's a DDOS.
  - gjulianm 4y ago
    I wouldn't be so dismissive of 71M rps. At around 100 bytes per packet, that's ~70 Gbps minimum, only on requests. It's quite the volume of traffic, you'd need quite a bit of cores, good parsing code and good core and memory distribution. It isn't impossible but it isn't trivial either. Edit: I misunderstood the comment and thought about "ingestion" and not "generation"
  - _8j50 4y ago
    You can spit out all the traffic you want but who is going to route at that rate for you? If you somehow get it down to say 600 byte per request, that is 42GBps and 340Gbps. That is 35 10G nics at wire rate.
  - TimWolla 4y ago
    71M requests per second gives you about 14ns per request. WolframAlpha says that's about 4 times the latency of a L2 cache access and 0.8 times the latency of a Mutex lock / unlock: https://www.wolframalpha.com/input?i=1+second+%2F+71+million https://www.wolframalpha.com/input?i=1+second+%2F+71+million. So even if you have a massive amount of cores in that machine, you still don't really have much time to spend on a single request and a single mutex operation will exceed your budget. For comparison: HAProxy was able to deal with 2M requests per second on a single machine in 2021: https://www.haproxy.com/de/blog/haproxy-forwards-over-2-million-http-requests-per-second-on-a-single-aws-arm-instance/ https://www.haproxy.com/de/blog/haproxy-forwards-over-2-mill... (Disclosure: I'm a HAProxy community contributor).
    - blibble 4y ago
      > So even if you have a massive amount of cores in that machine, you still don't really have much time to spend on a single request and a single mutex operation will exceed your budget. fortunately it's no longer 2000 and I have more than one core, and my NIC has more than one queue generating requests is a lot less CPU intensive than parsing requests how much CPU do I have to spend to get a pre-formed 100 byte request into the NICs queue? not much at all (the TCP negotiation will likely be the bottleneck)
- _8j50 4y ago
  I worked in space as well and it is very impressive. But also, this is a blog post, not news. I would also want to know if Akamai sees similar attacks.
- neom 4y ago
  Usually I wouldn't post a Cloudflare mitigates blah blah blah blog, but I thought this one was particularly interesting because of the range of addresses, an attack that large spawning from over 30,000 IPs is a pretty well orchestrated attack. The most I've read pervious was well under half that (granted I've been out of networking for a long time).
  - kortilla 4y ago
    30k isn’t even noteworthy in size for botnets thanks to shitty iot stuff. Mirai attacked Dyn with 100k unique devices.
    - nightfly 4y ago
      Comma in odd place. Could be typo of 30,000 or 300,000. Written the same way in the article
      - nbadg 4y ago
        Fun fact: how many digits get lumped together in a digit grouping is not universal and varies between different languages and cultures globally: https://en.wikipedia.org/wiki/Decimal_separator#Digit_grouping https://en.wikipedia.org/wiki/Decimal_separator#Digit_groupi...
        neom 4y ago
        I used to work on bizdev with a German group and it took me a long time to catch on: "The character used as the thousands separator In the United States, this character is a comma (,). In Germany, it is a period (.). Thus one thousand and twenty-five is displayed as 1,025 in the United States and 1.025 in Germany. In Sweden, the thousands separator is a space. The character used as the decimal separator In the United States, this character is a period (.). In Germany, it is a comma (,). Thus one thousand twenty-five and seven tenths is displayed as 1,025.7 in the United States and 1.025,7 in Germany."
        4y ago
- nosequel 4y ago
  Somewhere in Akamai there's a group of network engineers laughing at these numbers I'm sure. Akamai just doesn't blog about every neat thing they do.
realistik1 4y ago
The startup I work for blocks all of china, russia, belarus and serbian IP space, which saves us about $50kper month in bandwidth charges. The absolutely first thing I do at every company and on every project is ask if I can block russia, china and belarussian IP space, and add all of their ASNs to the bogon list if we run our own bgp. They are never customers to the businesses of the companies I work for, but they sure manage to harm our businesses. Russisns mostly try to disrupt and steal because thats their foreign policy. Chinese just try to destroy and censor. Worked at a company which hosted presentations, one of their 5 million presentations was critical of China. We received over 10,000 phone calls and close to a million emails in two days, and a sustained 10gbps ddos attack for a week. Letting russia and china on the internet was a massive security mistake we should rectify. The world spends literally BILLIONS because chinarussia are frankly assholes online. Edit: corrected $500k per month to $50kb per month.
- deleted 4y ago
  [deleted]
- iJohnDoe 4y ago
  Doesn’t deserve the downvotes. All of it is truth. So much garbage connections originate from the mentioned countries. Worst yet, these countries have poor connections in some cases and generate so many retires that also waste resources.
  - realistik1 4y ago
    [flagged]
    - deleted 4y ago
      [deleted]
- archerx 4y ago
  I would add Turkey to that list. Many many years ago I had a kind of popular podcast about computer graphics and all of my grief was coming from Turkey. After I banned all of Turkey‘s IPs my drama was over. Occasionally I would get an email from someone saying they couldn’t listen to the episodes and I would explain to them why their country is blocked and they would say it’s not fair, I couldn’t disagree, its not fair but such is life.
omgomgomgomg 4y ago
Cloudflare ddos mitigation is overrated. First they force you to upgrade your plan, if that does not help, anotyer upgrade, if that fails, chances are you will get a termination notice.
- sammy2255 4y ago
  >First they force you to upgrade your plan I don’t think that’s true, would you care to share more about that experience?
  - Maxious 4y ago
    Cloudflare recently admitted that while their policy wasn't officially to force upgrades, some employees had taken it upon themselves to do so > This throttle should not have happened. In addition, there is and was no need for the customer to upgrade to some other plan level. https://blog.cloudflare.com/how-cloudflare-erroneously-throttled-a-customers-web-traffic/ https://blog.cloudflare.com/how-cloudflare-erroneously-throt... This came about from this HN thread https://news.ycombinator.com/item?id=34639212 https://news.ycombinator.com/item?id=34639212
- vlakreeh 4y ago
  This is not the case, websites behind will get good ddos mitigation to begin with some options for improving that for some cost. But it's worth noting that in 2022 they blocked a 26M req/s attack on a website using the free plan, imo that's very generous for someone not paying for ddos protection. https://blog.cloudflare.com/26m-rps-ddos/ https://blog.cloudflare.com/26m-rps-ddos/
skepticalloy 4y ago
Is there any independent verification of this? Or any way of confirming any of the details? Just seems like a Cloudflare puff piece otherwise. I would be more interested in a critical article about where they failed to block a DDoS attack, and why. Plus what could be done in the future to mitigate.
kortilla 4y ago
This is blogspam complete with made up terms (“hyper-volumetric”), self-congratulatory back-patting, and no technical content. It’s an ad to scare you into signing up for cloudflare.
- throwaway67743 4y ago
  Which is why they're colloquially known as blogflare ;)
IYasha 4y ago
Me and millions of users of "obsolete" software and hardware wish CloudFlare slow and painful death. I mean, require solving 65535 useless captchas to die. And "sorry, you are not allowed to die now. Try some other time. Meanwhile why don't you learn how we protect the heavens and hell from freeloaders like you!" after that. And repeat. 71M times.
- SketchySeaBeast 4y ago
  What "obsolete" software and hardware are you using that's somehow blocked by Cloudflare?
  - IYasha 4y ago
    NetFront on the phone, IceCat mobile (never gets updates) or internal WebView browser on a smartpone, some unknown browser on samsung tv, Firefox 50 on a Linux laptop, but using others often doesn't help. For a while on these I get "screw yourself" message right away, while on others it's 5-30 captcha solving (AI feeding) sessions. And not only me. People with stock android 7 and older browsers have this too. And users of "suspicious" (e.g. non-chrome) browsers too. Oh, and these proxy pages forcing me to use JS (which I clearly don't want), and provide training to some AI (same here).
    - SketchySeaBeast 4y ago
      Unfortunately the internet moves fast, and if you're using a smart TV (prime for DDOS) or (in internet years) old tech it's the same as trying to drive a model t on the highway - sure it's a perfectly serviceable vehicle but you're probably going to get pulled over more often because you're a danger to yourself and others.
      - IYasha 4y ago
        That's why I hate proprietary sh*t. A great TV with no way to update it or make it usable (no, I won't plug it into another computer). A great smartphone/toaster/anything else with the same problem...
dustedcodes 4y ago
I took off all my properties from Cloudflare many years ago because I got increasingly more uncomfortable with a single US company who I know nothing about to sniffing the entire internet's traffic through their servers and ironically my properties had much less issues overall. Whatever value Cloudflare was supposedly adding to us, it couldn't be noticed or quantified in any measurable metric that we could see. The only thing we noticed is that we cut out one extra single point of failure. When Cloudflare had some outages we were unaffected by it, overall increasing our overall availability.
- ranguna 4y ago
  How do you handle attacks now? I tried to move one of my personal services off of cloudflare and it got DDoSed a few weeks later, mind you it's not even a popular services, just for internal use of mine and a few friends.
superkuh 4y ago
Cloudflare is a distributed denial of service to anyone that doesn't run javascript or have the latest browser.
causi 4y ago
Sure, Cloudflare can protect you against almost any realistic technical attack. Let me know when they have a service tier that protects me from a hate mob writing angry e-mails, because currently nobody on Cloudflare is safe from that.
rwky 4y ago
That would explain the vultr blocking https://news.ycombinator.com/item?id=34758393 https://news.ycombinator.com/item?id=34758393

204 points by neom 4y ago

atominte 4y ago

[flagged]

bloodyplonker22 4y ago
If you actually took the time to read the article, it says the following: >Over the past few days, Cloudflare, as well as other sources, have observed healthcare organizations targeted by a pro-Russian hacktivist group claiming to be Killnet.
- nielsbot 4y ago
  You quoted the wrong article. This article says > Is this related to the Super Bowl or Killnet? > No. This campaign of attacks arrives less than two weeks after the Killnet DDoS campaign that targeted healthcare websites.
- deleted 4y ago
  [deleted]

silisili 4y ago

This just feels like a typical sales team fluff piece. I worked in this space, and constantly read articles of different providers claiming to have blocked the biggest attack ever. Some go by total bandwidth, some by pps, some by number of actors, and apparently some by rps. Not that it isn't impressive, still, but hardly seems newsworthy.

blibble 4y ago
indeed I suspect a single machine could put out 71M HTTP requests/second without too much difficulty (but cloudflare never misses an opportunity for a blog post)
- tmikaeld 4y ago
  Smallest packet flood size is like ~60 bytes, so ~16M req/s on a 1Gbit connection or ~166M req/s on 10 Gbit. But I'm sure most providers or ISPs would block such output - thus, why it's a DDOS.
- gjulianm 4y ago
  I wouldn't be so dismissive of 71M rps. At around 100 bytes per packet, that's ~70 Gbps minimum, only on requests. It's quite the volume of traffic, you'd need quite a bit of cores, good parsing code and good core and memory distribution. It isn't impossible but it isn't trivial either. Edit: I misunderstood the comment and thought about "ingestion" and not "generation"
- _8j50 4y ago
  You can spit out all the traffic you want but who is going to route at that rate for you? If you somehow get it down to say 600 byte per request, that is 42GBps and 340Gbps. That is 35 10G nics at wire rate.
- TimWolla 4y ago
  71M requests per second gives you about 14ns per request. WolframAlpha says that's about 4 times the latency of a L2 cache access and 0.8 times the latency of a Mutex lock / unlock: https://www.wolframalpha.com/input?i=1+second+%2F+71+million https://www.wolframalpha.com/input?i=1+second+%2F+71+million. So even if you have a massive amount of cores in that machine, you still don't really have much time to spend on a single request and a single mutex operation will exceed your budget. For comparison: HAProxy was able to deal with 2M requests per second on a single machine in 2021: https://www.haproxy.com/de/blog/haproxy-forwards-over-2-million-http-requests-per-second-on-a-single-aws-arm-instance/ https://www.haproxy.com/de/blog/haproxy-forwards-over-2-mill... (Disclosure: I'm a HAProxy community contributor).
  - blibble 4y ago
    > So even if you have a massive amount of cores in that machine, you still don't really have much time to spend on a single request and a single mutex operation will exceed your budget. fortunately it's no longer 2000 and I have more than one core, and my NIC has more than one queue generating requests is a lot less CPU intensive than parsing requests how much CPU do I have to spend to get a pre-formed 100 byte request into the NICs queue? not much at all (the TCP negotiation will likely be the bottleneck)
_8j50 4y ago
I worked in space as well and it is very impressive. But also, this is a blog post, not news. I would also want to know if Akamai sees similar attacks.
neom 4y ago
Usually I wouldn't post a Cloudflare mitigates blah blah blah blog, but I thought this one was particularly interesting because of the range of addresses, an attack that large spawning from over 30,000 IPs is a pretty well orchestrated attack. The most I've read pervious was well under half that (granted I've been out of networking for a long time).
- kortilla 4y ago
  30k isn’t even noteworthy in size for botnets thanks to shitty iot stuff. Mirai attacked Dyn with 100k unique devices.
  - nightfly 4y ago
    Comma in odd place. Could be typo of 30,000 or 300,000. Written the same way in the article
    - nbadg 4y ago
      Fun fact: how many digits get lumped together in a digit grouping is not universal and varies between different languages and cultures globally: https://en.wikipedia.org/wiki/Decimal_separator#Digit_grouping https://en.wikipedia.org/wiki/Decimal_separator#Digit_groupi...
      - neom 4y ago
        I used to work on bizdev with a German group and it took me a long time to catch on: "The character used as the thousands separator In the United States, this character is a comma (,). In Germany, it is a period (.). Thus one thousand and twenty-five is displayed as 1,025 in the United States and 1.025 in Germany. In Sweden, the thousands separator is a space. The character used as the decimal separator In the United States, this character is a period (.). In Germany, it is a comma (,). Thus one thousand twenty-five and seven tenths is displayed as 1,025.7 in the United States and 1.025,7 in Germany."
        4y ago
nosequel 4y ago
Somewhere in Akamai there's a group of network engineers laughing at these numbers I'm sure. Akamai just doesn't blog about every neat thing they do.

realistik1 4y ago

The startup I work for blocks all of china, russia, belarus and serbian IP space, which saves us about $50kper month in bandwidth charges. The absolutely first thing I do at every company and on every project is ask if I can block russia, china and belarussian IP space, and add all of their ASNs to the bogon list if we run our own bgp. They are never customers to the businesses of the companies I work for, but they sure manage to harm our businesses. Russisns mostly try to disrupt and steal because thats their foreign policy. Chinese just try to destroy and censor. Worked at a company which hosted presentations, one of their 5 million presentations was critical of China. We received over 10,000 phone calls and close to a million emails in two days, and a sustained 10gbps ddos attack for a week. Letting russia and china on the internet was a massive security mistake we should rectify. The world spends literally BILLIONS because chinarussia are frankly assholes online. Edit: corrected $500k per month to $50kb per month.

deleted 4y ago
[deleted]
iJohnDoe 4y ago
Doesn’t deserve the downvotes. All of it is truth. So much garbage connections originate from the mentioned countries. Worst yet, these countries have poor connections in some cases and generate so many retires that also waste resources.
- realistik1 4y ago
  [flagged]
  - deleted 4y ago
    [deleted]
archerx 4y ago
I would add Turkey to that list. Many many years ago I had a kind of popular podcast about computer graphics and all of my grief was coming from Turkey. After I banned all of Turkey‘s IPs my drama was over. Occasionally I would get an email from someone saying they couldn’t listen to the episodes and I would explain to them why their country is blocked and they would say it’s not fair, I couldn’t disagree, its not fair but such is life.

omgomgomgomg 4y ago

Cloudflare ddos mitigation is overrated. First they force you to upgrade your plan, if that does not help, anotyer upgrade, if that fails, chances are you will get a termination notice.

sammy2255 4y ago
>First they force you to upgrade your plan I don’t think that’s true, would you care to share more about that experience?
- Maxious 4y ago
  Cloudflare recently admitted that while their policy wasn't officially to force upgrades, some employees had taken it upon themselves to do so > This throttle should not have happened. In addition, there is and was no need for the customer to upgrade to some other plan level. https://blog.cloudflare.com/how-cloudflare-erroneously-throttled-a-customers-web-traffic/ https://blog.cloudflare.com/how-cloudflare-erroneously-throt... This came about from this HN thread https://news.ycombinator.com/item?id=34639212 https://news.ycombinator.com/item?id=34639212
vlakreeh 4y ago
This is not the case, websites behind will get good ddos mitigation to begin with some options for improving that for some cost. But it's worth noting that in 2022 they blocked a 26M req/s attack on a website using the free plan, imo that's very generous for someone not paying for ddos protection. https://blog.cloudflare.com/26m-rps-ddos/ https://blog.cloudflare.com/26m-rps-ddos/

skepticalloy 4y ago

Is there any independent verification of this? Or any way of confirming any of the details? Just seems like a Cloudflare puff piece otherwise. I would be more interested in a critical article about where they failed to block a DDoS attack, and why. Plus what could be done in the future to mitigate.

kortilla 4y ago

This is blogspam complete with made up terms (“hyper-volumetric”), self-congratulatory back-patting, and no technical content. It’s an ad to scare you into signing up for cloudflare.

throwaway67743 4y ago
Which is why they're colloquially known as blogflare ;)

IYasha 4y ago

Me and millions of users of "obsolete" software and hardware wish CloudFlare slow and painful death. I mean, require solving 65535 useless captchas to die. And "sorry, you are not allowed to die now. Try some other time. Meanwhile why don't you learn how we protect the heavens and hell from freeloaders like you!" after that. And repeat. 71M times.

SketchySeaBeast 4y ago
What "obsolete" software and hardware are you using that's somehow blocked by Cloudflare?
- IYasha 4y ago
  NetFront on the phone, IceCat mobile (never gets updates) or internal WebView browser on a smartpone, some unknown browser on samsung tv, Firefox 50 on a Linux laptop, but using others often doesn't help. For a while on these I get "screw yourself" message right away, while on others it's 5-30 captcha solving (AI feeding) sessions. And not only me. People with stock android 7 and older browsers have this too. And users of "suspicious" (e.g. non-chrome) browsers too. Oh, and these proxy pages forcing me to use JS (which I clearly don't want), and provide training to some AI (same here).
  - SketchySeaBeast 4y ago
    Unfortunately the internet moves fast, and if you're using a smart TV (prime for DDOS) or (in internet years) old tech it's the same as trying to drive a model t on the highway - sure it's a perfectly serviceable vehicle but you're probably going to get pulled over more often because you're a danger to yourself and others.
    - IYasha 4y ago
      That's why I hate proprietary sh*t. A great TV with no way to update it or make it usable (no, I won't plug it into another computer). A great smartphone/toaster/anything else with the same problem...

dustedcodes 4y ago

I took off all my properties from Cloudflare many years ago because I got increasingly more uncomfortable with a single US company who I know nothing about to sniffing the entire internet's traffic through their servers and ironically my properties had much less issues overall. Whatever value Cloudflare was supposedly adding to us, it couldn't be noticed or quantified in any measurable metric that we could see. The only thing we noticed is that we cut out one extra single point of failure. When Cloudflare had some outages we were unaffected by it, overall increasing our overall availability.

ranguna 4y ago
How do you handle attacks now? I tried to move one of my personal services off of cloudflare and it got DDoSed a few weeks later, mind you it's not even a popular services, just for internal use of mine and a few friends.

superkuh 4y ago

Cloudflare is a distributed denial of service to anyone that doesn't run javascript or have the latest browser.

causi 4y ago

Sure, Cloudflare can protect you against almost any realistic technical attack. Let me know when they have a service tier that protects me from a hate mob writing angry e-mails, because currently nobody on Cloudflare is safe from that.

rwky 4y ago

That would explain the vultr blocking https://news.ycombinator.com/item?id=34758393 https://news.ycombinator.com/item?id=34758393