4 ms·

This just feels like a typical sales team fluff piece. I worked in this space, and constantly read articles of different providers claiming to have blocked the

This just feels like a typical sales team fluff piece. I worked in this space, and constantly read articles of different providers claiming to have blocked the biggest attack ever. Some go by total bandwidth, some by pps, some by number of actors, and apparently some by rps. Not that it isn't impressive, still, but hardly seems newsworthy.

blibble 4y ago
indeed I suspect a single machine could put out 71M HTTP requests/second without too much difficulty (but cloudflare never misses an opportunity for a blog post)
- tmikaeld 4y ago
  Smallest packet flood size is like ~60 bytes, so ~16M req/s on a 1Gbit connection or ~166M req/s on 10 Gbit. But I'm sure most providers or ISPs would block such output - thus, why it's a DDOS.
- gjulianm 4y ago
  I wouldn't be so dismissive of 71M rps. At around 100 bytes per packet, that's ~70 Gbps minimum, only on requests. It's quite the volume of traffic, you'd need quite a bit of cores, good parsing code and good core and memory distribution. It isn't impossible but it isn't trivial either. Edit: I misunderstood the comment and thought about "ingestion" and not "generation"
- _8j50 4y ago
  You can spit out all the traffic you want but who is going to route at that rate for you? If you somehow get it down to say 600 byte per request, that is 42GBps and 340Gbps. That is 35 10G nics at wire rate.
- TimWolla 4y ago
  71M requests per second gives you about 14ns per request. WolframAlpha says that's about 4 times the latency of a L2 cache access and 0.8 times the latency of a Mutex lock / unlock: https://www.wolframalpha.com/input?i=1+second+%2F+71+million https://www.wolframalpha.com/input?i=1+second+%2F+71+million. So even if you have a massive amount of cores in that machine, you still don't really have much time to spend on a single request and a single mutex operation will exceed your budget. For comparison: HAProxy was able to deal with 2M requests per second on a single machine in 2021: https://www.haproxy.com/de/blog/haproxy-forwards-over-2-million-http-requests-per-second-on-a-single-aws-arm-instance/ https://www.haproxy.com/de/blog/haproxy-forwards-over-2-mill... (Disclosure: I'm a HAProxy community contributor).
  - blibble 4y ago
    > So even if you have a massive amount of cores in that machine, you still don't really have much time to spend on a single request and a single mutex operation will exceed your budget. fortunately it's no longer 2000 and I have more than one core, and my NIC has more than one queue generating requests is a lot less CPU intensive than parsing requests how much CPU do I have to spend to get a pre-formed 100 byte request into the NICs queue? not much at all (the TCP negotiation will likely be the bottleneck)
    - TimWolla 4y ago
      I've interpreted the "put out" in the initial comment as in "put out a fire" (i.e. mitigate), instead of "send out".
    - robertlagrant 4y ago
      Could you do this and write it up? I'd love to see how you tune this stuff in hardware and software to get that sort of throughput.
    - markonen 4y ago
      These were HTTP/2 requests, not connections, so no per-request TCP negotiation. I'd say it's very easy to generate a huge number of requests like this, even if it's over TLS.
  - Aachen 4y ago
    But it's not one machine if you do anycast. It's a lot, but at cloudflare's scale you either have the budget for a crapton of machines per point of presence (hope I'm using the term correctly) or custom hardware that can deal with this sort of thing. It's kind of their core business.
- geraldwhen 4y ago
  You won’t get more than tens of thousands of concurrent network connections at most per machine. Your network switch is likely to fail first, or your operating systems ability to manage concurrent requests. 71M is a lot more network than you suspect.
- moremetadata 4y ago
  >I suspect a single machine could put out 71M HTTP requests/second without too much difficulty PacketsPerSecond=TransferRate(bytes/second)/PacketSize(bytes/packet) https://community.arubanetworks.com/community-home/digestviewer/viewthread?MID=30778 https://community.arubanetworks.com/community-home/digestvie... A single 24port switch with 1Gbps ports would see 48Gbps so with 64byte packets would see 71M packets, 1518byte packets would drop to 3.9M packets. If you are in the business of being the main auditor of your activities, water companies here in the UK being an example, news organisations as another, then does one ever find a problem with one's work? I've had DDOS on my systems before, and how it took my firewall out was simply increasing the number of requests the firewall had to handle using a SYN flood. https://en.wikipedia.org/wiki/SYN_flood https://en.wikipedia.org/wiki/SYN_flood Most firewalls are fixed and not adaptive, so a bit of low hanging fruit could be had by firewall systems that can adapt to SYN floods, like shortening the time the handshake process stays in memory, choosing to dump all that dont complete within a short period of time, choosing to dump those from ip address blocks linked to country's or regions your dont normally trade with. There are so many ways to handle these situations, but not all firewalls and switches provide the facilities that are needed. Hackers will exploit default configurations and industry standard best practices, including RFC's. During one of my lets see what happens if I push this metaphoric button exercises, I was tracing those random packets that appear at firewalls and I happened to have picked one which tracerouted back to a French IP. What was interesting and consistent with behaviour I've seen in the 90's with dead submarine internet connections at the New York exchange, was the trace route delivered its hop data until I crossed the English channel at which point a packet killed the machine, forcing the mains to be pulled as the machine went totally unresponsive. It makes me think the UK internet is like some giant virtual dome like the one seen in the film The Truman Show.
  - bogomipz 4y ago
    >"A single 24port switch with 1Gbps ports would see 48Gbps so with 64byte packets would see 71M packets, 1518byte packets would drop to 3.9M packets." This is not correct. An interface doesn't handle more packets per second with smaller packet sizes it handles less, much less. Getting the full 48 Gbps of 24 ports at line rate would only be possible if the packets were full size packets(1500 bytes). The PPS bottleneck with small packets(64 bytes) would mean you would never see line rate on those 24 interfaces as your throughput would fall off a cliff. Network hardware vendors quote PPS using the lower bound of packet size and line rate at full packet size. The PPS bottleneck on a network device is incurred long before the line rate limit is ever reached.
_8j50 4y ago
I worked in space as well and it is very impressive. But also, this is a blog post, not news. I would also want to know if Akamai sees similar attacks.
neom 4y ago
Usually I wouldn't post a Cloudflare mitigates blah blah blah blog, but I thought this one was particularly interesting because of the range of addresses, an attack that large spawning from over 30,000 IPs is a pretty well orchestrated attack. The most I've read pervious was well under half that (granted I've been out of networking for a long time).
- kortilla 4y ago
  30k isn’t even noteworthy in size for botnets thanks to shitty iot stuff. Mirai attacked Dyn with 100k unique devices.
  - nightfly 4y ago
    Comma in odd place. Could be typo of 30,000 or 300,000. Written the same way in the article
    - nbadg 4y ago
      Fun fact: how many digits get lumped together in a digit grouping is not universal and varies between different languages and cultures globally: https://en.wikipedia.org/wiki/Decimal_separator#Digit_grouping https://en.wikipedia.org/wiki/Decimal_separator#Digit_groupi...
      - neom 4y ago
        I used to work on bizdev with a German group and it took me a long time to catch on: "The character used as the thousands separator In the United States, this character is a comma (,). In Germany, it is a period (.). Thus one thousand and twenty-five is displayed as 1,025 in the United States and 1.025 in Germany. In Sweden, the thousands separator is a space. The character used as the decimal separator In the United States, this character is a period (.). In Germany, it is a comma (,). Thus one thousand twenty-five and seven tenths is displayed as 1,025.7 in the United States and 1.025,7 in Germany."
        4y ago
nosequel 4y ago
Somewhere in Akamai there's a group of network engineers laughing at these numbers I'm sure. Akamai just doesn't blog about every neat thing they do.

blibble 4y ago

indeed I suspect a single machine could put out 71M HTTP requests/second without too much difficulty (but cloudflare never misses an opportunity for a blog post)

tmikaeld 4y ago
Smallest packet flood size is like ~60 bytes, so ~16M req/s on a 1Gbit connection or ~166M req/s on 10 Gbit. But I'm sure most providers or ISPs would block such output - thus, why it's a DDOS.
gjulianm 4y ago
I wouldn't be so dismissive of 71M rps. At around 100 bytes per packet, that's ~70 Gbps minimum, only on requests. It's quite the volume of traffic, you'd need quite a bit of cores, good parsing code and good core and memory distribution. It isn't impossible but it isn't trivial either. Edit: I misunderstood the comment and thought about "ingestion" and not "generation"
_8j50 4y ago
You can spit out all the traffic you want but who is going to route at that rate for you? If you somehow get it down to say 600 byte per request, that is 42GBps and 340Gbps. That is 35 10G nics at wire rate.
TimWolla 4y ago
71M requests per second gives you about 14ns per request. WolframAlpha says that's about 4 times the latency of a L2 cache access and 0.8 times the latency of a Mutex lock / unlock: https://www.wolframalpha.com/input?i=1+second+%2F+71+million https://www.wolframalpha.com/input?i=1+second+%2F+71+million. So even if you have a massive amount of cores in that machine, you still don't really have much time to spend on a single request and a single mutex operation will exceed your budget. For comparison: HAProxy was able to deal with 2M requests per second on a single machine in 2021: https://www.haproxy.com/de/blog/haproxy-forwards-over-2-million-http-requests-per-second-on-a-single-aws-arm-instance/ https://www.haproxy.com/de/blog/haproxy-forwards-over-2-mill... (Disclosure: I'm a HAProxy community contributor).
- blibble 4y ago
  > So even if you have a massive amount of cores in that machine, you still don't really have much time to spend on a single request and a single mutex operation will exceed your budget. fortunately it's no longer 2000 and I have more than one core, and my NIC has more than one queue generating requests is a lot less CPU intensive than parsing requests how much CPU do I have to spend to get a pre-formed 100 byte request into the NICs queue? not much at all (the TCP negotiation will likely be the bottleneck)
  - TimWolla 4y ago
    I've interpreted the "put out" in the initial comment as in "put out a fire" (i.e. mitigate), instead of "send out".
  - robertlagrant 4y ago
    Could you do this and write it up? I'd love to see how you tune this stuff in hardware and software to get that sort of throughput.
  - markonen 4y ago
    These were HTTP/2 requests, not connections, so no per-request TCP negotiation. I'd say it's very easy to generate a huge number of requests like this, even if it's over TLS.
- Aachen 4y ago
  But it's not one machine if you do anycast. It's a lot, but at cloudflare's scale you either have the budget for a crapton of machines per point of presence (hope I'm using the term correctly) or custom hardware that can deal with this sort of thing. It's kind of their core business.
geraldwhen 4y ago
You won’t get more than tens of thousands of concurrent network connections at most per machine. Your network switch is likely to fail first, or your operating systems ability to manage concurrent requests. 71M is a lot more network than you suspect.
moremetadata 4y ago
>I suspect a single machine could put out 71M HTTP requests/second without too much difficulty PacketsPerSecond=TransferRate(bytes/second)/PacketSize(bytes/packet) https://community.arubanetworks.com/community-home/digestviewer/viewthread?MID=30778 https://community.arubanetworks.com/community-home/digestvie... A single 24port switch with 1Gbps ports would see 48Gbps so with 64byte packets would see 71M packets, 1518byte packets would drop to 3.9M packets. If you are in the business of being the main auditor of your activities, water companies here in the UK being an example, news organisations as another, then does one ever find a problem with one's work? I've had DDOS on my systems before, and how it took my firewall out was simply increasing the number of requests the firewall had to handle using a SYN flood. https://en.wikipedia.org/wiki/SYN_flood https://en.wikipedia.org/wiki/SYN_flood Most firewalls are fixed and not adaptive, so a bit of low hanging fruit could be had by firewall systems that can adapt to SYN floods, like shortening the time the handshake process stays in memory, choosing to dump all that dont complete within a short period of time, choosing to dump those from ip address blocks linked to country's or regions your dont normally trade with. There are so many ways to handle these situations, but not all firewalls and switches provide the facilities that are needed. Hackers will exploit default configurations and industry standard best practices, including RFC's. During one of my lets see what happens if I push this metaphoric button exercises, I was tracing those random packets that appear at firewalls and I happened to have picked one which tracerouted back to a French IP. What was interesting and consistent with behaviour I've seen in the 90's with dead submarine internet connections at the New York exchange, was the trace route delivered its hop data until I crossed the English channel at which point a packet killed the machine, forcing the mains to be pulled as the machine went totally unresponsive. It makes me think the UK internet is like some giant virtual dome like the one seen in the film The Truman Show.
- bogomipz 4y ago
  >"A single 24port switch with 1Gbps ports would see 48Gbps so with 64byte packets would see 71M packets, 1518byte packets would drop to 3.9M packets." This is not correct. An interface doesn't handle more packets per second with smaller packet sizes it handles less, much less. Getting the full 48 Gbps of 24 ports at line rate would only be possible if the packets were full size packets(1500 bytes). The PPS bottleneck with small packets(64 bytes) would mean you would never see line rate on those 24 interfaces as your throughput would fall off a cliff. Network hardware vendors quote PPS using the lower bound of packet size and line rate at full packet size. The PPS bottleneck on a network device is incurred long before the line rate limit is ever reached.

_8j50 4y ago

I worked in space as well and it is very impressive. But also, this is a blog post, not news. I would also want to know if Akamai sees similar attacks.

neom 4y ago

Usually I wouldn't post a Cloudflare mitigates blah blah blah blog, but I thought this one was particularly interesting because of the range of addresses, an attack that large spawning from over 30,000 IPs is a pretty well orchestrated attack. The most I've read pervious was well under half that (granted I've been out of networking for a long time).

kortilla 4y ago
30k isn’t even noteworthy in size for botnets thanks to shitty iot stuff. Mirai attacked Dyn with 100k unique devices.
- nightfly 4y ago
  Comma in odd place. Could be typo of 30,000 or 300,000. Written the same way in the article
  - nbadg 4y ago
    Fun fact: how many digits get lumped together in a digit grouping is not universal and varies between different languages and cultures globally: https://en.wikipedia.org/wiki/Decimal_separator#Digit_grouping https://en.wikipedia.org/wiki/Decimal_separator#Digit_groupi...
    - neom 4y ago
      I used to work on bizdev with a German group and it took me a long time to catch on: "The character used as the thousands separator In the United States, this character is a comma (,). In Germany, it is a period (.). Thus one thousand and twenty-five is displayed as 1,025 in the United States and 1.025 in Germany. In Sweden, the thousands separator is a space. The character used as the decimal separator In the United States, this character is a period (.). In Germany, it is a comma (,). Thus one thousand twenty-five and seven tenths is displayed as 1,025.7 in the United States and 1.025,7 in Germany."
      - 4y ago

nosequel 4y ago

Somewhere in Akamai there's a group of network engineers laughing at these numbers I'm sure. Akamai just doesn't blog about every neat thing they do.