2 ms·

indeed I suspect a single machine could put out 71M HTTP requests/second without too much difficulty (but cloudflare never misses an opportunity for a blog po

indeed I suspect a single machine could put out 71M HTTP requests/second without too much difficulty (but cloudflare never misses an opportunity for a blog post)

tmikaeld 4y ago
Smallest packet flood size is like ~60 bytes, so ~16M req/s on a 1Gbit connection or ~166M req/s on 10 Gbit. But I'm sure most providers or ISPs would block such output - thus, why it's a DDOS.
gjulianm 4y ago
I wouldn't be so dismissive of 71M rps. At around 100 bytes per packet, that's ~70 Gbps minimum, only on requests. It's quite the volume of traffic, you'd need quite a bit of cores, good parsing code and good core and memory distribution. It isn't impossible but it isn't trivial either. Edit: I misunderstood the comment and thought about "ingestion" and not "generation"
_8j50 4y ago
You can spit out all the traffic you want but who is going to route at that rate for you? If you somehow get it down to say 600 byte per request, that is 42GBps and 340Gbps. That is 35 10G nics at wire rate.
TimWolla 4y ago
71M requests per second gives you about 14ns per request. WolframAlpha says that's about 4 times the latency of a L2 cache access and 0.8 times the latency of a Mutex lock / unlock: https://www.wolframalpha.com/input?i=1+second+%2F+71+million https://www.wolframalpha.com/input?i=1+second+%2F+71+million. So even if you have a massive amount of cores in that machine, you still don't really have much time to spend on a single request and a single mutex operation will exceed your budget. For comparison: HAProxy was able to deal with 2M requests per second on a single machine in 2021: https://www.haproxy.com/de/blog/haproxy-forwards-over-2-million-http-requests-per-second-on-a-single-aws-arm-instance/ https://www.haproxy.com/de/blog/haproxy-forwards-over-2-mill... (Disclosure: I'm a HAProxy community contributor).
- blibble 4y ago
  > So even if you have a massive amount of cores in that machine, you still don't really have much time to spend on a single request and a single mutex operation will exceed your budget. fortunately it's no longer 2000 and I have more than one core, and my NIC has more than one queue generating requests is a lot less CPU intensive than parsing requests how much CPU do I have to spend to get a pre-formed 100 byte request into the NICs queue? not much at all (the TCP negotiation will likely be the bottleneck)
  - TimWolla 4y ago
    I've interpreted the "put out" in the initial comment as in "put out a fire" (i.e. mitigate), instead of "send out".
  - robertlagrant 4y ago
    Could you do this and write it up? I'd love to see how you tune this stuff in hardware and software to get that sort of throughput.
  - markonen 4y ago
    These were HTTP/2 requests, not connections, so no per-request TCP negotiation. I'd say it's very easy to generate a huge number of requests like this, even if it's over TLS.
- Aachen 4y ago
  But it's not one machine if you do anycast. It's a lot, but at cloudflare's scale you either have the budget for a crapton of machines per point of presence (hope I'm using the term correctly) or custom hardware that can deal with this sort of thing. It's kind of their core business.
geraldwhen 4y ago
You won’t get more than tens of thousands of concurrent network connections at most per machine. Your network switch is likely to fail first, or your operating systems ability to manage concurrent requests. 71M is a lot more network than you suspect.
moremetadata 4y ago
>I suspect a single machine could put out 71M HTTP requests/second without too much difficulty PacketsPerSecond=TransferRate(bytes/second)/PacketSize(bytes/packet) https://community.arubanetworks.com/community-home/digestviewer/viewthread?MID=30778 https://community.arubanetworks.com/community-home/digestvie... A single 24port switch with 1Gbps ports would see 48Gbps so with 64byte packets would see 71M packets, 1518byte packets would drop to 3.9M packets. If you are in the business of being the main auditor of your activities, water companies here in the UK being an example, news organisations as another, then does one ever find a problem with one's work? I've had DDOS on my systems before, and how it took my firewall out was simply increasing the number of requests the firewall had to handle using a SYN flood. https://en.wikipedia.org/wiki/SYN_flood https://en.wikipedia.org/wiki/SYN_flood Most firewalls are fixed and not adaptive, so a bit of low hanging fruit could be had by firewall systems that can adapt to SYN floods, like shortening the time the handshake process stays in memory, choosing to dump all that dont complete within a short period of time, choosing to dump those from ip address blocks linked to country's or regions your dont normally trade with. There are so many ways to handle these situations, but not all firewalls and switches provide the facilities that are needed. Hackers will exploit default configurations and industry standard best practices, including RFC's. During one of my lets see what happens if I push this metaphoric button exercises, I was tracing those random packets that appear at firewalls and I happened to have picked one which tracerouted back to a French IP. What was interesting and consistent with behaviour I've seen in the 90's with dead submarine internet connections at the New York exchange, was the trace route delivered its hop data until I crossed the English channel at which point a packet killed the machine, forcing the mains to be pulled as the machine went totally unresponsive. It makes me think the UK internet is like some giant virtual dome like the one seen in the film The Truman Show.
- bogomipz 4y ago
  >"A single 24port switch with 1Gbps ports would see 48Gbps so with 64byte packets would see 71M packets, 1518byte packets would drop to 3.9M packets." This is not correct. An interface doesn't handle more packets per second with smaller packet sizes it handles less, much less. Getting the full 48 Gbps of 24 ports at line rate would only be possible if the packets were full size packets(1500 bytes). The PPS bottleneck with small packets(64 bytes) would mean you would never see line rate on those 24 interfaces as your throughput would fall off a cliff. Network hardware vendors quote PPS using the lower bound of packet size and line rate at full packet size. The PPS bottleneck on a network device is incurred long before the line rate limit is ever reached.

indeed I suspect a single machine could put out 71M HTTP requests/second without too much difficulty (but cloudflare never misses an opportunity for a blog post)

tmikaeld 4y ago

Smallest packet flood size is like ~60 bytes, so ~16M req/s on a 1Gbit connection or ~166M req/s on 10 Gbit. But I'm sure most providers or ISPs would block such output - thus, why it's a DDOS.

gjulianm 4y ago

I wouldn't be so dismissive of 71M rps. At around 100 bytes per packet, that's ~70 Gbps minimum, only on requests. It's quite the volume of traffic, you'd need quite a bit of cores, good parsing code and good core and memory distribution. It isn't impossible but it isn't trivial either. Edit: I misunderstood the comment and thought about "ingestion" and not "generation"

_8j50 4y ago

You can spit out all the traffic you want but who is going to route at that rate for you? If you somehow get it down to say 600 byte per request, that is 42GBps and 340Gbps. That is 35 10G nics at wire rate.

TimWolla 4y ago

71M requests per second gives you about 14ns per request. WolframAlpha says that's about 4 times the latency of a L2 cache access and 0.8 times the latency of a Mutex lock / unlock: https://www.wolframalpha.com/input?i=1+second+%2F+71+million https://www.wolframalpha.com/input?i=1+second+%2F+71+million. So even if you have a massive amount of cores in that machine, you still don't really have much time to spend on a single request and a single mutex operation will exceed your budget. For comparison: HAProxy was able to deal with 2M requests per second on a single machine in 2021: https://www.haproxy.com/de/blog/haproxy-forwards-over-2-million-http-requests-per-second-on-a-single-aws-arm-instance/ https://www.haproxy.com/de/blog/haproxy-forwards-over-2-mill... (Disclosure: I'm a HAProxy community contributor).

blibble 4y ago
> So even if you have a massive amount of cores in that machine, you still don't really have much time to spend on a single request and a single mutex operation will exceed your budget. fortunately it's no longer 2000 and I have more than one core, and my NIC has more than one queue generating requests is a lot less CPU intensive than parsing requests how much CPU do I have to spend to get a pre-formed 100 byte request into the NICs queue? not much at all (the TCP negotiation will likely be the bottleneck)
- TimWolla 4y ago
  I've interpreted the "put out" in the initial comment as in "put out a fire" (i.e. mitigate), instead of "send out".
- robertlagrant 4y ago
  Could you do this and write it up? I'd love to see how you tune this stuff in hardware and software to get that sort of throughput.
- markonen 4y ago
  These were HTTP/2 requests, not connections, so no per-request TCP negotiation. I'd say it's very easy to generate a huge number of requests like this, even if it's over TLS.
Aachen 4y ago
But it's not one machine if you do anycast. It's a lot, but at cloudflare's scale you either have the budget for a crapton of machines per point of presence (hope I'm using the term correctly) or custom hardware that can deal with this sort of thing. It's kind of their core business.

geraldwhen 4y ago

You won’t get more than tens of thousands of concurrent network connections at most per machine. Your network switch is likely to fail first, or your operating systems ability to manage concurrent requests. 71M is a lot more network than you suspect.

moremetadata 4y ago

>I suspect a single machine could put out 71M HTTP requests/second without too much difficulty PacketsPerSecond=TransferRate(bytes/second)/PacketSize(bytes/packet) https://community.arubanetworks.com/community-home/digestviewer/viewthread?MID=30778 https://community.arubanetworks.com/community-home/digestvie... A single 24port switch with 1Gbps ports would see 48Gbps so with 64byte packets would see 71M packets, 1518byte packets would drop to 3.9M packets. If you are in the business of being the main auditor of your activities, water companies here in the UK being an example, news organisations as another, then does one ever find a problem with one's work? I've had DDOS on my systems before, and how it took my firewall out was simply increasing the number of requests the firewall had to handle using a SYN flood. https://en.wikipedia.org/wiki/SYN_flood https://en.wikipedia.org/wiki/SYN_flood Most firewalls are fixed and not adaptive, so a bit of low hanging fruit could be had by firewall systems that can adapt to SYN floods, like shortening the time the handshake process stays in memory, choosing to dump all that dont complete within a short period of time, choosing to dump those from ip address blocks linked to country's or regions your dont normally trade with. There are so many ways to handle these situations, but not all firewalls and switches provide the facilities that are needed. Hackers will exploit default configurations and industry standard best practices, including RFC's. During one of my lets see what happens if I push this metaphoric button exercises, I was tracing those random packets that appear at firewalls and I happened to have picked one which tracerouted back to a French IP. What was interesting and consistent with behaviour I've seen in the 90's with dead submarine internet connections at the New York exchange, was the trace route delivered its hop data until I crossed the English channel at which point a packet killed the machine, forcing the mains to be pulled as the machine went totally unresponsive. It makes me think the UK internet is like some giant virtual dome like the one seen in the film The Truman Show.

bogomipz 4y ago
>"A single 24port switch with 1Gbps ports would see 48Gbps so with 64byte packets would see 71M packets, 1518byte packets would drop to 3.9M packets." This is not correct. An interface doesn't handle more packets per second with smaller packet sizes it handles less, much less. Getting the full 48 Gbps of 24 ports at line rate would only be possible if the packets were full size packets(1500 bytes). The PPS bottleneck with small packets(64 bytes) would mean you would never see line rate on those 24 interfaces as your throughput would fall off a cliff. Network hardware vendors quote PPS using the lower bound of packet size and line rate at full packet size. The PPS bottleneck on a network device is incurred long before the line rate limit is ever reached.