4 ms·

I'm very much not a network engineer, but I'd like to understand the magnitude of this issue because my intuition is wrong: 250 Gbps seems like it would defini

I'm very much not a network engineer, but I'd like to understand the magnitude of this issue because my intuition is wrong: 250 Gbps seems like it would definitely be a lot for a server or website but it also seem like a drop in the bucket for an ISP providing broadband for many customers. Clearly I'm wrong because it is an issue here. I'd like to understand why I'm wrong, and I hope that here, on HN, that's taken in the spirit of curiosity intended and not negativism. So, what am I missing? Maybe Belgian broadband is lower capacity than what I'm used to in a US metropolitan area? Maybe this particular ISP served a population too small to have a... um... "fat pipe"? I'd like to understand.

xhrpost 5y ago
250Gbps is still a lot of bandwidth for a network even today. An enterprise ISP like Verizon or Spectrum may have less of a capacity issue with it but for many other networks this is a lot. Quick example, though not an ISP, but the entire Internet archive operates on 60Gbps of bandwidth. So an attack of 250 would be quite a problem. http://blog.archive.org/2020/05/11/thank-you-for-helping-us-increase-our-bandwidth/ http://blog.archive.org/2020/05/11/thank-you-for-helping-us-...
- ineedasername 5y ago
  the entire Internet archive operates on 60Gbps of bandwidth Wow, yes, that definitely puts the magnitude of this into perspective. Thank you.
- Groxx 5y ago
  Though that's a substantial amount of traffic: isn't this currently going through one ISP, which (I would hope) has many more customers than just archive.org? "top 160 site" is significant, but still minuscule compared to all internet traffic.
  - therealdrag0 5y ago
    I think I’m thinking of it reverse from you. A top 160 site could have more concurrent users than a small regional ISP.
    - Groxx 5y ago
      It might! I suspect my intuition fails at this scale. I have absolutely no data, so I'd definitely be interested in any that can be found. In my defense though! 60Mbps is literally less than one billionth the estimated internet bandwidth these days: https://en.m.wikipedia.org/wiki/Internet_traffic https://en.m.wikipedia.org/wiki/Internet_traffic . I doubt there are even one million ISPs, so it still seems relatively "small" to me. It could definitely be bigger than [random company X] can handle, but it seems like it stands a good chance of being smaller than most in aggregate, so even if it's hard for X to support it's not particularly "interesting" in a state-of-the-art way. And I can completely believe I'm wildly wrong in this. 999/1000+ low-technical-effort ISPs doesn't seem unlikely to me, so 60Mbps could very well be a major achievement as far as I know. I'd be surprised, but... I've been surprised enough that I'm no longer surprised at being surprised. Humans are weird at scale.
      - markonen 5y ago
        The Wikipedia entry you linked to estimates 2021 internet traffic at 276 EB/month. That’s an average 839,625 gigabits per second. So 60 Gbps is one 14,000th of that. Of course, one figure was about capacity and the other about actual traffic, but still.
        Groxx 5y ago
        Ah, per month. Yep, I missed that, I thought it was per second.
- Aeolun 5y ago
  I dunno, my home operates on 2Gbps of bandwidth. 250Gbps is half of our neighborhood (not sure what the shared uplink size is though).
Hikikomori 5y ago
An ISP has two types of edge connections where ddos traffic comes into their network. Private peering where they are directly connected to their neighbor in one or more locations, typically neither party pays for traffic, can be 1 to 100G interfaces and multiple of them. Second type is transit, smaller or mid sized buys it from larger ISPs and pay for traffic. Everything that does not go directly to a peer mentioned before goes over this connection. Can also be very different in size. You can just add upp all edge Links and know their theoretical limit, but most of the ddos traffic likely comes from Asia and it's unlikely that they peer directly with those ISPs. So most of the traffic will go over transit, but the larger ISP likely has an automated system that handles volumetric attacks. So if there's any impact it will be on transit mostly affecting traffic to/from outside of Europe. 250 might be a lot for this ISP, but they might also have a lot more transit headroom, or it light be handled by their transit provider. Who knows.
- ineedasername 5y ago
  As an outsider to network topology that's fascinating to understand. Thanks for explaining. If you don't mind going further, how does an ISP (or anti-DDoS services) mitigate something like this without incurring even more overhead to examine the traffic & not route it onwards? Just a flat block against the sources, or something more sophisticated?
  - Hikikomori 5y ago
    Haven't done much traditional networking for a few years so my knowledge is not that current. They use bgp flowspec to send out filter lists using the bgp protocol to routers, essentially a long list of ACL rules with srcip dstip dstport. These routers can filter any amount of traffic it gets in hardware, so as long as your upstream (edge transit or peering links) are not full you won't even notice the ddos. But you need something to analyse traffic that can understand which traffic is ddos and what is normal. We used devices from Arbor which are basically regular x86 servers. These devices also received netflow so whenever a ddos target received a lot of weird traffic (lots of DNS for example) it was configured to redirect all traffic for that IP to itself where it could gain a better understanding than what you can from netflow and either filtered it locally or sent out flowspec updates to filter it on routers. You could also enable this manually for non-volumetric ddos and do some filtering, but it's hard to do if it's encrypted and if they are abusing some http call that is just expensive to handle for the servers.
bstpierre 5y ago
I don’t know anything about this particular isp or attack, but if that 250 gbps is aimed at specific parts of their infrastructure that are not designed for this much load, it’s likely to saturate individual links, servers, and/or network gear. Their announcement said “all of our services”, so this could be anything from their authoritative/recursive dns to monitoring or billing servers to specific routing nodes. So even if they had nice fat 100G pipes coming in, and those pipes weren’t saturated, a ddos of this size with a mix of vectors can be debilitating on enough discrete pieces of infrastructure that the isp is effectively “down” for many end users. (Again I am only speaking generally with guesses about what might be going on at this isp. I’m also not a network engineer but I do write code for network gear that does anti ddos stuff.)
perlgeek 5y ago
ISPs have a very decentralized network, that is, they have routers in a lot of places. If you sum up the available traffic capacity, you get a number that's far bigger than 250 Gbit/s. But that's not the right comparison to make. Much of that DDoS traffic goes through multiple of the ISP's routers, and if you're doing just a numbers comparison, you'd have to multiply the attack rate by the (average) number of hops. But things are even more complicated in reality: many smaller links will have only 10Gbit/s or 20Gbit/s or 40Gbit/s capacity (or maybe even 1G), and saturating them not only causes increased latency, but also routing the traffic around causes more overall load on the network. Then there are management systems behind firewalls, and if you DDoS them, it's typically the firewalls that give out first. Finally, customers don't get the full throughput internally available to an ISP; if you DDoS a customer's IP, their link will be saturated, even if the ISP itself has plenty to spare. (Not a network engineer myself, but working closely with some).
- Groxx 5y ago
  Wouldn't that all be true of normal web traffic too, presumably already capable of rates beyond 250Gbps? Though they may not have that much free headroom on top of normal traffic.
- oars 5y ago
  So in the case where the attackers are aiming their traffic at one of the ISP's routers to saturate a link, how do they get the IP address/domain name of this router?
  - perlgeek 5y ago
    Traceroutes. Peering information is also often public. More sinister options like insider information or leaked/exfiltrated network plans are also an option, but usually you get pretty far with standard network diagnostic tools.
deleted 5y ago
[deleted]
benp84 5y ago
For comparison, the entire internet averages on the order of 50 Tbps*, so this is about 0.5% of that. Residential internet might have a max rate of 1Gbps in the best conditions but if you use even 1% of that consistently you'll get overage fees. Steady bandwidth numbers are typically much lower than max bandwidth numbers. * Based on 122,000 PB/month in 2017 from https://en.wikipedia.org/wiki/Internet_traffic https://en.wikipedia.org/wiki/Internet_traffic, if my calcs are right
Gigachad 5y ago
It's not just 250Gbps, its 250 on top of normal usage. Most ISPs run a fairly efficient setup where they are only paying for a little bit over the average usage.

xhrpost 5y ago

250Gbps is still a lot of bandwidth for a network even today. An enterprise ISP like Verizon or Spectrum may have less of a capacity issue with it but for many other networks this is a lot. Quick example, though not an ISP, but the entire Internet archive operates on 60Gbps of bandwidth. So an attack of 250 would be quite a problem. http://blog.archive.org/2020/05/11/thank-you-for-helping-us-increase-our-bandwidth/ http://blog.archive.org/2020/05/11/thank-you-for-helping-us-...

ineedasername 5y ago
the entire Internet archive operates on 60Gbps of bandwidth Wow, yes, that definitely puts the magnitude of this into perspective. Thank you.
Groxx 5y ago
Though that's a substantial amount of traffic: isn't this currently going through one ISP, which (I would hope) has many more customers than just archive.org? "top 160 site" is significant, but still minuscule compared to all internet traffic.
- therealdrag0 5y ago
  I think I’m thinking of it reverse from you. A top 160 site could have more concurrent users than a small regional ISP.
  - Groxx 5y ago
    It might! I suspect my intuition fails at this scale. I have absolutely no data, so I'd definitely be interested in any that can be found. In my defense though! 60Mbps is literally less than one billionth the estimated internet bandwidth these days: https://en.m.wikipedia.org/wiki/Internet_traffic https://en.m.wikipedia.org/wiki/Internet_traffic . I doubt there are even one million ISPs, so it still seems relatively "small" to me. It could definitely be bigger than [random company X] can handle, but it seems like it stands a good chance of being smaller than most in aggregate, so even if it's hard for X to support it's not particularly "interesting" in a state-of-the-art way. And I can completely believe I'm wildly wrong in this. 999/1000+ low-technical-effort ISPs doesn't seem unlikely to me, so 60Mbps could very well be a major achievement as far as I know. I'd be surprised, but... I've been surprised enough that I'm no longer surprised at being surprised. Humans are weird at scale.
    - markonen 5y ago
      The Wikipedia entry you linked to estimates 2021 internet traffic at 276 EB/month. That’s an average 839,625 gigabits per second. So 60 Gbps is one 14,000th of that. Of course, one figure was about capacity and the other about actual traffic, but still.
      - Groxx 5y ago
        Ah, per month. Yep, I missed that, I thought it was per second.
Aeolun 5y ago
I dunno, my home operates on 2Gbps of bandwidth. 250Gbps is half of our neighborhood (not sure what the shared uplink size is though).

Hikikomori 5y ago

An ISP has two types of edge connections where ddos traffic comes into their network. Private peering where they are directly connected to their neighbor in one or more locations, typically neither party pays for traffic, can be 1 to 100G interfaces and multiple of them. Second type is transit, smaller or mid sized buys it from larger ISPs and pay for traffic. Everything that does not go directly to a peer mentioned before goes over this connection. Can also be very different in size. You can just add upp all edge Links and know their theoretical limit, but most of the ddos traffic likely comes from Asia and it's unlikely that they peer directly with those ISPs. So most of the traffic will go over transit, but the larger ISP likely has an automated system that handles volumetric attacks. So if there's any impact it will be on transit mostly affecting traffic to/from outside of Europe. 250 might be a lot for this ISP, but they might also have a lot more transit headroom, or it light be handled by their transit provider. Who knows.

ineedasername 5y ago
As an outsider to network topology that's fascinating to understand. Thanks for explaining. If you don't mind going further, how does an ISP (or anti-DDoS services) mitigate something like this without incurring even more overhead to examine the traffic & not route it onwards? Just a flat block against the sources, or something more sophisticated?
- Hikikomori 5y ago
  Haven't done much traditional networking for a few years so my knowledge is not that current. They use bgp flowspec to send out filter lists using the bgp protocol to routers, essentially a long list of ACL rules with srcip dstip dstport. These routers can filter any amount of traffic it gets in hardware, so as long as your upstream (edge transit or peering links) are not full you won't even notice the ddos. But you need something to analyse traffic that can understand which traffic is ddos and what is normal. We used devices from Arbor which are basically regular x86 servers. These devices also received netflow so whenever a ddos target received a lot of weird traffic (lots of DNS for example) it was configured to redirect all traffic for that IP to itself where it could gain a better understanding than what you can from netflow and either filtered it locally or sent out flowspec updates to filter it on routers. You could also enable this manually for non-volumetric ddos and do some filtering, but it's hard to do if it's encrypted and if they are abusing some http call that is just expensive to handle for the servers.

bstpierre 5y ago

I don’t know anything about this particular isp or attack, but if that 250 gbps is aimed at specific parts of their infrastructure that are not designed for this much load, it’s likely to saturate individual links, servers, and/or network gear. Their announcement said “all of our services”, so this could be anything from their authoritative/recursive dns to monitoring or billing servers to specific routing nodes. So even if they had nice fat 100G pipes coming in, and those pipes weren’t saturated, a ddos of this size with a mix of vectors can be debilitating on enough discrete pieces of infrastructure that the isp is effectively “down” for many end users. (Again I am only speaking generally with guesses about what might be going on at this isp. I’m also not a network engineer but I do write code for network gear that does anti ddos stuff.)

perlgeek 5y ago

ISPs have a very decentralized network, that is, they have routers in a lot of places. If you sum up the available traffic capacity, you get a number that's far bigger than 250 Gbit/s. But that's not the right comparison to make. Much of that DDoS traffic goes through multiple of the ISP's routers, and if you're doing just a numbers comparison, you'd have to multiply the attack rate by the (average) number of hops. But things are even more complicated in reality: many smaller links will have only 10Gbit/s or 20Gbit/s or 40Gbit/s capacity (or maybe even 1G), and saturating them not only causes increased latency, but also routing the traffic around causes more overall load on the network. Then there are management systems behind firewalls, and if you DDoS them, it's typically the firewalls that give out first. Finally, customers don't get the full throughput internally available to an ISP; if you DDoS a customer's IP, their link will be saturated, even if the ISP itself has plenty to spare. (Not a network engineer myself, but working closely with some).

Groxx 5y ago
Wouldn't that all be true of normal web traffic too, presumably already capable of rates beyond 250Gbps? Though they may not have that much free headroom on top of normal traffic.
oars 5y ago
So in the case where the attackers are aiming their traffic at one of the ISP's routers to saturate a link, how do they get the IP address/domain name of this router?
- perlgeek 5y ago
  Traceroutes. Peering information is also often public. More sinister options like insider information or leaked/exfiltrated network plans are also an option, but usually you get pretty far with standard network diagnostic tools.

deleted 5y ago

[deleted]

benp84 5y ago

For comparison, the entire internet averages on the order of 50 Tbps*, so this is about 0.5% of that. Residential internet might have a max rate of 1Gbps in the best conditions but if you use even 1% of that consistently you'll get overage fees. Steady bandwidth numbers are typically much lower than max bandwidth numbers. * Based on 122,000 PB/month in 2017 from https://en.wikipedia.org/wiki/Internet_traffic https://en.wikipedia.org/wiki/Internet_traffic, if my calcs are right

Gigachad 5y ago

It's not just 250Gbps, its 250 on top of normal usage. Most ISPs run a fairly efficient setup where they are only paying for a little bit over the average usage.