3 ms·

I'm interning at a major managed DNS provider and this actually happens more often than you think. Most of the time it ends by working with telcos like level3,

I'm interning at a major managed DNS provider and this actually happens more often than you think. Most of the time it ends by working with telcos like level3, cogent etc, blackholing routes, sacrificing a certain geographical location in favor of keeping others alive and basically just waiting it out. It's interesting how because of the nature of the internet, DDoSes are hard to deal with.

drivingmenuts 16y ago
Too bad we can't just permanently sacrifice China.
- dnsworks 16y ago
  There's an anti-DDOS service out there, Progent or something. Part of their solution adds all of APNIC's ip addresses into your router's bogon list when you're under attack. It turns out for most DDOS's if you turn off china, the attack just ends.
  - tptacek 16y ago
    I think you're thinking of Prolexic, and my recommendation is that you take any of their claims with a heaping helping of salt. Things may have changed in the 5 years since I've totally forgotten everything I knew about DoS mitigation, but it wasn't the case then that you could simply black hole China to evade them. A more common mechanism used by real networks to mitigate these attacks in production is to pinpoint the target of the attack and offramp its traffic in the ISP core to a "regional scrubbing center" where advanced filtering tools (for instance, packet filters that can handle ACLs with high tens of thousands of terms) can try to sort through the crap.
    - jacquesm 16y ago
      At my colo that is exactly the sort of strategy they use, they have bought a bunch of routers that use FPGA based filters for that. The one scenario I can think of where that might be a problem is if they'd start flooding all the known hosts in a network for the specific purpose of overwhelming the routers. And even those hardware based filtering tools have upper limits.
      - dnsworks 16y ago
        There's really no need to do that. A large enough BotNet can take out almost any host doing nothing more than connecting and doing a GET / HTTP/1.1. Few websites can handle a sudden surge in traffic from 100,000 bots.
        jacquesm 16y ago
        The idea behind an ACL for such an attack is that the same hosts will be used over and over again, taken from a large pool of zombies. So, let's take your example of 100,000 bots, an ACL in the upstream router (as seen from the host) could be used to checked against to identify those packets from the zombies and discard them. After all, a single attack of all 100,000 bots at once will just bring the host to its knees for the time-out of the connections and then it will bounce back up again. So, to increase the effectiveness of the attack they reconnect after every lost connection asking for another resource. If they're smart they'll vary agent strings and other characteristics to make it hard to narrow down who is and who is not legit. Initially you don't have much to go on during such an attack and packet filtering is a reasonably expensive operation when you want to do it for a large number of hosts. So the strategy is to route all the traffic destined for that particular host through a router that has ACLs that are large enough to hold the total IP list for the botnet that is attacking the host, as these IPs become identified. You don't want to route all your traffic through there because then you'd have to do the relatively expensive filtering on all of the packets, even those not destined for that particular host. Now if an attacker were targeting the hosting facility they could thwart this strategy by sending requests to a larger number of hosts in the network in order to make life much harder for the crew fighting the attack. After all, you can't partition the problem anymore in to a portion that is targeted to the host and 'normal' traffic, effectively all the traffic could be bot traffic or it could be normal traffic, for all the receiving hosts. To be able to partition the problem into a smaller one where you can let say 90% or more of the traffic through unfiltered and only concentrate on the remaining 10% would make solving it a bit easier. On the other hand if the attackers are silly enough to re-use the same bots to attack different hosts they've actually given you a clue as to which IPs are bots. I hope that makes sense :)
- vkdelta 16y ago
  If we want, we can :)
- mrtron 16y ago
  It is easy to say that now - would you have said that about US cable modem accounts 10 years ago?
tptacek 16y ago
From 2001-2003 I led development on Arbor Peakflow DoS, which (when I left, at least) was far and away the most popular tool tier-1 ISPs used to manage these kinds of attacks, and my the experience I can relate is: * These attacks happen with shocking frequency (the major incentive ISPs have to mitigate them is not angry customers, but rather the drag it creates on their third-tier engineering staff to hunt down DDoS sources and craft ACLs for them) * There is actually not a whole hell of a lot you can do about them, even if you light up a whole tier 1 core network with mitigation tools. At the end of the day, a well-crafted DDoS attack looks pretty much identical to normal traffic; if you can black-hole China to defeat an attack, it wasn't very cluefully done.
- makmanalp 16y ago
  Regarding your statement that it wasn't very cluefully done, isn't that sort of a hit and miss area? You can't really control much where your malware spreads (and thus where your bots are). As a sidenote, I've been reading your posts for a while now and it seems you always have something insightful to say - cheers!
  - peterwwillis 16y ago
    it's not difficult to control where malware spreads. directly attacking specific ranges is one easy method, as is coding the limitation into your bot or only sending commands to the ones in a certain range.

drivingmenuts 16y ago

Too bad we can't just permanently sacrifice China.

dnsworks 16y ago
There's an anti-DDOS service out there, Progent or something. Part of their solution adds all of APNIC's ip addresses into your router's bogon list when you're under attack. It turns out for most DDOS's if you turn off china, the attack just ends.
- tptacek 16y ago
  I think you're thinking of Prolexic, and my recommendation is that you take any of their claims with a heaping helping of salt. Things may have changed in the 5 years since I've totally forgotten everything I knew about DoS mitigation, but it wasn't the case then that you could simply black hole China to evade them. A more common mechanism used by real networks to mitigate these attacks in production is to pinpoint the target of the attack and offramp its traffic in the ISP core to a "regional scrubbing center" where advanced filtering tools (for instance, packet filters that can handle ACLs with high tens of thousands of terms) can try to sort through the crap.
  - jacquesm 16y ago
    At my colo that is exactly the sort of strategy they use, they have bought a bunch of routers that use FPGA based filters for that. The one scenario I can think of where that might be a problem is if they'd start flooding all the known hosts in a network for the specific purpose of overwhelming the routers. And even those hardware based filtering tools have upper limits.
    - dnsworks 16y ago
      There's really no need to do that. A large enough BotNet can take out almost any host doing nothing more than connecting and doing a GET / HTTP/1.1. Few websites can handle a sudden surge in traffic from 100,000 bots.
      - jacquesm 16y ago
        The idea behind an ACL for such an attack is that the same hosts will be used over and over again, taken from a large pool of zombies. So, let's take your example of 100,000 bots, an ACL in the upstream router (as seen from the host) could be used to checked against to identify those packets from the zombies and discard them. After all, a single attack of all 100,000 bots at once will just bring the host to its knees for the time-out of the connections and then it will bounce back up again. So, to increase the effectiveness of the attack they reconnect after every lost connection asking for another resource. If they're smart they'll vary agent strings and other characteristics to make it hard to narrow down who is and who is not legit. Initially you don't have much to go on during such an attack and packet filtering is a reasonably expensive operation when you want to do it for a large number of hosts. So the strategy is to route all the traffic destined for that particular host through a router that has ACLs that are large enough to hold the total IP list for the botnet that is attacking the host, as these IPs become identified. You don't want to route all your traffic through there because then you'd have to do the relatively expensive filtering on all of the packets, even those not destined for that particular host. Now if an attacker were targeting the hosting facility they could thwart this strategy by sending requests to a larger number of hosts in the network in order to make life much harder for the crew fighting the attack. After all, you can't partition the problem anymore in to a portion that is targeted to the host and 'normal' traffic, effectively all the traffic could be bot traffic or it could be normal traffic, for all the receiving hosts. To be able to partition the problem into a smaller one where you can let say 90% or more of the traffic through unfiltered and only concentrate on the remaining 10% would make solving it a bit easier. On the other hand if the attackers are silly enough to re-use the same bots to attack different hosts they've actually given you a clue as to which IPs are bots. I hope that makes sense :)
vkdelta 16y ago
If we want, we can :)
mrtron 16y ago
It is easy to say that now - would you have said that about US cable modem accounts 10 years ago?

tptacek 16y ago

From 2001-2003 I led development on Arbor Peakflow DoS, which (when I left, at least) was far and away the most popular tool tier-1 ISPs used to manage these kinds of attacks, and my the experience I can relate is: * These attacks happen with shocking frequency (the major incentive ISPs have to mitigate them is not angry customers, but rather the drag it creates on their third-tier engineering staff to hunt down DDoS sources and craft ACLs for them) * There is actually not a whole hell of a lot you can do about them, even if you light up a whole tier 1 core network with mitigation tools. At the end of the day, a well-crafted DDoS attack looks pretty much identical to normal traffic; if you can black-hole China to defeat an attack, it wasn't very cluefully done.

makmanalp 16y ago
Regarding your statement that it wasn't very cluefully done, isn't that sort of a hit and miss area? You can't really control much where your malware spreads (and thus where your bots are). As a sidenote, I've been reading your posts for a while now and it seems you always have something insightful to say - cheers!
- peterwwillis 16y ago
  it's not difficult to control where malware spreads. directly attacking specific ranges is one easy method, as is coding the limitation into your bot or only sending commands to the ones in a certain range.