5 ms·

Can you describe Cogent's reaction to your request to blackhole the route? We had a very small incident with XO recently (perhaps 500Mbps reflected DNS flood),

by cft 14y ago

Can you describe Cogent's reaction to your request to blackhole the route? We had a very small incident with XO recently (perhaps 500Mbps reflected DNS flood), and it took them 24 hrs to get back to me to blackhole the target! We just packet filtered in our edge switch. Was Cogent any faster?

dsl 14y ago
If you run a network of any size you should be speaking BGP with your upstream, even if its using a private AS. You can then announce a prefix to them specially tagged with a "blackhole community" that drops traffic at the edge of their network. The exact details vary by network, but here is how Hurricane does it http://www.he.net/adm/blackhole.html http://www.he.net/adm/blackhole.html
- lsc 14y ago
  Yup. The details are slightly different for Cogent (I think we had it setup as a separate BGP session rather than as a community tag like he.net does it, but that was because my customer requested it that way.) But yeah, you give them /32s to null, and they drop those /32s at the network edge. It stops the attack, well, almost immediately, but the problem is that it kills the target site completely. (well, often people have web frontends to this, which, well, work poorly when your pipe is completely full, and for that matter, just getting the bgp data to your peer can take a few tries. but yeah, it's still pretty quick and effective, compared to calling someone to whine.) What we really need is to get everyone to implement bcp38 anti-spoofing rules. If everyone did that, we'd be able to block the sources of the problem, rather than the destination. But, well, that's unlikely to happen, so for now, you just, ah, finish the job.
  - eps 14y ago
    This is what I don't understand about DoS mitigation. So i'd null route the target, but then I have to move the service to another IP. I will also announce this IP to the customers, but that would simply redirect the DoS to the new address and take it down too... I mean, there seems to be no way to deal with DoS without throwing a baby (customers) with the water (attackers) or am I missing something?
    - codesuela 14y ago
      Well services like Cloudflare or Blockdos try to mitigate this problem by absorbing the bad traffic and filtering it so that it is spread out across nodes and then dropped by firewalls with custom rules.
      - eps 14y ago
        I don't follow. How do they know which traffic is "bad" and which is legit?
        codesuela 14y ago
        With ordinary DDoS attacks an effective method (which Cloudflare uses) is to prompt you with an captcha before letting you pass or dropping your connection when you fail often enough. As far as I understand it was not that hard to block this attack because it follows traffic patterns (DNS responses from open resolvers). The actual problem was that the attack was so massive that it clogged the pipes before it could reach a router (belonging to Cloudflare) that would've been able to drop the packages. Disclaimer: I am no network engineer so don't rely on my reply being factually 100% correct.
        eps 14y ago
        Captcha. Right.
    - lsc 14y ago
      meh, it's hard. First prerequisite to effective DDoS mitigation? you need to have enough capacity (in terms of upstream transit/peering ports) to absorb the whole attack. If not? it's over, you blackhole the target at your upstream's routers (and thus throw out the baby) and the attack ends, or you don't, and your service is dead for all customers. That's why I don't advertise any sort of "DoS protection" - I know that attacks that are bigger and badder than my network are fairly common. This is also why I'm not going to take any promises of DoS mitigation from anyone who doesn't have a terrifyingly huge network seriously. Now, once you have enough upstream port capacity to soak the attack, you then filter the good traffic from the bad. This is a whole 'nother can of "very hard" but it's easy compared to getting the capacity in the first place. Note, this filtering becomes /way/ easier if you have some idea of the sort of traffic you are expecting, but it's still difficult. There are "clean pipes" services that claim to do this for you, with varying degrees of credibility. The thing is, CloudFair is one of the smaller companies to offer this. I was looking at the offering from level3, (in my mind, considering their network capacity, probably the most credible provider of such a service. Also, their service claimed to work with all traffic, not just http and the like, so it would work for me.) but it sounded like the price was somewhere along the lines of "give us 25% of your revenue, and we'll give you half the cleaned capacity you need." I mean, even the regular level 3 bandwidth is between one and two orders of magnitude more expensive than the cogent/he.net mix that is common in my industry, so uh, yeah. I didn't spend the requisite six months with the salesman to get the real price, but I think "more than I can afford" is a likely guess. I mean, the idea here, usually, is that the network providing this service is large enough that it has a whole bunch of peering connections and can filter the incoming traffic fairly close to the source. Even if you've got hundreds of gigabits of capacity at one location, if it's all at that one location, it's very likely that something else is going to gum up the works between, say Austria and you. If you've got a giant, global network, though, your traffic from Austria goes to your POP in Austria, where you can filter the "bad" traffic (assuming you figured out how to do that.) And really, you don't have to filter /all/ the attack traffic, just enough that the target isn't completely overwhelmed. Like spam-filtering or anything else, nothing is 100%. And that's how most of the low-end hosting world feels about it. The upshot is that we throw out the baby; the small customer who gets hit repeatedly by large DDoS attacks, generally speaking, has to change to a different provider, 'cause they get kicked off. I mean, if you are paying someone $20/month, and your enemies take that service down hard? yeah, after the problem is fixed, you are very likely to need to find a new provider.

by cft 14y ago

dsl 14y ago

If you run a network of any size you should be speaking BGP with your upstream, even if its using a private AS. You can then announce a prefix to them specially tagged with a "blackhole community" that drops traffic at the edge of their network. The exact details vary by network, but here is how Hurricane does it http://www.he.net/adm/blackhole.html http://www.he.net/adm/blackhole.html

lsc 14y ago
Yup. The details are slightly different for Cogent (I think we had it setup as a separate BGP session rather than as a community tag like he.net does it, but that was because my customer requested it that way.) But yeah, you give them /32s to null, and they drop those /32s at the network edge. It stops the attack, well, almost immediately, but the problem is that it kills the target site completely. (well, often people have web frontends to this, which, well, work poorly when your pipe is completely full, and for that matter, just getting the bgp data to your peer can take a few tries. but yeah, it's still pretty quick and effective, compared to calling someone to whine.) What we really need is to get everyone to implement bcp38 anti-spoofing rules. If everyone did that, we'd be able to block the sources of the problem, rather than the destination. But, well, that's unlikely to happen, so for now, you just, ah, finish the job.
- eps 14y ago
  This is what I don't understand about DoS mitigation. So i'd null route the target, but then I have to move the service to another IP. I will also announce this IP to the customers, but that would simply redirect the DoS to the new address and take it down too... I mean, there seems to be no way to deal with DoS without throwing a baby (customers) with the water (attackers) or am I missing something?
  - codesuela 14y ago
    Well services like Cloudflare or Blockdos try to mitigate this problem by absorbing the bad traffic and filtering it so that it is spread out across nodes and then dropped by firewalls with custom rules.
    - eps 14y ago
      I don't follow. How do they know which traffic is "bad" and which is legit?
      - codesuela 14y ago
        With ordinary DDoS attacks an effective method (which Cloudflare uses) is to prompt you with an captcha before letting you pass or dropping your connection when you fail often enough. As far as I understand it was not that hard to block this attack because it follows traffic patterns (DNS responses from open resolvers). The actual problem was that the attack was so massive that it clogged the pipes before it could reach a router (belonging to Cloudflare) that would've been able to drop the packages. Disclaimer: I am no network engineer so don't rely on my reply being factually 100% correct.
        eps 14y ago
        Captcha. Right.
  - lsc 14y ago
    meh, it's hard. First prerequisite to effective DDoS mitigation? you need to have enough capacity (in terms of upstream transit/peering ports) to absorb the whole attack. If not? it's over, you blackhole the target at your upstream's routers (and thus throw out the baby) and the attack ends, or you don't, and your service is dead for all customers. That's why I don't advertise any sort of "DoS protection" - I know that attacks that are bigger and badder than my network are fairly common. This is also why I'm not going to take any promises of DoS mitigation from anyone who doesn't have a terrifyingly huge network seriously. Now, once you have enough upstream port capacity to soak the attack, you then filter the good traffic from the bad. This is a whole 'nother can of "very hard" but it's easy compared to getting the capacity in the first place. Note, this filtering becomes /way/ easier if you have some idea of the sort of traffic you are expecting, but it's still difficult. There are "clean pipes" services that claim to do this for you, with varying degrees of credibility. The thing is, CloudFair is one of the smaller companies to offer this. I was looking at the offering from level3, (in my mind, considering their network capacity, probably the most credible provider of such a service. Also, their service claimed to work with all traffic, not just http and the like, so it would work for me.) but it sounded like the price was somewhere along the lines of "give us 25% of your revenue, and we'll give you half the cleaned capacity you need." I mean, even the regular level 3 bandwidth is between one and two orders of magnitude more expensive than the cogent/he.net mix that is common in my industry, so uh, yeah. I didn't spend the requisite six months with the salesman to get the real price, but I think "more than I can afford" is a likely guess. I mean, the idea here, usually, is that the network providing this service is large enough that it has a whole bunch of peering connections and can filter the incoming traffic fairly close to the source. Even if you've got hundreds of gigabits of capacity at one location, if it's all at that one location, it's very likely that something else is going to gum up the works between, say Austria and you. If you've got a giant, global network, though, your traffic from Austria goes to your POP in Austria, where you can filter the "bad" traffic (assuming you figured out how to do that.) And really, you don't have to filter /all/ the attack traffic, just enough that the target isn't completely overwhelmed. Like spam-filtering or anything else, nothing is 100%. And that's how most of the low-end hosting world feels about it. The upshot is that we throw out the baby; the small customer who gets hit repeatedly by large DDoS attacks, generally speaking, has to change to a different provider, 'cause they get kicked off. I mean, if you are paying someone $20/month, and your enemies take that service down hard? yeah, after the problem is fixed, you are very likely to need to find a new provider.