5 ms·

I used to run a hosting company similar to Linode back in the day, and DDOS's were the most annoying thing ever. The main reason DDOS attacks exist is poor sec

I used to run a hosting company similar to Linode back in the day, and DDOS's were the most annoying thing ever. The main reason DDOS attacks exist is poor security and lack of cooperation between ISPs. Lack of adequate security on desktops (usually Windows) makes it possible to build large bot networks. Lack of cooperation between ISPs makes it very hard to track down the source of the DOS. Very often the DDOS isn't as distributed as it may seem - it can just be a couple of machines on a very well connected network (e.g. a university). But getting a hold of someone in the middle to filter that traffic can take a very long time or be outright impossible. First-responder network engineers (typically referred to as "security") are overworked and underqualified, and the people who really know their stuff typically can't be bothered with silly DOS attacks. We've also observed that (D)DOS's happen because of content. Anything political, religious, or whatever other shade of the many things someone out there disapproves of is a potential target for a DOS. Contrary to what you may read in the press, extortion is only a small minority of all DOS attacks out there. We've actually told customers to go away because their content was too DOS-prone. And because these things usually happen across countries, even though they are very real crimes that cause serious damage and cost money, they are hardly ever prosecuted. As the target of a DOS all you want is for it to stop, nobody ever bothers reporting it to the authorities afterwards (because how would you even know who the "authority" is).

devicenull 11y ago
> The main reason DDOS attacks exist is poor security and lack of cooperation between ISPs. Lack of adequate security on desktops (usually Windows) makes it possible to build large bot networks. These days it's the hundreds of thousands of misconfigured NTP servers, recursive DNS servers, and various other protocols being abused for reflection attacks. Granted, it still requires that the attacker have the ability to spoof packets, but preventing that requires even more time investment and has very little benefit to the ISP.
- jsmthrowaway 11y ago
  BCP 38 is like herd immunity with immunization, and much like anti-vaccine folks, networks that don't follow it are knowingly choosing to infect people for any number of reasons. Despite your claim, it is extremely easy to implement and has been a known best practice, with accompanying educational Web sites devoted to the topic[0], for many years. There are nearly zero reasons for your AS to transmit forged packets, and if your configuration allows it, you are knowingly making the Internet a worse place with your laziness and transferring your laziness to other people like me that carry pagers. This isn't a surprise to any network administrator unless they've spent their entire career not reading RFCs. I'm of the opinion that networks that allow customers to emit forged source addresses should be depeered until they take the literal hour to fix it. "But we have to update equipment when we get new blocks!" Boo hoo. Automate it or get off the Internet so I can stop spending my life dealing with your customer's amplified traffic. If you run a non-transit/eyeball AS, you are in the absolute best position to stop these types of attacks from ever happening. The rest of the Internet, particularly your transit peers, can't really clean up after you on this one. Do us all a favor. [0]: http://bcp38.info http://bcp38.info
  - devicenull 11y ago
    I'd argue that the end networks really aren't the best place to be implementing this. If the big transit providers (Level3, NTT, etc) started enforcing it, it would significantly reduce the effectiveness of spoofed traffic pretty much overnight.
    - jsmthrowaway 11y ago
      It might seem that way, but they can't scale that, as you might know. It's tougher for DFZ transit to do it because they must know, and programmatically configure, all downstream space to be whitelisted. Now you have a similar conversation to BGP filtering when downstream networks change space, and that's huge administrative overhead (which is why announcements are just trusted without filtering at the higher levels, to avoid this overhead for the larger AS). Your end network is a better place because you know your assigned space better, as well as how you number it; you might be holding half an /18 and not assigning it, whereas your peer would whitelist it all, for example. If the technicals of the Internet were programmatically available in a sane way (PeeringDB doesn't count here, since it just automates manual work), the Tier 1s could potentially automate against their downstream AS' space and enable your (mostly correct) point. However, we pretty much fly blind in this respect and rely on emails and ticketing and decentralized systems to manage the control plane of the Internet. Which honestly continues to shock me, even though it makes sense since the Internet is designed as "decentralized" despite being anything but in usage. Edit: While in the car, I realized that Paul Vixie's paper on this discusses the CPE source-filtering angle in great detail, which might illustrate my opinion a little better for you than I ever could: https://queue.acm.org/detail.cfm?id=2578510 https://queue.acm.org/detail.cfm?id=2578510
scurvy 11y ago
All of this +1000000 We've found that the more distributed the attack, the easier it is to defend. If it's truly distributed in nature, you can help spread the attack surface through things like anycasting if you have you multiple datacenters. Also, the more distributed it is, the more likely it is to be spread out among your different transit links. The less distributed, the more likely it will hit a single transit link and take that out. You might have 10 different transit providers with 10gig links. 100 gbps of transit is nothing to scoff at for small companies. You can have an 11 gbps DDoS that makes you unreachable for large numbers of people if that 11 gbps lands on a single link (due to AS path length, MED, preference, etc). People at your company are asking you "WTF network dude I thought we had 100gigs?" You can then try to mess with padding route advertisements, but that just messes more with legit traffic than attack traffic. Same thing with aggregated ethernet links (LACP/802.3ad). You might be 10x10gigs with a single provider, giving you a "100gbps pipe" but if it comes from a few sources all going after multiple destinations, a single link in that bundle could easily be overwhelmed. Then you start playing with the hashing algorithm, but it never really solves the problem. The only solution right now is to get a bigger boat and get 100gbps links. It's very expensive, not every transit provider offers that to every customer, lol if you run Cisco, and even then it's honestly not big enough. Overrunning 100gbps interfaces with DoS traffic isn't very hard to do if it comes from a few sources. One must realize that not every country has the network diversity that the US does. In a lot of places, there are only a handful of AS networks that actually provide transit out of the country. This means that an entire country's network traffic can flow over a few (or often times one) providers. This turns into a giant laser that blows out single provider or single LACP member links. China is a great example of this. There are really only 2 AS that provide egress from China. The third is reserved for high ranking party officials and ultra, megarich people who pay the bribes. Korea (and all the broadband that HN people drool over) has a few more transit networks, but not many. Korea is a potent DDoS cannon. As far as content goes it's by far mostly political and religious (> 70%). The remaining is the usual junk e-commerce, gambling, whistleblower sites, etc. The lack of communication between transit providers and even between their customers is astoundingly bad. IMHO, there are good transit providers that will help you (NTT, Telia, XO) but most will tell you to go pound sand or not do anything until it starts disrupting their own network (Level3, Cogent, GTT, VZ, AT&T).
blisterpeanuts 11y ago
>> We've actually told customers to go away because their content was too DOS-prone. Isn't that a good way to earn yourself some nasty one-star reviews? "They shut us down because of our political/religious content." That doesn't sound like a way to attract and keep loyal customers through stellar reviews and word-of-mouth advertising. I hope there's a better way for ISPs to solve the problem than just to get rid of the customers who are the targets.

devicenull 11y ago

> The main reason DDOS attacks exist is poor security and lack of cooperation between ISPs. Lack of adequate security on desktops (usually Windows) makes it possible to build large bot networks. These days it's the hundreds of thousands of misconfigured NTP servers, recursive DNS servers, and various other protocols being abused for reflection attacks. Granted, it still requires that the attacker have the ability to spoof packets, but preventing that requires even more time investment and has very little benefit to the ISP.

jsmthrowaway 11y ago
BCP 38 is like herd immunity with immunization, and much like anti-vaccine folks, networks that don't follow it are knowingly choosing to infect people for any number of reasons. Despite your claim, it is extremely easy to implement and has been a known best practice, with accompanying educational Web sites devoted to the topic[0], for many years. There are nearly zero reasons for your AS to transmit forged packets, and if your configuration allows it, you are knowingly making the Internet a worse place with your laziness and transferring your laziness to other people like me that carry pagers. This isn't a surprise to any network administrator unless they've spent their entire career not reading RFCs. I'm of the opinion that networks that allow customers to emit forged source addresses should be depeered until they take the literal hour to fix it. "But we have to update equipment when we get new blocks!" Boo hoo. Automate it or get off the Internet so I can stop spending my life dealing with your customer's amplified traffic. If you run a non-transit/eyeball AS, you are in the absolute best position to stop these types of attacks from ever happening. The rest of the Internet, particularly your transit peers, can't really clean up after you on this one. Do us all a favor. [0]: http://bcp38.info http://bcp38.info
- devicenull 11y ago
  I'd argue that the end networks really aren't the best place to be implementing this. If the big transit providers (Level3, NTT, etc) started enforcing it, it would significantly reduce the effectiveness of spoofed traffic pretty much overnight.
  - jsmthrowaway 11y ago
    It might seem that way, but they can't scale that, as you might know. It's tougher for DFZ transit to do it because they must know, and programmatically configure, all downstream space to be whitelisted. Now you have a similar conversation to BGP filtering when downstream networks change space, and that's huge administrative overhead (which is why announcements are just trusted without filtering at the higher levels, to avoid this overhead for the larger AS). Your end network is a better place because you know your assigned space better, as well as how you number it; you might be holding half an /18 and not assigning it, whereas your peer would whitelist it all, for example. If the technicals of the Internet were programmatically available in a sane way (PeeringDB doesn't count here, since it just automates manual work), the Tier 1s could potentially automate against their downstream AS' space and enable your (mostly correct) point. However, we pretty much fly blind in this respect and rely on emails and ticketing and decentralized systems to manage the control plane of the Internet. Which honestly continues to shock me, even though it makes sense since the Internet is designed as "decentralized" despite being anything but in usage. Edit: While in the car, I realized that Paul Vixie's paper on this discusses the CPE source-filtering angle in great detail, which might illustrate my opinion a little better for you than I ever could: https://queue.acm.org/detail.cfm?id=2578510 https://queue.acm.org/detail.cfm?id=2578510

scurvy 11y ago

All of this +1000000 We've found that the more distributed the attack, the easier it is to defend. If it's truly distributed in nature, you can help spread the attack surface through things like anycasting if you have you multiple datacenters. Also, the more distributed it is, the more likely it is to be spread out among your different transit links. The less distributed, the more likely it will hit a single transit link and take that out. You might have 10 different transit providers with 10gig links. 100 gbps of transit is nothing to scoff at for small companies. You can have an 11 gbps DDoS that makes you unreachable for large numbers of people if that 11 gbps lands on a single link (due to AS path length, MED, preference, etc). People at your company are asking you "WTF network dude I thought we had 100gigs?" You can then try to mess with padding route advertisements, but that just messes more with legit traffic than attack traffic. Same thing with aggregated ethernet links (LACP/802.3ad). You might be 10x10gigs with a single provider, giving you a "100gbps pipe" but if it comes from a few sources all going after multiple destinations, a single link in that bundle could easily be overwhelmed. Then you start playing with the hashing algorithm, but it never really solves the problem. The only solution right now is to get a bigger boat and get 100gbps links. It's very expensive, not every transit provider offers that to every customer, lol if you run Cisco, and even then it's honestly not big enough. Overrunning 100gbps interfaces with DoS traffic isn't very hard to do if it comes from a few sources. One must realize that not every country has the network diversity that the US does. In a lot of places, there are only a handful of AS networks that actually provide transit out of the country. This means that an entire country's network traffic can flow over a few (or often times one) providers. This turns into a giant laser that blows out single provider or single LACP member links. China is a great example of this. There are really only 2 AS that provide egress from China. The third is reserved for high ranking party officials and ultra, megarich people who pay the bribes. Korea (and all the broadband that HN people drool over) has a few more transit networks, but not many. Korea is a potent DDoS cannon. As far as content goes it's by far mostly political and religious (> 70%). The remaining is the usual junk e-commerce, gambling, whistleblower sites, etc. The lack of communication between transit providers and even between their customers is astoundingly bad. IMHO, there are good transit providers that will help you (NTT, Telia, XO) but most will tell you to go pound sand or not do anything until it starts disrupting their own network (Level3, Cogent, GTT, VZ, AT&T).

blisterpeanuts 11y ago

>> We've actually told customers to go away because their content was too DOS-prone. Isn't that a good way to earn yourself some nasty one-star reviews? "They shut us down because of our political/religious content." That doesn't sound like a way to attract and keep loyal customers through stellar reviews and word-of-mouth advertising. I hope there's a better way for ISPs to solve the problem than just to get rid of the customers who are the targets.