17 ms·

How to Launch a 65Gbps DDoS, and How to Stop One

• glazemaster 14y ago
  We tried to use Cloudflare when they teamed with Dreamhost a few months ago. We had more downtime than uptime.... Though this is super-relevant because during the struggle with Cloudflare, we released an article about LOIC and how easy it is to reveal the locations and identities of individuals involved in a DDoS attack using LOIC. http://www.thepowerbase.com/2012/03/low-orbit-ion-cannon-exposed/ http://www.thepowerbase.com/2012/03/low-orbit-ion-cannon-exp...
  • maskedinvader 14y ago
    that was an interesting read, thanks
  • eastdakota 14y ago
    Sorry for the trouble you had. Did you submit a support ticket to CloudFlare? Sounds like something was blocking requests from our network. LOIC and a number of the more public DDoS tools make the attackers' identities relatively easy to track. The big attack we saw last Saturday is much more difficult to trace both because it is originated with a UDP request (the headers of which can be forged) and because it is reflected off open resolvers (essentially laundering the identity of the attack's source).
    • glazemaster 14y ago
      Honestly we were confused by the whole partnership. We kept submitting tickets to Dreamhost which they did a poor job of fulfilling. We liked the idea of Cloudflare and we were mostly willing to stick it out. Not only that, but we had several great ideas about how we might leverage your offering against some other things that we wanted to accomplish. The real reason that we stopped using Cloudflare is that when you were unable to serve the cached page, it throws up a Cloudflare branded page. We thought that those sorts of error pages would diminish our image as an open source publication because it seems to suggest that we can't rely on our own tools and abilities. If we were able to display a "Powerbase" branded 404 type page, we would have been more satisfied.
      • eastdakota 14y ago
        If you're a paying customer, you can fully customize the error pages. Probably easiest if you just sign up directly through us if you want to do something like that. We can get you setup: http://www.cloudflare.com/sign-up http://www.cloudflare.com/sign-up
        • glazemaster 14y ago
          We will consider it. Thanks for letting me know!
    • buro9 14y ago
      I also had problems when I tried CloudFlare. Different problems, but it cost me a few days contacting users and apologising before I U-turned and switched the nameservers back. I unfortunately use PayPal and their Instant Notification thing, basically a callback to a web page with a POST about the transaction that just happened. Upon receiving the POST I can then do things like notify the customer, dispatch goods, award virtual goods, etc. The problem I had was that after putting my website (in the London Linode datacenter) behind CloudFlare, PayPal started randomly failing to reach the callback page. PayPal, being PayPal, failed silently for a few days before finally sending an email to say that they couldn't notify me of transactions. I figured it out just before that though, because users were getting the CloudFlare "site offline" message. The pages on my site are 100% dynamic, so nothing was cached by the feature that keeps a site online. My biggest problem with CloudFlare was visibility for debugging: I had no visibility. If it wasn't for my users letting me know and PayPal emailing me confirming what I thought... I wouldn't have known. Even then it took too long to find out, over a week from when it started to fail silently. According to Linode there was no downtime in that period, and according to my server logs load was never above average and there was no reason it should've been unable to be reached. Did I submit a ticket? I submitted some questions beforehand and got back answers that were very friendly but not technically detailed. That's also how I found the interface, I couldn't debug using CloudFlare, no way to answer the questions "What is happening? Why is it happening?". So no, when I figured it out I wasn't going to stay with CloudFlare as even if it was resolved I would still lack visibility for future problem-solving. In the end it was costing me goodwill with my users. I wanted things you didn't provide: How often did CloudFlare fail to contact my network? Can I see a chart of such failures over time? What were the failure error codes and times so that I can cross-reference them to my logs? Basically: I wanted transparency so I could have confidence in the service, and detail so that I can debug failures when they occur. And I was going to email this as it's really a "just to let you know". But you have no email in your HN profile, and looking through the support emails I had I see tenderapp.com and can't make a guess what your email address may be, and I pinged you on Twitter but no response and it's very late here... so posting it so you can see. If you add ways for developers to debug issues when using CloudFlare then I may well be tempted back in the future. The fundamental premise is a good one and I really wanted it to work (paid for the Enterprise level, had every intention of using it). But when failing silently costs real money and customer goodwill, I don't feel I had a choice but to U-turn very rapidly. As soon as I was off CloudFlare, PayPal Instant Payment Notification worked again and there hasn't been a single failure since.
  • mtgx 14y ago
    From my experience with both Dreamhost and Hostgator, Hostgator is a much better service overall, and more uptime, too.
    • meritt 14y ago
      Why would you arbitrarily mention Hostgator? Shilling? Yes, Dreamhost sucks. So does Hostgator. So does every other oversold host.
• superkvn 14y ago
  Interesting. DNS reflection is one I hadn't heard about before. Very interesting.
  • krakensden 14y ago
    DJB gave a presentation on it like... last week. Quick turnaround time on the part of the botnet herders.
    • jwegan 14y ago
      DNS reflection has been known (and used) for years. That is why cloudflare mentions there has been an ongoing effort to clean up open resolvers.
    • count 14y ago
      Was that the bit about amplification via DNSSEC?
      • ibotty 14y ago
        i'm pretty sure it was. but he has been telling the world about dnssec amplification for years now, so this is hardly news.
      • belorn 14y ago
        The amplification is by asking the misconfigured resolver about a DNSSEC zone. Basically, DNSSEC just mean you do not need to search the for a large zone to request. Given that large zones are not directly in shot supply, and that searching for them is (in the age of ipv4) rather easy, I wonder if DNSSEC actually have any affect on the issue what so ever.
      • smountcastle 14y ago
        DNSSEC-signed responses can be very large. Here's an example of turning a 31 byte request into a 3974 byte response: http://dnscurve.org/amplification.html http://dnscurve.org/amplification.html That's ~128x amplification -- with a 100Mbps connection, roughly 12.8Gbps of responses would be sent to the forged IP source.
• robotmay 14y ago
  I feel bad that I'm not currently paying for Cloudflare; I use it on a few sites but they don't have any traffic worth adding the extra fee for. However it's an excellent service and something I recommend often; hopefully I'll have something to make better use of it in the future :)
• rachelbythebay 14y ago
  Who are these irresponsible network operators that allow spoofed source addresses out of their network? The only way to make a reflection attack like this work is to make the responses go back to the victim. For that to happen, it has to look like they generated the request. Remember smurf? Spoof-ping a broadcast address for a multiplication effect. It's from 1997 or so. 15 years later and we're still living with that kind of problem. http://en.wikipedia.org/wiki/Smurf_attack http://en.wikipedia.org/wiki/Smurf_attack
  • udpheaders 14y ago
    Re-read the blog post. He's speculating the attack used DNS. (Though he has no proof.) In that case, with UDP, spoofed headers are allowed out. Connectionless. Cloudflare uses anycast DNS - machines in different geographically located data centers all sharing the same IP. If you want to try to make your site DOS-proof (and potentially faster), one way is to move the site to the network edge. Move the data closer to the user. Put a copy on a machine in the data center nearest the user. Do this in data centers around the world. ("CDN") Give all the machines the same IP. ("Anycast") Your users will be accessing a mirrored copy of your site at some regional data center, instead of actually sending requests that go out to the internet. Does Google do this? Akamai? Netflix? Next time you access a popular website ask yourself "Am I actually accessing the internet? Or am I just downloading a copy of something from a local data center?" A lot of these services are just marketing. In theory they sound great, but things may be different in practice. And that's why we frequently see comments that things did not work as expected. I did some CDN experiments downloading pages using Akamai where I accessed content on the "true IP address" (the master copy so to speak) versus the regional IP address they provide through stupid DNS tricks. Guess which one was faster? It all depends on caching: what is in the cache and what isn't. Same applies to DNS. A DNS caching server (resolver) is only faster than non-caching DNS server (authoritative) if it's primed with the records you're after. If they are not in the cache, it will not be faster. In fact, it will be slower because there are more steps to the process. These strategies are often based on 80/20, power law thinking. If you are not in the 20 percent of content being accessed 80 percent of the time, then you do not see the benefits. If no one in your region has requested a given page, and you're the first, it will be slower to wait for it to be cached at your regional data center than if you just grabbed it from the internet.
• zobzu 14y ago
  The TL;DR; version is "blabla 65GBps DDOS blabla" "we solve it by having 100's GBps networks" (and redirect whatever is legitimate to the client ofc) Okay. Maybe my expectations were set too high :) Leaves me to wonder what they can do if the traffic looks 100% legitimate.
  • elliottcarlson 14y ago
    "We know, for example, that we haven't sent any DNS inquiries out from our network. We can therefore safely filter the responses from DNS resolvers. We can therefore drop the response packets at our routers or, in some cases, even upstream at one of our bandwidth providers. The result is that these types of attacks are relatively easily mitigated." This seems to be a bit more specific and does indeed give insight on how to mitigate this type of attack.
• dfc 14y ago
  I am surprised that the article did not mention egress filtering alongside closing open resolvers. If more edge routers did proprer egress filtering these attacks would be harder to pull off.
• patdennis 14y ago
  Do they inform the target? While it's nice that they can stop an attack without the intended victim noticing, it's still probably a good idea to let them know.
  • hahla 14y ago
    From my short experience using cloudflare I believe they do. If a comprised computer tries to visit a site that has dns routing through cloudflare, they serve up a page that says their computer might be comprised and they have to enter a captcha to verify its human traffic to enter the site.
  • eastdakota 14y ago
    With a Layer 4 attack, we don't always know for sure what site is the target of the attack. Traffic is being directed at an IP address and many customers may share that IP. We have techniques to scatter customers across IPs and watch attacks follow, but we generally only do that if the attack is causing a problem. In this case, we were able to mitigate it at our edge and upstream to the point that we didn't need to do any shuffling so from this alone we probably wouldn't have known exactly who the target was. From what happened next, when they attacker shifted to a Layer 7 attack, we were able to determine the target and did reach out to the customer to let them know what was going on and what we were going to do to protect them.
    • andrewcooke 14y ago
      more on layer 7 attacks and cloudflare - http://blog.cloudflare.com/saturday-night-fever-layer-7-attacks-against http://blog.cloudflare.com/saturday-night-fever-layer-7-atta... (i love these posts; i'm old + jaded and have no specialist knowledge of networks and protocols, but they're like the spaghetti westerns of the internet age :o)
  • eps 14y ago
    Another question is how many repeated attacks does it take for an ISP to drop a customer over this. Surely, DDoS doesn't come cheap on the receiving end, even if it can be stopped in a timely manner.
• tayl0r 14y ago
  Does Cloudflare have any competitors yet?
  • snowwrestler 14y ago
    DOSArrest predates Cloudflare and does a fantastic job of DDOS mitigation. But they are not cheap.
  • v33ra 14y ago
    Akamai?
    • moe 14y ago
      Apples and oranges. Akamai recently deflected an attack on the scale of 1 TBit/s and is present in pretty much every DC (~1000 POPs). CloudFlare has 23 POPs and brags about handling 65 GBit/s...
      • dsp 14y ago
        Do you have a source for the 1 Tbps attack? I'd like to read about it, but my search came up empty-handed.
        • eastdakota 14y ago
          :-)
        • donavanm 14y ago
          there are some public studies on DDOS size over the years. IIRC Arbor networks published some numbers earlier this year. IME tens of Gb/s is a mid size attack these days. Large is a hundred plus Gb/s.
          • dsp 14y ago
            I can't find any public reports topping ~100 Gbps. For example, here's a recent article that interviewed an Arbor employee and mentions a ~100 Gbps plateau: http://www.techweekeurope.co.uk/news/ddos-attacks-power2012-86926 http://www.techweekeurope.co.uk/news/ddos-attacks-power2012-...
• ttttannebaum 14y ago
  so, if I'm understanding correctly, this is what's going on? http://i.imgur.com/iSxTQ.jpg http://i.imgur.com/iSxTQ.jpg
  • rhizome 14y ago
    Start here: http://www.pentics.net/ http://www.pentics.net/
• ericcholis 14y ago
  Part of me has to wonder, how wise is it to attack somebody such as Cloudflare? I know they are a juicy target. But, part of their job is to learn and defend against downtime. If their ops are worth a salt (and it appears they are), they've been logging every bit of information they can about these attacks. Logging allows them to do two things: 1) Learn how to mitigate the attack in the future 2) Catalog data on botnets Cataloging data on these botnets is one sure way to get them shut down.
  • dfc 14y ago
    If you were one to brag about such things, what would you rather boast about? I DOS'ed Billy Bob's Bike Repair website or I DOS'ed CloudFlare?
    • dotBen 14y ago
      Billy Bob's Bike Repair, because I know I can bring their site down. I also know that I won't be able to bring down CF or sites running behind their infrastructure so no g33k cred to be had, etc.
      • dfc 14y ago
        You think CloudFlare is infallible? I use their service but I doubt they are the first company in the history of commerce that is perfect.
        • TazeTSchnitzel 14y ago
          They aren't, but they are very, very hard to bring down or slow down without a very big botnet.
  • eastdakota 14y ago
    Yup. We've got tons of data on the machines launching these attacks. We're working on a plan to begin publishing it so even sites not on CloudFlare can better protect themselves. Stay tuned.
• belorn 14y ago
  One can still have a public accessible resolver, so long it is TCP-only. Amplification requires UDP spoofing.
• jakozaur 14y ago
  EDITED: CloudFlare got great technology, got minor issues with billing (mine case), but solved them after this post.
  • eastdakota 14y ago
    That makes me sad too. Just asked our billing team to look into it. PS - found the error on our side. Fixing. You'll be getting an email shortly.
• chacham15 14y ago
  I dont really know much about security hacks, but if open dns is such a problem, then why does google have one (https://developers.google.com/speed/public-dns/docs/using https://developers.google.com/speed/public-dns/docs/using)?
  • nucleardog 14y ago
    Because they're doing it intentionally and knowingly and have worked to mitigate the risks (as detailed on their site - https://developers.google.com/speed/public-dns/docs/security https://developers.google.com/speed/public-dns/docs/security) unlike the vast majority of the open resolvers which have done so unintentionally or without understanding.
    • udpheaders 14y ago
      The real solution is not to use open resolvers and open caches, full stop. Run your own cache on localhost. djb has always advised against third party DNS, but people don't listen. I've even caught the author of the DNS/BIND book admitting it's a smart idea. Moreover you'll be immune from DNS poisoning.
      • sp332 14y ago
        Doesn't you local DNS server have to query other DNS servers to resolve requests anyway?
        • udpheaders 14y ago
          Only if it's not already in your cache or in /etc/hosts You can put hosts file on RAM disk. With some servers it's also possible to save and reload caches. Assuming you're not doing hundreds of thousands of new lookups (sites you've never visited before) every day, it's very easy to configure a system for yourself that is faster than any open resolver. There is one trade-off: if sites switch IP's without telling their users (preferring instead to wait for ttl's in open caches to expire) then for those sites that like to hop from IP to another unexpectedly, you need to monitor for this. This is rare though, and you can try to safeguard against it by "pinging" less oft visited sites periodically, but it does happen occasionally.
• acdha 14y ago
  That might explain djb's tweets a couple days back: https://twitter.com/hashbreaker/status/246745440798781440 https://twitter.com/hashbreaker/status/246745440798781440 and https://twitter.com/hashbreaker/status/246746124222865409 https://twitter.com/hashbreaker/status/246746124222865409 — he's, uh, not a fan of dnssec but this really seems like more of a failure to apply late-90s recommended practice
• astrojams 14y ago
  Error 310 (net::ERR_TOO_MANY_REDIRECTS): There were too many redirects. Am I the only one getting this error?
• frannk 14y ago
  I tried to send a udp packet with fake source Ip(no evil, i am not a attacker;),but i was failed. I seems that the router of the datacenter censor the packets and drop it; who can taught me how to make it?
• Shenglong 14y ago
  I may definitely be missing something here, but I find it difficult to believe not a single packet from that attack made it to their network or affected their operations. I understand how the amplifications were mitigated, but how do you distinguish between legitimate and illegitimate traffic and then block just the illegitimate? I ran an MMO a while ago, and we would have a few hundred login packets spammed every minute. When we were DDoS'd, I responded by moving my server to a larger line (1 gbps) since the DDoS itself wasn't nearly as massive. Yet, we had no way of figuring out (at a base level) what was a legitimate packet.
  • param 14y ago
    As mentioned in the article, they dropped all packets that looked like responses from DNS resolvers. All client applications hosted in cloudfare shouldn't normally receive responses from DNS resolvers.
• tezza 14y ago
  Worth re-stating that they still had a severe outage due to other speculative corrective measures they took. http://blog.cloudflare.com/post-mortem-what-todays-network-outage-looked http://blog.cloudflare.com/post-mortem-what-todays-network-o... Yesterday I posted a post mortem on an outage we had Saturday. The outage was caused when we applied an overly aggressive rate limit to traffic on our network while battling a determined DDoS attacker Kudos for documenting what you did and what worked.
• donavanm 14y ago
  I'm curious about the observed PPS rate. 65 Gb/s is annoyingly large, but network interfaces generally hit pps limits first. The bandwidth graph in this post and post mortem entry is quite interesting. A lot of incoming bytes from customer origins. I'd guess the system cache hit ratio is only 60-70% at peak, dropping to maybe 20-30% during trough. From that I would assume the cache width is quite small, maybe 8-12 hours LRU? I could be misreading that if the average object size is closer to 5kb than 50kb, or if a large number of customers are using it a proxy only fashion.
• TazeTSchnitzel 14y ago
  It's scary that a 65Gbps DDos might soon only require about 100 KC home broadband lines.
• mchahn 14y ago
  If I read this correctly, then googles 8.8.8.8 dns service is an "open resolver". Are they used for dns reflection attacks?