6 ms·
400Gbps: Winter of Whopping Weekend DDoS Attacks
- the_arun 11y agoI wish we could see an article like this for AWS.
- caleblloyd 11y agoI don't think AWS mitigates DDOS. If an app is under DDOS at AWS and was using an Elastic Load Balancer, the app would have to pay for the inbound bandwidth through the ELB and would have to pay for scale-up to respond to the increased traffic load. It'd be much more cost effective to pay CloudFlare a fixed monthly fee and stop the DDOS attack before it even gets to your AWS resources.
- themartorana 11y agoThe more I read from CF, the more I'm impressed and am definitely considering this.
- virtuallynathan 11y agoLast time I looked at the AWS jobs site, they were hiring for a DDoS mitigation team.
- pyvpx 11y agohaving a slight understanding but no direct knowledge of their network infrastructure, that's probably for financial not technical reasons (they can handle the largest volumetric attacks and then some)
- themartorana 11y agoAgreed. We're full-stack AWS hosted, and I have no real indication of whether or not we've ever been attacked (doubtful) or if AWS could automatically mitigate an attack (possibly??). I presume Netflix is probably a common target, and they're AWS-heavy, but I have no way of knowing for sure.
- mschuster91 11y ago> I presume Netflix is probably a common target, and they're AWS-heavy, but I have no way of knowing for sure. Well, DDoS operators usually don't do their job for fun (okay, I'm discarding Lulzsec and other vandalism-motivated folks here). It's much more profitable to hunt for not-CDN-backed sites and extort them for money. Or to look for processor-heavy API endpoints (e.g. login with huge work factors) to improve the DDoS effectivity, but then again a lot of DDoS ops are script-kiddies looking for a quick bunch of monies, all too often not skilled enough for this.
- caleblloyd 11y agoELB is simply a Reverse Proxy with an upstream pointing to your app servers. Last time I checked, I think they were running nginx. I doubt they have advanced DDOS detection code running on that reverse proxy. Netflix likely implements their own DDOS protection layer somewhere in their routing library. For example, since most resource-intensive Netflix endpoints are protected (you have to login to watch a movie), they could use rate-limiting based on User Account ID.
- falcolas 11y agoHe last DDoS article from Amazon was a white paper which amounted to "we have enough capacity to let clients out-scale any attack." AKA it becomes a DDoS on your wallet instead of your server. Edit: found it. https://d0.awsstatic.com/whitepapers/DDoS_White_Paper_June2015.pdf https://d0.awsstatic.com/whitepapers/DDoS_White_Paper_June20...
- rhavar 11y agoI run one of the largest legal bitcoin gambling sites (name in profile if interested) and was recently extorted for a whole week by DDoS's unless I paid (something that I never would do). The specifics of our casino require us to have a pretty quality low-latency connection, so AWS was a first choice. I was recommended to use ELB which could mitigate the level3 attacks (primariy SYN floods we were seeing) and automatically scale. I setup a test instances, and tried orchestrate my own SYN flood which it handled without issue (while a normal ec2 instance would just collapse). So I put production behind ELBs, and a few hours later the attacks started. The first thing was that under stress ELB stops being able to make connections, and starts failing your origins for status check (even when they were getting no load), but was able to successfully scale to mitigate the attack within 5-7 minutes. The attacker tried again a few hours later, but it seems things were nice and pre-warmed and the attack didn't get far. For about a day there was a nice calm, and ELBs was performing absolutely fantastic (in terms of latency/jitter), and then I got the email that he's about to throw everything at me. And within a few minutes, he took out all the ELBs and it wasn't able to scale to handle the traffic for over an hour this time. During this time I was in contact with CF's excellent support, and they offered me protection for websockets. It has slightly more latency than ELBs, but is actually an anti-ddos service, and works wonderfully. Since being fully behind CF we've only had one DDoS related outage, and it was specific to Seoul for about ~3 or 4 hours. Can't recommend them enough, quite literally my business might not be around if it wasn't for them.
- chinathrow 11y agohttps://www.youtube.com/watch?v=OT2y3DzMEmQ https://www.youtube.com/watch?v=OT2y3DzMEmQ
- nxzero 11y agoFunny, stories like this to me always read as someone hacking themselves for job security; clearly not the case. Curious who the target was Cloudflare, or their clients.
- stepvhen 11y agoThe article mentions some of the recent attacks were on fairly benign websites, though that's subjective and could be construed as a sales pitch.
- nxzero 11y agoMissed that, thanks. Seems if they truly were websites of little note that the target was likely Cloudflair; my opinion, though guessing if they went down, the real attack would happen.
- eastdakota 11y agoTargets are typically clients. Sometimes we can reasonable speculate as to motive, other times we're perplexed. One example: we saw a huge uptick in attacks targeting adoption agencies. Couldn't figure it out until we learned that adoption of Russian children by US/EU parents was a major hot button political issue in Russia. In other words, there's always a motive, it's just not always clear to us what it is.
- nxzero 11y agoYeah, that's obvious to me that the adoption sites like that are targets, but see how it be bit puzzling off the cuff, or something of the sort. Not sure in fact what for sure would be a target that clearly lacks anything of note.
- eastdakota 11y agoUsually forums of one sort or another. Obviously they're being attacked for something that's being posted there, but usually we have no idea exactly what or why.
- jacquesm 11y agoHas there been any recent jump in the size or grade of bots available to botnet operators or is there some other technological reason underlying this change in magnitude?
- amiraliakbari 11y agoIt seems that the magnitude of monitored attacks is increased, not necessarily the original size. From the last paragraph: > You might think that as CloudFlare grows, the relative scale of attacks will shrink in comparison to legitimate traffic. The funny thing is, the opposite is true. Every time we upgrade our network capacity, we see larger attacks because we don’t have to resort to blackholing. With more capacity, we’re able to withstand and accurately measure even bigger attacks.
- ChuckMcM 11y agoI think its a good question to ask if the way in which attacks are generated these days has enabled greater size. Cloudflare points out that they may have already been this big, they just couldn't measure them before. What I find most interesting is that this problem is providing a lot of pressure on ISPs to co-operate more. There are a lot of competing agendas, which we've seen with the very outsized fights of NetFlix against tier 1 and tier 2 transit providers, but those transit providers are being hurt by these people. By cooperating at some level it may be possible to mitigate the attack surface available to DDoS agents.
- coldcode 11y agoGiven the size of the attacks, shouldn't it be possible to identify who is behind these (or at least where it is coming from)? Or are they all distributed attacks from random hacked PCs? I find it hard to believe there are still enough vulnerable PCs: the count of those should be going down, not up. I guess if the source is nations the actually source might be difficult to pinpoint.
- caleblloyd 11y agoThere are plenty of vulnerable PCs: http://krebsonsecurity.com/tag/botnet/ http://krebsonsecurity.com/tag/botnet/
- cft 11y agoThese volumetric attacks are likely UDP amplification attacks from hacked servers, not PC s.
- pilsetnieks 11y agoThey don't even have to be hacked, just expose vulnerable services.
- bcook 11y agoI think OP is referring to the origin, not the mirror.
- deleted 11y ago[deleted]
- deleted 11y ago[deleted]
- codecamper 11y agoHmm Looks like if you want to benefit from this level 3 DDos protection, you'll need to hand $200 / month over to Cloudflare. Sounds fishy to me.
- eastdakota 11y agoActually, in the case of the DDoS that is primarily the focus of this post, the site being attacked used the free version of our service. Volumetric attacks have become automated enough to deal with that we find the benefits of stopping them, even for free customers, and recording the resources used by the attackers, is well worth it. Plus we generally believe you shouldn't have to pay in order to be safe from bullies online. Our free level of DDoS mitigation today is better than what you could have paid for, at any cost, from any provider 18 months ago.
- pilif 11y ago$200/month feels cheap to me considering how much effort and money it would take to get this defense if you are on your own. Just talking about the bandwidth cost, if it was even possible for me to get the 400gbit/s bandwidth required to weather such an attack, it would cost me way more than $200 per day with my current ISP. And that doesn't yet include my own time and stress. If the stuff we're running was valuable enough to DOS, I would totally consider the 200/month extra
- contrav_throwa 11y agoI think what the GP was saying was that it looks like it's possible that CloudFlare is causing these attacks themselves. They can magically block them for clients because they can just turn the attack off.
- cft 11y agoToo bad this was written as a Cloudflare advert, instead of emphasizing the changes that ISPs need to make (ingress source address verification) to make these attacks much harder. https://www.ietf.org/rfc/rfc2827.txt https://www.ietf.org/rfc/rfc2827.txt
- ra1n85 11y agoEgress address verification? Are you referring to RPF, or http://www.bcp38.info/index.php/Main_Page http://www.bcp38.info/index.php/Main_Page ? uRPF is a challenge. In some cases, the consequence is flows must route symmetrically through a given network. That's not always the most efficient, flexible, or cost-effective way to go, particularly if you're a large provider sloshing massive amounts of traffic through your network.
- pyvpx 11y agosmart edge, dumb core. if edge networks implemented uRPF, it would put a huge dent in the ease of mounting DDoS attacks. unfortunately, the economics aren't there.
- ra1n85 11y agoAgreed with your first point, but not sure about your second - how many of these attacks could really be dented with uRPF? My understanding is the majority are either distributed (coming from multiple ASes) or from compromised accounts in Azure, GC, or AWS. If uRPF was stopping me from successfully attacking a destination, I would just not spoof. ACLs, uRPF, and anything stateful are still not going to 100% mitigate an attack from a broad range of addresses.
- jgrahamc 11y agoWe've written about BCP 38 repeatedly: https://blog.cloudflare.com/understanding-and-mitigating-ntp-based-ddos-attacks/ https://blog.cloudflare.com/understanding-and-mitigating-ntp... https://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplification-ddos-attack/ https://blog.cloudflare.com/technical-details-behind-a-400gb... https://blog.cloudflare.com/the-ddos-that-knocked-spamhaus-offline-and-ho/ https://blog.cloudflare.com/the-ddos-that-knocked-spamhaus-o...
- deleted 11y ago[deleted]
- artursapek 11y agoInternet connection speeds didn't improve 15x overnight. That's part of why this is newsworthy.
- ryao 11y agoIf internet connection speeds continue improve, it is fairly obvious DDOS traffic will increase. Claiming that it is bigger than ever when Internet connection speeds are higher than ever is like claiming that gravity from the Sun is stronger when you are closer to the Sun.
- taf2 11y agoPretty sure they are saying that because they can handle more traffic they can measure the attack now instead of blackholing...
- eXpl0it3r 11y agoMaybe a little less "we are the best", "our hardware can handle nearly everything", "we can take it", etc. next time...
- pyvpx 11y agothere is a reason some folks call them blogflare. They have great engineering talent and a marketing team that knows exactly how to acquire customers and more engineering talent. and that's through their blog posts.
- jgrahamc 11y agoActually we've only recently created a marketing team at CloudFlare and hired someone to run it. Most of the blog posts you see are generated organically by engineers or others who want are interested in writing about something they've worked on.
- pyvpx 11y agogreat engineers who can also articulate the technology in a polished, publishable form? not saying it's impossible, or that you didn't just setup a marketing team. but please pass my regards to whomever is editing before publishing. it's got a perfect, polished tone that any marketer would want.
- jgrahamc 11y agoNow that we have a marketing team we do have someone on that team who helps polish posts but that's a pretty new thing. In the past posts were polished by the likes of me or eastdakota. But that's assuming that they need polishing; one thing we look for in hiring is ability to explain.
- Drdrdrq 11y agoA bit of feedback... I usually love your articles, but I really hated this one. It is just so blatantly self-promoting it turns me off. Just something to consider... Marketing can be overdone easily.
- joelthelion 11y agoWho's behind these?
- contrav_throwa 11y agoMaybe it's someone with an investment in CloudFlare? 'Look at these awful DDOS attacks everyone is getting? We have magic techniques to stop them if you pay us.'
- ChuckMcM 11y agoI always appreciate Cloudflare's write ups. That said, once folks are attacking it, and since you can measure things without black holing traffic, is it possible to identify machines participating ? Specifically to work with ISPs and network service vendors such that such attacks would serve to highlight compromised machines and then working with the ISPs of those compromised machines create an automated channel back to the operator?
- kkirsche 11y agoIt depends heavily on the type of attack. In many cases it is not like a syn flood which commonly used random spoofed (i.e. fake) source ip addresses.
- plainOldText 11y agoWith more and more network attacks and an increase in security breaches, I think we need to rethink some parts of the Internet for better security and reliability. There's an interesting project going on - SAFENetwork[1] - similar to ipfs, which has the potential to render DDoS attacks obsolete for some use cases, if it gets adoption of course. [1] https://safenetwork.org https://safenetwork.org
- r1ch 11y agoI'm curious what happens if a free website is on the receiving end of a L3 attack, since only business plans and higher are advertised as having "Advanced DDoS support". Does CF dynamically adjust DNS and send the malicious traffic to your origin once it hits a certain threshold?
- jgrahamc 11y agoMatthew answered that below, but at this point our L3/L4 attack handling is mostly automatic and we would protect the FREE web site. We can dynamically adjust DNS to do all sorts of things when we feel like it.
- mmmBacon 11y agoIn the lab it's difficult to generate 400Gb/s traffic. From my perspective it seems like this attack is pretty sophisticated and has access to some pretty fat pipes. It's hard to imagine that this scale of attack could come from some infected PCs. Can an expert weigh in here on the difficulty of generating this much traffic in an attack?
- pmlnr 11y agoDejavu. "the largest distributed denial of service (DDoS) attacks ever seen" ... returns again part 2