29 ms·
Ask HN: How did Dyn fail to fend off DDOS?
I'd imagine that DDoS attacks is something that DYN and other DNS providers would spend a lot of resources to prevent. Was there something specific about this DDoS attack that DYN was unprepared for? Or is there some reason that distributed natural of DNS makes it hard to prevent DDoS? Anyone know of any steps that DNS guys are taking to prevent another DDoS?
- meira 10y agoProbably they got beaten because of orders of magnitude. They were prepared, but not for cyber nuclear war.
- qaq 10y agoif the attack is sufficiently distributed and scale is very large it can knock out even much bigger targets. I think there have been attacks at over 600 Gbps scale.
- nerdy 10y agoOVH DDoS late last month was over 1.5Tbps: https://twitter.com/olesovhcom/status/779297257199964160 https://twitter.com/olesovhcom/status/779297257199964160 I believe the Dyn attack was via Mirai also.
- matheweis 10y agoIndeed, flashpoint (1) confirmed that the botnet attacking Dyn was the same one that attacked Krebs (2), and Krebs has more details as well (3). The previous attack on Krebs was seen to exceed 620Gbps. 1. https://www.flashpoint-intel.com/mirai-botnet-linked-dyn-dns-ddos-attacks/ https://www.flashpoint-intel.com/mirai-botnet-linked-dyn-dns... 2. https://krebsonsecurity.com/2016/09/krebsonsecurity-hit-with-record-ddos/ https://krebsonsecurity.com/2016/09/krebsonsecurity-hit-with... 3. https://krebsonsecurity.com/2016/10/hacked-cameras-dvrs-powered-todays-massive-internet-outage/ https://krebsonsecurity.com/2016/10/hacked-cameras-dvrs-powe...
- nickthemagicman 10y agoWow. That means the same culprits are still out there with their botnet? And it's still growing?
- idlewords 10y agoThe code for it has been released on Github, so there are now likely to be many botnets.
- nickthemagicman 10y agoLol. Thats insane!
- Matt3o12_ 10y agoI'm not too sure. I have heard that the attack also fixed the security vulnerability (changing the default root password) after installing the back door so other people cannot use it. Although the source code is out there, those will not be able to control all those devices.
- nerdy 10y agoI'm not sure. Maybe that's the case for the passwords which can be changed via the administrative app but I read many of these are in firmware and not able to be disabled or changed: “The issue with these particular devices is that a user cannot feasibly change this password,” Flashpoint’s Zach Wikholm told KrebsOnSecurity. “The password is hardcoded into the firmware, and the tools necessary to disable it are not present. - https://krebsonsecurity.com/2016/10/hacked-cameras-dvrs-powered-todays-massive-internet-outage/ https://krebsonsecurity.com/2016/10/hacked-cameras-dvrs-powe... That's not to say it couldn't flash the devices but I don't recall seeing that capability in the Mirai source and haven't read about it doing so.
- asddddd 10y ago> While Flashpoint has confirmed that Mirai botnets were used in the October 21, 2016 attack against Dyn, they were separate and distinct botnets from those used to execute the DDoS attacks against “Krebs on Security” and OVH. So not quite. > Dale Drew, chief security officer at Level 3, an internet service provider, found evidence that roughly 10 percent of all devices co-opted by Mirai were being used to attack Dyn’s servers. Just one week ago, Level 3 found that 493,000 devices had been infected with Mirai malware, nearly double the number infected last month. http://www.nytimes.com/2016/10/22/business/internet-problems-attack.html http://www.nytimes.com/2016/10/22/business/internet-problems... If they aren't significantly underestimating the number of devices participating in this attack, it paints an ugly picture of things to come. My understanding is these botnets are almost impossible to eradicate due to how fast/easy it is to re-compromise the devices, so traditional methods of taking out C2s do almost nothing. Bonus - Mirai source code is freely and easily available for skids to use now, so there's no single threat actor for attribution/retaliation/arrest/etc.
- inetsee 10y agoHackers have started to use insecure Internet of Things devices, especially internet connected video cameras, to produce DDoS attacks larger than have ever been seen before. The KrebsonSecurity website was hit by a DDoS that was twice as large as the previous largest attack seen by Akemai, and there have been larger attacks since. The problem will continue, and may get even worse, since many of the insecure internet attached video cameras are insecure because of passwords hard-coded into the devices; they can't be easily made more secure.
- hannob 10y agoI think the answer is surprisingly simple: The attack was just huge. The unfortunate truth is that with the Internet of Things the amount of devices that can easily be taken over has grown so fast that we see DDoS attacks of unprecedented size. Even more unfortunate is that there is no sign whatsoever that this is going down again.
- ericcholis 10y agoDoes anybody have solid recommendations for secure IoT devices? Initial searches lead me to believe that they are non-existent.
- joshmn 10y agoWell, a good initial step is usually changing the default password.
- snowwolf 10y agoA good initial step is not to have a default password. There was a time when all routers came with a default password and people were told to change it. They didn't. Now most new routers come with a randomly generated unique password printed on a sticker under the router. IoT devices should follow the same practice.
- mathrawka 10y agoChange the default admin password. The original Mirai program tried a little over 60 passwords and it would just brute force into an IoT device.[1] From what I read, it seems that one specific manufacturer in China is the owner of a lot of devices used in the Mirai botnet attacks.[2] 1: https://github.com/jgamblin/Mirai-Source-Code/blob/master/mirai/bot/scanner.c#L123-L185 https://github.com/jgamblin/Mirai-Source-Code/blob/master/mi... 2: (I cannot find the link, but it was an article from yesterday) EDIT: Found this when googling the strange '7ujMko0admin' password in Mirai: http://www.cam-it.org/index.php?topic=9396.0 http://www.cam-it.org/index.php?topic=9396.0 So it looks like the Chinese manufacturer that they target is Dahua.
- t3ra 10y agoI would also like to know what exactly are "a lot of resources to prevent."?
- bklyn11201 10y agoI've been waiting for some announcement around the Gbps of the DDOS similar to this Cloudflare announcement: https://blog.cloudflare.com/technical-details-behind-a-400gbps-ntp-amplification-ddos-attack/ Does DYN routinely deal with very large DDOS which would past this attack in a new category? Can someone who attends security conferences with DYN personnel comment?
- beachstartup 10y agolast night the consensus was 1.2 tbps.
- 45h34jh53k4j 10y agoor 2x krebs, the 2nd? previously largest in history; we could use that or this incident as the future benchmark of ddos capacity. Attacker may have been involved with the 1.5Tb against OVH.
- quantumhobbit 10y agoIs Brian Krebs going to become a unit of measurement for ddos attacks? Because that would be awesome. Ex. "I can't believe our network can't handle that traffic, it is only 20 milliKrebs!"
- deleted 10y ago[deleted]
- bklyn11201 10y agoFor a DNS-only service provider, it seems like 1.2Tbps could be 1000x normal traffic. But Akamai claims 30Tbps+ is their routine traffic[1]. Some have commented that this DDOS questions consolidation around cloud providers, but I think it will cause consolidation among service providers. You can no longer be a critical service provider if you don't have the capacity to absorb attacks like this. http://www.csoonline.com/article/3123797/security/some-thoughts-on-the-krebs-situation-akamai-made-a-painful-business-call.html http://www.csoonline.com/article/3123797/security/some-thoug...
- NelsonMinar 10y agoI've been wondering if the UDP nature of a DNS server makes it harder to protect. Particularly coupled with the amplification attacks that DNS makes possible.
- Animats 10y agoThat's part of the problem. DNS servers should probably reject queries that require long answers when they come in over UDP. If you want a zone transfer, use TCP. That prevents amplification attacks.
- marcosdumay 10y agoYes, it does. But no, it does not seem to make any difference this one time. In a DNS based amplification attack, you use several DNS servers to take down some other unrelated service, this time it's just a lot of devices in a botnet attacking the DNS servers directly.
- akulbe 10y agoI wonder how much of this would be mitigated/avoided if folks would just change to something other than the default credentials on IoT devices? Is it that simple? or am I missing something?
- akulbe 10y agoA downvote for a legitimate question. NICE.
- gagan2020 10y agoJust thinking, Is there any chinese production of IOT involved? might be firmware involved?
- ag_47 10y agoHere's a pretty fun 2 year old talk from Blackhat, re surveillance camera firmware and how s*tty they are: https://www.youtube.com/watch?v=B8DjTcANBx0 https://www.youtube.com/watch?v=B8DjTcANBx0
- sn41 10y agoThat's what the following blog claims: https://krebsonsecurity.com/2016/10/hacked-cameras-dvrs-powered-todays-massive-internet-outage/ https://krebsonsecurity.com/2016/10/hacked-cameras-dvrs-powe... I don't know any other independent researcher who confirms this.
- gagan2020 10y agoThat's interesting since they faced the attack so they could have data to analyze that. Apart from firmware, chinese companies also pushing UC browser and WeChat like anything.
- 45h34jh53k4j 10y agoI think this is a plausible theory of the attack - (first seen in from npr report on incident): NANOG 68 BackConnects Suspicious BGP Hijacks is shown 4ish days ago. Last talk of the night, discusses BGP hijacking shenanigans and krebs; touches on MO of possible attacker. Speaker is Director at Dyn. Attack in retaliation. So far the targets have been organisations that have responded to or made allegations of corrupt DDoS business. Please don't buy into all this cyberwar bullshit, this may just be a well resourced (its really not that hard to pop boxes with default passwords.....) attacker doing criminal response to commentary.
- ryanlol 10y agoThis is likely, Backconnect hosted Mirai in the past right before attacks on Krebs. (however not during them.) There's also no small amounts of publicly available evidence that Backconnect used insider information provided by their CEO (ex Staminus employee) to compromise Staminus network earlier this year by hijacking a management range of theirs.
- zzzcpan 10y agoPresumably it would take a lot of cooperation with ISPs they are peering with, which is not something easily done. Or a google-sized network.
- beachstartup 10y agoi think there is a larger strategy at play. this is pure speculation and anecdote. recently there has been an aggressive uptick of dns ddos attacks against smaller companies/service providers that run their own dns infrastructure. this includes small/regional internet service providers and individual sites/hosts that still run their own servers. in almost all of these cases that i'm aware of, the smaller companies immediately outsourced their dns services to a larger company, one that ostensibly is able to either absorb, scrub, or otherwise defend against these types of attacks. extrapolating to a global scale, what's happening is a forced consolidation of dns infrastructure into a handful of large players. even in the case of having redundant providers, it's usually two very large providers. and as we just saw today, a terabit-level attack is not something we can readily defend against. what if there's even more in reserve? in other words, we're putting all of our eggs into one basket. and someone is aggregating enough attack capacity to take out nearly the entire internet at once. it doesn't help that everyone is voluntarily consolidating their infrastructure onto a small handful of public cloud providers. we are setting ourselves up for a massive internet outage.
- Silhouette 10y agoThe real question here is whether there was anything they could realistically have done to prevent it at all. In order to defend against a DDoS attack, you really only have two options. One is to have sufficient capacity to cope with the extra load without undermining your normal service. The other is to reduce the amount of extra load you have to handle, by identifying and blocking the hostile traffic at some point before your main system deals with it fully. In this case, the scale of the attack was huge thanks to all the woefully insecure IoT devices out there. But worse, from the initial reports it appears that the requests being sent were effectively indistinguishable from valid DNS requests: they came from diverse sources, and asked DynDNS to do exactly what it's normally supposed to do, just for random subdomains that don't actually exist. Unless there is some pattern in those requests that allows for identification of the hostile incoming traffic so it can be dropped early, there's probably very little DynDNS could have done here. And of course the attack is particularly effective because by taking out infrastructure rather than attacking a specific site, it brings down large numbers of high profile sites all at once. It is disturbing, but apparently the reality we face, that there are now so many hopelessly insecure devices on the public Internet that this is possible. The best long term strategy for dealing with it seems to be trying to improve the standards of Internet-connected devices and reduce the number of highly vulnerable devices with access to the Internet, but this was always going to be difficult with IoT products aimed at the general public. I suspect some sort of remediation/recall scheme for manufacturers/vendors and some sort of throttling of users' Internet connections to force them to respond to security recall/update notices may be necessary if this kind of attack starts to become a pattern.
- mrcabada 10y agoI wonder if there's any way to tell apart real-users-requests from fake-users-requests. If I'm not wrong, it's only preventable by increasing the resources of the server, doing anti-bots things like CAPTCHAS (not feasible for stand-alone IoT devices) or detecting weird patterns (which can be masked really easily). How will DDoS attack be preventable in the future? There will be so many things and nano-thing connected to the internet that can act as "attackers". Is getting harder and harder everyday.
- mabbo 10y agoWhat software is the piece that is answering the question "is this a real user or fake?". Because that's the piece that will fall over during a DDoS, as it's doing per-request processing.
- 45h34jh53k4j 10y agoI would like to remind those that think all is lost with this: A serious conversation with vendors about default passwords and backdoors post this incident will help prevent recurrence. This has forced this talk and we are better for it. There was a time when your windows box would get popped from being online for more than 4 minutes. We recovered from this. Conficker in 2008. Blaster in 2003. It was a 'BIG BOTNETS OH NO', but we cleaned up, recovered, hardened. Microsoft went from being botnet enabler to an active force in dismantling bots and crime rings. It sucks, and some of us have a bad day, but we recover ever stronger. XiongMai Technologies may well find themselves in some international hot water over this incident, and I think they deserve it. They sold a faulty product that caused billions of dollars in lost revenue to some very large internet properties for a day in October 2016. I would encourage vendors look at these incidents from last decade and how these were turning points for upping their security game. I would encourage its victims to investigate legal recourse. Specifically the current vulnerable nodes of Mirai, i am sure these will be removed from the internet pretty soon. One only gets to fire something like this a few times before the feds are on the door. Your regularly scheduled program will commence shortly.
- ebbv 10y agoif you draw the line between attacks of the past through this one you see that the scale of DDoS attacks continues to get worse. It's all well and good to say that the enablers of the past learn and improve their products. The problem is continually the enablers of the future. It's been said before but I will repeat it here; manufacturers have no reason to expend any resources on security until they are held liable for the damage they facilitate. We must make selling insecure devices a liability just like selling unsafe devices is in meatspace.
- Silhouette 10y agoI would encourage its victims to investigate legal recourse. It's all well and good saying that, and yes, if manufacturers are repeatedly/grossly negligent then maybe they should pay compensation and/or punitive financial penalties. However, unless you know something the rest of us don't about how to guarantee Internet-connected devices are perfectly secure, that sort of financial pressure can't be the whole solution, or even the main part of the solution. Ultimately, it may just mean that smaller players can't afford to risk participating in the industry any more, and no-one will be better off if reduced competition is the main result of this. We must be able to handle this more constructively than just demanding perfection and punishing those who inevitably fail to deliver it.
- Animats 10y agoIt's time to apply some serious pain to the junk IoT manufacturers, retailers, distributors, and importers. A nice big billion-dollar lawsuit against Amazon for gross negligence would be a good way to start. US consumer law allows suing everybody in the supply chain. (They can then sue each other and try to sort out who pays, but that's not the victim's problem.) We also need some big recalls. If Homeland Security tells the Consumer Product Safety Commission this is a national safety issue, the CPSC can order a recall. Something like this worked with those exploding "hoverboards". CPSC ordered recalls, Amazon took the junk back, and Amazon refused to pay manufactures in Shentzen. The manufacturers were furious, but hoverboards with crap batteries disappeared from the market very fast.
- deleted 10y ago[deleted]
- verroq 10y agoI think the more realistic solution is that a vigilante group of hackers continuously scan and take over vulnerable IOT boxes with the intention of bricking and/or disabling their network access would be the most feasible.
- snowwrestler 10y agoThe problem with this idea is that it is illegal, and federal agents are much better at tracking people down on the Internet than they were even 5 years ago. So while I think a lot of us would cheer the vigilantes on, they would be taking a serious personal risk.
- dadi123 10y agothe dns attack was called "dns water torture":fix a domain such as google.com and random the subdomain such as {random str}.google.com. This attack method is hard to prevent