6 ms·
Dyn Statement on 10/21/2016 DDoS Attack
- helthanatos 10y agoI couldn't use twitter or github from noon till 4. If they mitigated the third attack and no customers were affected, why were github and twitter still down for me?
- herlitzj 10y agoSame for us. We're backed by Shopify and it was down until 4pm. Not sure I buy they quashed this by 1pm
- Zancarius 10y agoCan confirm as well. GitHub was inaccessible until about 5PM, and I'm in the southwest. My provider peers with Level3, so it looks like nearly everything gets routed through Los Angeles.
- mirimir 10y ago> We observed 10s of millions of discrete IP addresses associated with the Mirai botnet that were part of the attack. OK, so why don't Dyn staff identify owners of all IP subnets represented, and provide each with a list of participating IP addresses? That's a trivial exercise, right? And maybe they could publicly shame ISPs that didn't act on the information.
- ryanlol 10y ago> And maybe they could publicly shame ISPs that didn't act on the information. so... all of them? Why do you think Dyn should waste their time and money on this?
- mirimir 10y agoISPs are dunning users about "illegal" file sharing. Because of pressure from MPAA etc. These IoT botnets are just testing capabilities, and they're already formidable. Going after operators is good, but it's so damn easy to hide. It seems that IoT botnets are using mainly direct attacks, so they're easy to identify. And why would ISPs want to support botnets? Don't ISP ToS typically say that attacks, threats and spam are grounds for service termination? Maybe this isn't Dyn's job. Maybe some neutral NGO should handle it. But seriously, this is something I could do at home, on an old gaming box in MySQL. If I had the list of IP addresses, anyway. But why? How about acting in the public interest? Or good PR?
- ssharp 10y agoIf they are residential or business IPs, it would be nice for the ISPs to let them know something on their local network is part of a massive botnet.
- mirimir 10y agoThat makes sense to me.
- lilott8 10y agowhy would they? I mean with how they are charging by the GB these days, it gives them financial incentive to not tell you your refrigerator (or any device) is part of a botnet, that's free money for them.
- totalZero 10y agoAssuming you mean this seriously and not just as a tongue-in-cheek dig at internet carriers... Even if every refrigerator were on a pay-as-you-go plan, handling all of the data may ruin throughput for other customers, especially at peak usage.
- kpcyrd 10y agoIn this case the IPs probably aren't spoofed, but there's always the chance that you're blaming IPs that didn't engage in the attack but somebody just put it randomly as source address in their packets.
- mirimir 10y agoTrue. But once you have the IP, you can nmap to get the OS, right? That would probably exclude most false positives, where Mirai was implicated.
- Zancarius 10y agoProbably not in cases where the device is behind a NAT/router that happens to forward remote access ports that lead to the initial compromise. Then you'd be nmapping the customer's endpoint. Plus, it'd be too time consuming. Tens of millions of participating IPs spread across countless ISPs...
- mirimir 10y agoGood point. But still, even with risk of false positives, I believe that ISPs should be informed. And ISPs could acknowledge uncertainty when informing customers.
- NetStrikeForce 10y agoBut in many cases you don't even know if the IP addresses are spoofed or not.
- feld 10y agoYes, because an ISP has nothing better to do than waste money on contacting customers and telling them to unplug their refrigerators. Not going to happen.
- mirimir 10y agoWell, https://en.wikipedia.org/wiki/Copyright_Alert_System https://en.wikipedia.org/wiki/Copyright_Alert_System is happening.
- hga 10y agoAt some point, if the Internet is going to continue to function, one of the likely outcomes is that these customers wake up some morning and the only thing they can get is a web page telling them they have a choice of disabling their fridge's Internet of Shit device or connection, or doing without their full connection to the greater Internet. Compare to, for example, our stopping our use of lead in paint and gasoline/petrol.
- mirimir 10y agoThat's a good analogy. In many places, owners of lead-contaminated buildings must remediate. One issue, of course, is poisoning of resident children. But another is contamination of the surrounding environment. Paint manufacturers are also liable, but that's a separate issue. Basically, this is just environmental pollution.
- ademarre 10y agoI was hoping to read a more detailed account of the attack and specific mitigation strategies. Have such details emerged anywhere?
- rayvd 10y agoThe blog post indicates additional details are forthcoming.
- ademarre 10y agoSure. But they also say, understandably: > It is worth noting that we are unlikely to share all details of the attack and our mitigation efforts to preserve future defenses.
- kpcyrd 10y agoYou can watch their bgp history from that day https://stat.ripe.net/widget/bgplay#w.resource=208.78.70.16 https://stat.ripe.net/widget/bgplay#w.resource=208.78.70.16
- kev009 10y agoDoes anyone know details on the throughput or packets per second? This smells so much of gross negligence on the part of Dyn executives and all the unicorn web executive teams for single sourcing to me. I present as a counterexample and encourage people to go research the architecture of Verisign. I attended a talk by Verisign, which runs .com and .org as well as root servers. They are a constant DDOS target. They are necessarily a single entity of failure and appointed by ICANN to perform these services for the most common TLD in the world. If they mess up, they will probably lose that status. They've had over 16 years of uptime. Every layer of the stack is dual or triple sourced. Two server makers, two generations, two router vendors, two switch vendors, two network architectures, POP diversity, peering diversity. Services and capacity always added in pairs. Two separate NOC and Ops teams. FreeBSD, Linux and Solaris. nsd, bind, and internally developed userspace on Netmap. Code upgrade deployed in halves. Everything is structured for at least a full halving for The Big One zero day I don't think we've really seen yet. They were doing 10gbps of DNS on a single commodity server 3 years ago. This makes it easy to absorb any DDOS and gradually clamp it at the public peering points. The above should be standard operations structure for any breakaway success web business. It's not that hard, but you have to claw the charlatans out of the management chain and put in professionals that take the career seriously. Professionals. What's really appalling is hearing some unicorn web biz uses one cloud vendor, one service provider like Dyn. This is absolutely trivial stuff to multi-source and has almost no effect on OpEx. Last I heard from a Dyn insider was they were eliminating FreeBSD and had a brain drain a few years ago. Overall pretty unsurprising outcome. Internet reliability no more complex and certainly much cheaper than analogous critical infrastructure like electrical generation for a region. I am constantly disappointed in this field for its lack of recognizing and installing professionals into management. We've been doing Internet architecture in modern form for 30 years. Get your shit together.
- detaro 10y agoWhat kind of single-sourcing do you think was an issue on friday/how does double-sourcing help against pure DDOS attacks? I haven't read anything about a specific bug being exploited?
- 10y ago
- cagenut 10y agothat's a killer IP list to license out
- pragone 10y agoEntertainingly, this page 503s currently.
- Crosseye_Jack 10y agohttps://web.archive.org/web/20161022220033/http://hub.dyn.com/dyn-blog/dyn-statement-on-10-21-2016-ddos-attack https://web.archive.org/web/20161022220033/http://hub.dyn.co...
- stordoff 10y ago400 error for me (Bad Request - Your browser sent a request that this server could not understand).
- CaveTech 10y agoThe attack was surely not mitigated by 1 PM ET. We were experiencing issues until well after 3.
- HappyTypist 10y agoOur Dyn account manager confirmed to us that issues were ongoing at 2:27pm ET.
- miken123 10y agoDyn states their problems were fixed at 17 UTC, but me and most other people in the Netherlands have been seeing issues for the whole evening. Something about their story does not add up...
- kimputin 10y agoIn the caribbean there were issues up to 5 PM
- rasz_pl 10y agoThe attack was not mitigated _at all_, attacker simply stopped. This post mortem is a big fat lie.
- libeclipse 10y ago>It is worth noting that we are unlikely to share all details of the attack and our mitigation efforts to preserve future defenses. Security through obscurity.
- labster 10y agoObscurity is a valid defense layer, so long as it is not the only layer, and so long as the obscurity is not just a way of covering up garbage.
- dllthomas 10y agoBut it can also be substantially more costly than its worth, and a lot of the cost isn't super visible (people finding it marginally harder to get their job done, being unable to learn from informed discussion on HN, &c).
- Shank 10y agoI would really like it if Dyn could give some credence to the 1.2Tpbs number or if it was higher. As these attacks grow in scale, it becomes more and more important to know if this was an attack at record capacity or if it was just that Dyn had lower capacity hardware/links in place. If we're already facing, for example, 2Tbps attacks, a lot has to be done to make mitigating these attacks easier, either through hardware or strategic upgrades.
- jimjimjim 10y agono. what you get then is groups trying to outdo each other trying to get bragging rights for being the largest.
- nmjohn 10y ago> We observed 10s of millions of discrete IP addresses associated with the Mirai botnet that were part of the attack. (linked article) > ... the Mirai botnet was at about 550,000 nodes, and that approximately 10 percent were involved in the attack on Dyn (from Level 3 CISO) [0] Something really doesn't add up there - even if it turned out 100% of infected hosts in the Mirai botnet were targeting dyn (ie: 5.5 million nodes) - that still is a fraction of the number dyn is claiming. [0]: https://threatpost.com/mirai-fueled-iot-botnet-behind-ddos-attacks-on-dns-providers/121475/ https://threatpost.com/mirai-fueled-iot-botnet-behind-ddos-a...
- dmourati 10y agoI believe Dyn's numbers are conflating two things: 1. The number of source IPs seen 2. The size of the botnet They are most certainly not equal because of spoofing.
- dronemallone 10y agoDoes anyone here know the actual technical details of the attack? How exactly did the DDoS occur? Just a ton of bots making DNS requests over UDP?? Links to articles with tech details would be greatly appreciated.
- nodesocket 10y agoOverall great write up, however: "Again, at no time was there a network-wide outage, though some customers would have seen extended latency delays during that time." That can't be true. Here in the west coast, I know of over 10 "name-brand" sites that were absolutely down for over an hour. The east coast was apparently hit even harder and for longer.
- joatmon-snoo 10y agoAre you sure that it was Dyn that was down, or if there was a break somewhere in the nameserver chain? I know that the digs I was running periodically were getting SERVFAIL responses from Google DNS even though my local nameservers were actually succeeding.
- diegorbaquero 10y agoI haven't seen comment from any IP-transit/peering company that provides service to them. In fact, I haven't seen any infrastructure provider of them comment anything. This is starting to seem more like a marketing campaign or human error (action?) blamed on DDoS.
- paradite 10y agoThe style of this post mortem is quite different from a typical post mortem of an attack on a large tech company, such as AWS, heroku or GitHub. There is very little technical details on the investigation and mitigation process beyond phrases like "the NOC team was able to mitigate the attack and restore service to customers", "...but was mitigated in just over an hour". Why was this written by a Chief Strategy Officer but not someone with more technical knowledge and insights?
- mino 10y agoBecause it's not a post-mortem but a "statement", which they felt like publishing to thank all the actor involved.
- laluser 10y agoThis should be a warning to everyone that was affected by this. Even the dependencies that you don't normally think about go down at some point. For DNS, make sure you are using multiple DNS providers.
- buro9 10y ago> For DNS, make sure you are using multiple DNS providers. In this case, that would have made it worse. If your domain CNAMEs off another provider (as nearly all SaaS solutions do, as do mapping to AWS servers, etc) then you would have been affected by the attack on Dyn regardless of whether you could change your nameserver in time and have multiple providers at that level. If you don't have any CNAMEs, then I think the better choice is to go same-origin same-provider for everything. Which may sound bizarre... my personal sites (a lot of forums) would either be up or down. I would argue that availability having a binary nature is a lot better for end users than a constantly broken half-state that is frustrating or impossible to use at any time even though a graph might show partial availability. A hard sudden failure shows up better in monitoring, triggers end user feedback faster, and is in many ways easier to debug and solve. With so much CNAME'd, and SaaS still growing, replicating and failing over DNS that sits above those wouldn't buy you anything. A lot of sites lost Zendesk (their support.), Statuspage (their status.), Pagerduty (hey, they were quiet because no-one could let them know). They couldn't even help the end users having a bad time.
- eslater 10y ago> In this case, that would have made it worse. It might have made little difference depending on the specifics of the records in question but it would not have made things worse.
- willvarfar 10y agoWhy might the Mirai botnet attack Dyn? What might their motives be?
- deleted 10y ago[deleted]
- paulddraper 10y agoIs Dyn any less vulnerable to this attack than 48 hours ago? That's kind of the point of retrospection, but I missed the part where they say what they're doing now that they failed to do last week.