11 ms·
Dyn Analysis Summary of Friday October 21 Attack
- edutechnion 10y agoChanges made by large companies that relied 100% on DYN: * us-east-1.amazonaws.com: split between internal, UltraDNS, DYN * spotify.com: all internal nameservers now * reddit.com: all Route53 now * github.com: all Route53 now * netflix.com: all Route53 now * paypal.com: split between UltraDNS and DYN No changes made: * twitter.com: 100% with DYN
- JohnTHaller 10y agoAll our eggs in that basket got crushed. Let's take this basket over here and put all our eggs in that instead. Facepalm
- MichaelGG 10y agoIt's probably temporary? Move to R53, then figure out what's needed to manage records on two providers (if only for internal processes). Top engineering teams aren't going to knee-jerk this, right? Or did Dyn show some unfixable incompetence?
- deathanatos 10y agoIs it? I thought GitHub, at least, was split Dyn/Route53 shortly after the Dyn outage started as a means of getting back online. Now, they've removed their Dyn, and are now exclusively Route53. I don't think Dyn showed any incompetence; the parent-poster was merely remarking on relying entirely on a single provider, who, if they get DDoS'd, causes your site goes down. (There was some previous discussion about splitting between providers, but some commenters noted that it was difficult, or at least non-trivial, to replicate records between two providers.)
- MrMullen 10y agoThe problem is that you need to find a DNS provider that allows master and slave configurations of your DNS information. For example, Dyn can act as a master and UltraDNS can act as a slave, however, Route53, you can't be either. With Route53, you are all in. Lucky for Route53 users, Route53 DNS surface is really large and there is a really good chance that not even is attack could hurt it.
- js2 10y agoAXFR isn't the only way to sync records between providers. You just need a tool that speaks to the APIs of each provider and can sync between them that way. Heck, I had syncing in place at a startup between Route 53, DNS Made Easy, a pair of TinyDNS servers, and a git repo (which was our historical backup of changes) years ago. It was 300 lines of Python and 100 lines of shell. Albeit, we only had a few dozen or so records to manage, but this isn't rocket science. Aside: I came out of college as a sys admin with a CS degree and writing tools like this was par for the course. If devops folks aren't writing tools like this today, what are they doing?
- shanemhansen 10y agoHonestly: I think they are spending most of their time moving existing working infrastructure into containerized infrastructure and figuring out how to deploy their blog on k8s. They are working on learning libraries that abstract abstractions.
- BinaryIdiot 10y agoTo be fair Route 53 will split over other providers so you should be good in theory. But yeah if something more specifically targets Route 53 then that could be the same problem. At least that's my understanding anyway.
- bks 10y agoSo we now run a split view with DYN and AWS. My biggest issue with AWS is again they are a large attack surface, but also they don’t really play super nice with others and no DNSSec. We are currently evaluating the Netflix denominator tools to spread DNS and sync our alternate providers. My biggest problem during this outage was that I could not login to my registrar and make change to DNS directly - I had to login to DYN and ADD Route 53, it was impossible to remove DYN completely. And that's how we landed up with a split view. NOW if anyone can tell me of a competitor for the Traffic Director product that works on Port 25 I'll be happy to consider a migration. Cloudflare has something in the works, but I’d really just like a DNS provider with a virtual load balancer that can handle my 250qps at a reasonable price.
- SysArchitect 10y agoYou could build your own DNS set up for Traffic Director. Based upon the recursive server IP hitting you, send back responses that are closest to the user. There is an RFC to pass information about the client subnet: https://tools.ietf.org/html/draft-ietf-dnsop-edns-client-subnet-08 https://tools.ietf.org/html/draft-ietf-dnsop-edns-client-sub... Which Google DNS uses to tell your name servers more information about where the client is located. This allows you then direct them to the nearest server.
- nameless912 10y agoAlright, what the fuck? Shame on reddit, github, and netflix for learning literally fuck-all from this.
- kavok 10y agoI think Heroku is switching to two DNS companies after this as well. https://status.heroku.com/incidents/965 https://status.heroku.com/incidents/965 "This outage exposed a critical weakness in our DNS hosting configuration. We are taking immediate steps to add additional DNS providers. This should allow us to avoid impact in the future, provided that at least one of our DNS providers is operational."
- blakesterz 10y ago* spotify.com: all internal nameservers now I don't know much about running nameservers but moving to all internally hosted seems like an odd choice to me, can anyone explain whey that's a good move?
- eggoa 10y agoSpotify clients can just go to the IP address?
- user5994461 10y agoDNS is a rather simple service that was always meant to be run internally. The question you should ask is why did these companies used an external DNS in the first place?
- i__believe 10y agoSo they don't waste cycles on something not part of their core business or competency? Pretty standard reasons to pay someone to solve a problem. I think what this really showed is Dyn was not as competent in mitigating as what people thought.
- deleted 10y ago[deleted]
- blakesterz 10y agoYeah, that's exactly why I asked. Seems like one of those things where it makes sense to me to outsource, but I don't really know if I'm right on that.
- user5994461 10y ago[I'll try to make it simple, ignoring edge cases and real world complexity] You can't outsource DNS. It's one of the critical piece of networking that must be in every infrastructure. The common DNS server is BIND. It's been there for 30 years, it's well known, well manageable and well understood. Sysadmins have to know it and manage it. It's especially critical for worldwide multi-site tech organizations. There is no need for anything else. BIND can do everything and is the most flexible. Some of the alternatives lack some or most of the features (e.g. some type of DNS records). You should assume that any organization is running it's own DNS servers. (ignore the edge cases). --- In practise for large scale operations, the DNS tree will get very complex. What the websites changed was only the public DNS server for reddit.com or airbnb.com. It's only the top of the iceberg. There is likely a very complex DNS setup underneath including public domains, private domains, special internal domains, CDN, per datacenter, per continent, etc... which could imply 10 different DNS services. Who serves the top level public domain is a details. We should assume that the companies put whatever they could in little time to fix the ongoing issue.
- gist 10y agoZone data that I have access to shows that they lost roughly 1500 domains on the 23rd and 250 on the 24th and 155 on the 25th. On previous 3 Saturdays they lost between 40 and 60 domains.
- arkadiyt 10y agoTwitter is making many changes including adding a secondary provider - the work is ongoing but should be out soon.
- pilom 10y agoWe use CloudFlare and after this my boss said "set us up with secondary DNS somewhere." Unfortunately, CloudFlare doesn't support being a primary DNS provider with NOTIFY messages. They are designed to handle the DDoS for us by proxying content. It's an interesting problem and I don't know whether to push back to CloudFlare or my boss. Anybody else running secondary DNS after this with CloudFlare?
- majke 10y ago(I work for CF) Indeed, if you want the HTTP/HTTPS traffic to go through Cloudflare, the DNS must go through Cloudflare. There are generally two ways to set it up: a) You move your DNS auth to Cloudflare and allow it to manage it. b) You keep managing your domain yourself, and CNAME to Cloudflare. See: https://support.cloudflare.com/hc/en-us/articles/200168706-How-do-I-do-CNAME-setup- https://support.cloudflare.com/hc/en-us/articles/200168706-H... What you should do depends on your setup and threat model. Do you fear DNS auth going down? Do you think your DNS will be a target? Do you use Cloudflare to hide your HTTP origin IP addresses? For example, if you fear DNS auth going down, but you must use Cloudflare for HTTPS (say: for caching and SSL certs), then changing DNS off CF makes little sense. You already assume stability by expecting it to work HTTP layer. If you think you can be a target of DNS attack, I'd say having multiple auth is unlikely to give you more mileage. If you can afford disabling CF on HTTP layer, exposing your HTTP origin IP and want to have two different DNS auth providers, fine, you can do CNAME. But then you have three vendors to worry about, and problems with each can lead to trouble.
- song 10y agoBy the way, slightly out of topic but I was very frustrated with a Cloudflare sales guy who reached out to my customer during the outage and told him that we should switch to Cloudflare to be protected from DDOS. It comes a bit as gloating in the face of the attack on Dyn and there's no reason to believe that Cloudflare's DNS would fare any better.
- dx034 10y agoFrom the numbers that were published, it seems that Cloudflare would've probably handled the attack without outages. They have significantly more PoPs, especially in the regions that were attacked (Dyn has 2 in US-East and 8 in US, Cloudflare has 6 US-East and ~20 in US overall). I think it's unlikely that an attack of 1-2Tbps would've brought them down. Answering DNS is not very costly, so if you have enough capacity to the servers, answering shouldn't be the bottleneck. I agree that it's very bold to do that, but I'd trust them with handling DDOS more than most other providers.
- fred256 10y agoMinor correction w.r.t. Netflix: netflix.com itself is on Route53 but parts of the CDN appear to still be 100% on Dyn: $ dig +short ns nflxvideo.net ns1.p19.dynect.net. ns4.p19.dynect.net. ns2.p19.dynect.net. ns3.p19.dynect.net.
- song 10y agoI'm in the process of looking for a secondary DNS server for a client but because they rely heavily on geolocation load balancing it's not simple... I wonder if anyone has other recommendation beside UltraDNS for a good slave?
- dx034 10y agoWhat happens if you have a DDOS on Route53? I'm sure they can handle the attack, but do you have to pay for the requests? Or are there clauses that they drop the fees if the requests were malicious? If not, the financial risk could easily outweigh the benefits of availability for smaller companies.
- mnx 10y agoOT: Is my eyesight going away, or is the font unreadably light-coloured?
- cbg0 10y agoNo, the font weight and color are horrible.
- ben_jones 10y agoMy company only just noticed our font renders terribly for one of our newer web apps - all the developers had Macs and didn't notice an egregious difference when rendered on lower quality displays. We changed it immediately. I feel empathy for this particular development oversight among small teams.
- aurenen 10y agoYeah, a content font shouldn't be at 300. It's way too thin at that font size.
- rpeden 10y agoThe color is #555, so it's not terrible. I think the font-weight of 300 is the bigger culprit here. If you pop open the Chrome dev tools and change font-weight to normal, everything becomes much easier to read.
- rabboRubble 10y agoNot terrible. Just mostly terrible. For Chrome there is a "High Contrast" add-in to normalize shiat font color selection.
- heywire 10y agoAlso OT: Anyone else think the Dyn logo is reminiscent of the old Symantec logo? I actually had to go look to see if there was a relationship there (also discovering that the logo was considerably different than I remembered).
- rasz_pl 10y agoHow exactly was the attack mitigated when in the same sentence they confess it STOPPED _on its own_.
- tlunter 10y agomit·i·gate ˈmidəˌɡāt/ verb make less severe, serious, or painful Don't think they said they solved the problem. They just made it less bad. Some DNS queries got through, but not all.
- dsp1234 10y agoMitigated means "make less severe, serious, or painful.". In other words, they took steps that stopped the attack from effecting legitimate traffic. Once the attacker sees that the attack is no longer effective, the attack stops. After all, it's a denial of service attack. If the service is no longer being denied, then what is the point of continuing the attack.
- hueving 10y agoTo cost the company significant amounts of money. Bandwidth in those volumes isn't cheap.
- subie 10y agoWouldn't it be costing the attacker some amount of money running the C&C server? I'd think at some point if the attacker isn't getting the same results from the attack he'll wait for them to scale back then strike again. Is my thinking right here?
- tomschlick 10y ago> Wouldn't it be costing the attacker some amount of money running the C&C server? Not necessarily. They could configure the C&C to run off of something like pastebin/gist where they just input a list of IPs to attack and the botnet checks a specific url every X minutes for new instructions.
- youeeeeeediot 10y agoDyn aside, xTbps DDOS is the new norm. Expect more of this in the near future, single-source infrastructure is becoming a huge liability (not that it wasn't before). I wonder what impact on SLAs it will have when cloud services providers are taken down - will they honor their SLAs or inject DDOS clauses into them to shield themselves. You won't see many standing up to multi-Tbps attacks, at least for the moment.
- dkopi 10y agoNot sure DDOS clauses are that unreasonable. Especially when attackers might be targeting your DNS server, even though you're the target.
- ben_jones 10y agoDoesn't that also provide a huge incentive for cloud offerings to sell DDOS-resistance products and services? Isn't that a huge market already with gigantic margins?
- SysArchitect 10y agoOVH throws their DDoS mitigation system in for free when you host with them.
- raarts 10y agoHow does their system work and how effective is it?
- Thaxll 10y agoIt's built on FPGA and it's very effective. https://www.ovh.com/ca/en/anti-ddos/ https://www.ovh.com/ca/en/anti-ddos/
- dogma1138 10y agoGame servers that are hosted on OVH get DDOSed daily and they drop like flies. Myself and a lot of friends had servers killed both when renting VPS/Dedicated server or a dedicated "Game Server". And all and all with considerably smaller botnets like the ones you rent for a few $ per hour. If you are running a public server you learn quite quickly that if you permaban a cheater or just some annoying kid you should expect to be DDoSed these days.
- bradleyjg 10y agoOne thing I don't understand about this attack: Virtually everyone is behind NAT these days, often multiple layers of NAT. So how does the botnet manage to telnet or ssh into these set top boxes or lightbulbs or whatever? When I want to SSH into my home computer I have to go through elaborate maneuvers to get it to work.
- seany 10y agoRouters by default shipping with permissive upnp settings?
- sjnair96 10y agoSo I may be flat out wrong about this, but I believe the clients, once infected phone home, thus (mostly) removing the need for worrying about NAT, as they router will forward the Server's connections to the correct machine once it sees that a prior outgoing connection has been made.
- hexane360 10y agoI think the question is about the initial infection.
- jackweirdy 10y agoI believe UPNP is used commonly by many of the IoT devices that end up in botnets.
- bradleyjg 10y agoAs I understand it UPNP allows a device to negotiate a port map with a NAT box. But the developer of the IoT device would have to specifically want to map port 22 or 23. It's not something that would happen automatically. So are you saying that these IoT device makers not only hard-coded root usernames and passwords into their devices, but deliberately set up UPnP mappings to those ports? That looks malicious rather than negligent. Am I missing something?
- eternalban 10y agoI've read a few comments elsewhere that the attack on 21st may have been an element of a state actor MITM attack. What's the expert opinion? [p.s.] The question is precisely this: is it possible that a DDoS attack on DNS can be used to affect/mask a MITM attack.
- user5994461 10y agoA new article got just published to say that it was done by script kiddies from hackforums
- takeda 10y ago"State actor" today is an abused term, because it helps the victim look less bad, the government will back it up, because it paves way for new regulations and there's no way to prove one way or the other. Especially when it comes to DDoS.
- papabearshoe 10y agoThis is insane how big these things are now: "There have been some reports of a magnitude in the 1.2 Tbps range; at this time we are unable to verify that claim."
- e12e 10y agoThey mention 100k participating devices/IPs - that would mean an average of 12 Mbps upload. Sounds high, but plausible? [ed: I actually think it's kind of sad that most users aren't on symmetric gigabit links yet... But in this context I guess it's a blessing of sorts... ]
- gamache 10y agoIn the case of a DNS amplification attack (like this one), a malicious host can cause much more traffic than it can itself supply.
- bogomipz 10y ago"They mention 100k participating devices/IPs - that would mean an average of 12 Mbps upload" I doesn't have to be. An attack can be highly asymmetric such amplification/reflection attack. I am not saying this is what happened in the Dyn case but rather addressing your average comment. Just an example having hosts in the botnet send queries to non-Dyn open resolvers on the internet and in those queries they spoof the source IPs of Dyn DNS servers. Now all of those open resolvers on internet start sending return traffic to Dyn. If in those spoof request they request EDNS0 those responses could be up to 4K. Compare that to the size of query packet and you have a lot leverage.
- meowface 10y agoWe already know Mirai has been able to reach over 1 Tbps, and we know Mirai was at least one of the cannons hitting Dyn. So 1.2 Tbps is definitely plausible. Mirai has decreased in size to a degree due to more public awareness, but it's still massive.
- ryanlol 10y ago
- snowy 10y agoCan any one explain why they keep referring to this as a complex attack? From the article it seems to be a simple volumetric attack. They mention that it uses UDP port and TCP port 53, nothing complex about that... Am I missing something here. It wasn't an L7 attack (or was it?) Why keep referring to it as complex?
- marmot777 10y agoThere are two different things. Please correct me if I'm wrong on this: 1. Device backdoor open Port 23 (telnet), used to take over loT devices. 2. The loT devices attacked through Port 53 (DNS).
- colonelxc 10y agoMuch of the complexity is likely in building and managing such a large botnet. There also isn't a lot of details here on the exact nature of the traffic. They say it was hard to distinguish between legitimate traffic and this malicious traffic. So the botnet is at least rotating their requests through lists of customers hosted with them (though that isn't complex, but it is forward thinking. If the botnet was all making non-stop requests for just a few domains, that would be a strong signal to start filtering traffic, first internally, then pushing ISPs to block it upstream).
- chjohasbrouck 10y agoI think the market's expectation is that a DNS provider is prepared for a DDoS of any size, but not necessarily any level of complexity, so that's a lot of incentive to talk up the complexity of the attack. What's described in this incident report is totally within the capabilities of a single individual with public knowledge, though. If they could have proven otherwise, they probably would have (unless that somehow conflicted with their criminal investigation).
- rasz_pl 10y agoIts the alternative of "state actor haxored us". Marketing BS.
- bluedonuts 10y agoI wonder what 1.2Tb would do if pointed at a ELB or an AZ in AWS. Someone must have tried at some point but i've never heard of any widespread outages caused by DDOS on AWS. Is it just that bandwidth available to an AWS datacenter is that much bigger than what dyn have?
- bklyn11201 10y agoIt's safe to assume that 1.2Tbps isn't a big deal for Google/MSFT/AWS/Cloudflare/Akamai/Yahoo/Verizon/etc. DNS is normally a low-bandwidth protocol so if you only provide DNS services, needing to purchase 1000x your normal bandwidth to handle these bursts would be miserable. If a DNS provider were also providing video services (Vimeo/Twitch/etc), then a 1.2Tbps increase in traffic could be easily absorbed.
- rasz_pl 10y agoYou would think that, yet Akamai dropped Krebs over half that.
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- capkutay 10y agoCan anyone recommend a good place to learn more about this field of network security? I'd really appreciate!
- argio 10y agomaybe this is something for you -> https://cybersecuritybase.github.io/introduction/ https://cybersecuritybase.github.io/introduction/
- 1_2__3 10y agoNot smart of them to have a product guy write this. Even if he had it would have been better delivered (if shadow written) by someone in engineering. They're customers are not necessarily people who want to hear technical claims from Product.
- rasz_pl 10y agoThere is a reason behind this, engineering had nothing positive to say, even this announcement admits attack stopped when it stopped, there wasnt anything technical they did that fixed the situation, unless by mitigation they meant paying ransom and that was the cause attack stopped on its own.
- justinlardinois 10y ago> We began to see elevated bandwidth against our Managed DNS platform in the Asia Pacific, South America, Eastern Europe, and US-West regions that presented in a way typically associated with a DDoS attack. As we initiated our incident response protocols, the attack vector abruptly changed, honing in on our points of presence in the US-East region Interesting. Much of the reporting on the day of suggested that the attack was felt exclusively in the United States, but this says otherwise.
- dx034 10y agoSounds like someone found a way to target one server with all devices worldwide. Shouldn't that be impossible with Anycast? Or did they reveal an ip address that was just referring to US-EAST?
- R_haterade 10y agoAny word anywhere on motive? Or are we still left to speculate?