7 ms·

Update Regarding DDoS Event Against Dyn Managed DNS on Oct 21

71 points by sajal83 10y ago

deleted 10y ago
[deleted]
nucotano 10y ago
I'm so looking forward at IPv6, the death of NAT, and billions of IoT devices with all ports exposed to the world :-)
- skarap 10y ago
  A brave new world!
- heypete 10y ago
  NAT != stateful firewall.
- acomjean 10y ago
  Arent most IOT devices behind a router and thus unexposed directly to the internet (excepting routers)? This part of these attacks confuses me.
  - sajal83 10y ago
    Compromised routers can be used to compromise devices behind it. Also many devices (like IP cameras) usually have port forwarding to allow the users to access it from outside.
  - kijin 10y ago
    Many devices use UPNP to automatically punch a hole through the NAT and expose their ports to the world.
  - dendory 10y ago
    Pretty much every router is sold with UPnP turned on. Many IOT devices use UPnP to open their interface to the world.
  - tyingq 10y ago
    This is an interesting approach to get past NAT: https://thehackerblog.com/sonar-a-framework-for-scanning-and-exploiting-internal-hosts-with-a-webpage/ https://thehackerblog.com/sonar-a-framework-for-scanning-and...
- Yuioup 10y ago
  /s
- walk3r 10y ago
  Most CPE's running IPv6 will be following RFC 6092. Everything is blocked apart from ICMPv6 basicly.
  - nucotano 10y ago
    >Most CPE's running IPv6 will be following RFC 6092. It's pretty naive to think that any CPE will be following any kind of norm or rule.
ComodoHacker 10y ago
This attack looks like another probing into critical internet infrastructure Bruce Schneier had talked about. Who's next?
- pyvpx 10y ago
  because a number of high profile sites decided to put all their DNS eggs in a single, vulnerable basket does not make Dyn "critical internet infrastructure" nor this volumetric DDoS attack a "probe"
- kukx 10y ago
  You suggest it's probing. Then how would the full scale attack look like? Also, how does probing help the perpetrators, doesn't it lead to better defenses in the future?
  - Taek 10y ago
    These attacks seem to be getting sophisticated faster than defenses are being thrown up. It might lead to better defenses in 5 or 10 years, but the Internet at large is built out of infrastructure is difficult and slow to upgrade. And is everyone scrambling for a solution? Not really... Most are taking the mentality "well sucks to by Dyn but it's not affecting me so I don't need to respond". Especially because the vulnerabilities being exploited right now seem to by systemic. It's not like patching a zero day. "Uh... we've got 45,000,000 un-upgradeable IOT devices from dozens of different manufacturers executing a DoS attack". You can't fix that the same way you fix a privilege escalation bug in the linux kernel. If the attackers end up finding the Internet's equivalent to a jugular, there might not be much we can do about it. BGP isn't going to be replaced in the next 10 years, it's here whether we like it or not. DNS is also not going to be replaced in the next 10 years. And neither are the major centralized internet exchange points. Any vulnerabilities an attacker can find related to the fundamental design of those things are going to remain vulnerabilities for many years. If the attacker can get good at exploiting them, we are in trouble.
- vonklaus 10y ago
  Not sure what the disagreement is, seems logical. The attack knocked out service and presumably attackers recorded the failure threshold and mitigation attempt. Then recalibrated again to confirm/get more data and pushed another high vol attack.
TekMol 10y ago
I am sure the DDoS problem is something that the free market will sort out. The individual players will make it costly for the other players to send problems their way. I expect a chain of "charge the next node for resource usage" to evolve. If this chain will go all the way to the end user, I don't know. If it will, then end users will probably start using routers that feature restrictions / monitoring / control of outbound traffic. So you know your coffee machine will not be able to use up too much resources. Just like you have a fuse for the power line.
- Kenji 10y ago
  I am not sure why this is being downvoted. I think it's an interesting perspective. After all, it's in pretty much every internet user's and provider's interest to have a working internet.
  - tyingq 10y ago
    I think it's being downvoted because it's not a great strategy to address a distributed denial of service. In a DDOS, you have a high number of attack sources, none of them using a great amount of bandwidth on their own. The bandwidth use isn't notable until you get pretty close to the destination. Thus, the costs are (mostly) borne by the victim. Basically it doesn't put the pain where it needs to go.
    - rlpb 10y ago
      If the transaction cost can become insignificantly low, then money could flow in the direction of the victim. Each attack source could end up making a small contribution to the victim's bandwidth bill. I'm not sure economic incentives will lead to this happening. There's also the difficulty in attributing traffic to the person who requested it ("is this a request, so we should bill the packet source, or is this a reply, so we should bill the packet destination?"). But in theory, I don't see the DDoS pattern (a high number of attack sources) making any difference to this proposal.
      - 10y ago
nik736 10y ago
Is there any info on what volume (how much bandwidth) the DDoS attacks were?
- vonklaus 10y ago
  Curious as well. I didnt see any in the status here nor the original.
honksillet 10y ago
So is it retaliation for cutting Assange's internet, retaliation for threatening Russia with a cyber attack or both?
- leephillips 10y ago
  https://twitter.com/wikileaks/status/789574436219449345 https://twitter.com/wikileaks/status/789574436219449345
mjec 10y ago
Pardon my ignorance, but why don't companies run their own nameservers? I get why you don't want to run email - it's highly reputation driven. But as far as I can tell, running nameservers is no harder than running webservers or DB servers. HA is potentially even easier, because the system was designed that way from day zero. I'm not suggesting I'd run one for my personal website, but twitter and github are already managing distributed networks for this. What are the services Dyn and others provide that are so invaluable?
- ropiku 10y ago
  It's not that hard but DynDNS can offer a much higher performance, reliable and advanced service. They use anycast with a lot more servers than it's practical for each company to manage. They also offer advanced georouting, failover, etc. In the face of a DDoS I'm not sure a custom nameserver network would do much better than a company who does that for a living. The only advantage is that attacks would have to target individual services (which did happen other times).
- Yodoshi 10y ago
  Latency, Ops, Cost, specialised features like latency based routing (nearest datacentre to the user making a request). It comes down to the same reasons as someone using the cloud or a cdn, why spend more running it yourself (staff, equipment etc) instead getting someone who's who job it is to run that specific piece of software to the absolute best of their ability. It's just not a core competency of almost all companies.
- deleted 10y ago
  [deleted]
- majke 10y ago
  The complex DNS products exist for a reason. For one they can do really good geo-routing. This makes your services go faster for a global audience. Then, some of the big companies use multiple CDN's. You might want to use one CDN provider in Asia and another in Europe. Furthermore, you may want to select CDN not only on geo-routing dimension, but on arbitrary criteria. Imagine that you had a fixed budget for, say, cloufront, and wanted to route to them as much as you could, but never exceed your budget. Modern DNS services allow all this complex of scenarios. Furthermore, running your own DNS infrastructure is far from obvious these days. In May 2015 I gave this talk on defending DNS from DDoS: https://idea.popcount.org/2015-05-11-oarc---dealing-with-dns-packet-floods/ https://idea.popcount.org/2015-05-11-oarc---dealing-with-dns... Draw your own conclusions, but I'd say that running your own DNS makes you _more_ exposed to DDoS and extortion than using someone else's DNS infrastructure.
DyslexicAtheist 10y ago
you can watch BGP routes changing (as we speak) here https://stat.ripe.net/widget/bgplay#w.resource=208.78.70.16 https://stat.ripe.net/widget/bgplay#w.resource=208.78.70.16
amq 10y ago
Did Dyn mitigate the attack, or did it stop by itself?
zer0gravity 10y ago
So I'm wondering, what are the implications of this ? Someone is controlling a powerful enough botnet to do this. How powerfull is it really ? What else can it do ? Was this just a message or a test ? Or both ? What would happen if they would point it to google's nameservers ? What should we expect next ?
- tyingq 10y ago
  From this krebsonsecurity.com post: https://krebsonsecurity.com/2016/10/source-code-for-iot-botnet-mirai-released/ https://krebsonsecurity.com/2016/10/source-code-for-iot-botn... According to research from security firm Level3 Communications, the Bashlight botnet currently currently is responsible for enslaving nearly a million IoT devices and is in direct competition with botnets based on Mirai. “Both [are] going after the same IoT device exposure and, in a lot of cases, the same devices,” So, roughly 1 million IoT devices. And the source code for Mirai is freely available, so there's a DIY kit for anyone to use.
- neom 10y ago
  It's complicated because the internet is a complicated(and I'm no expert)- but, it's not good, but it's not insurmountable. The big issues today are: lots of network attached compute, lots of types of traffic, highly highly distributed network. There are numerous ways of mitigating DDoS, today a lot of it is via BGP route announcement[1] - although we've seen folks using BGP in questionable ways to mitigate DDoS recently[2]. As more and more of the internet becomes software defined (like SD WAN sees large global rollout)[3] more and more granular but non-disruptive control will be enabled. To answer your question about google, their DNS is probably resilient enough to deal with a pretty huge attack but who knows really -- In my dream world we get some good neural networks built and deployed to the edge watching for unusual traffic patterns and disrupt them. It used to be biggest pipes win, I don't know that will continue to exclusively be the case. [1] http://www.enterprisenetworkingplanet.com/netsp/article.php/3615896/Networking-101-Understanding-BGP-Routing.htm http://www.enterprisenetworkingplanet.com/netsp/article.php/... [2] https://www.youtube.com/watch?v=LFJzu0AFDpU https://www.youtube.com/watch?v=LFJzu0AFDpU (Dyn Engineer gave the talk) [3 ]http://www.rcrwireless.com/20160408/telecom-software/using-sd-wan-combat-ddos-aggressive-attacks-tag2 http://www.rcrwireless.com/20160408/telecom-software/using-s...
djhworld 10y ago
I think it's about time they updated http://dyn.com/ddos/ http://dyn.com/ddos/ It was an unprecedented attack, sure, but I'm not sure how their sales guys are going to spin this.
peter303 10y ago
The movie Fantasia Sorcerers Apprentice was right! All the the household objects will rise and overthrow their masters!
philiphodgen 10y ago
The free market will solve this problem with an existing and efficient tool. Tort law. A few wins in court will do the trick. Here is how. A victim of a DDoS attack sues manufacturers, distributors, and retailers of that product for selling a defective product--the IoT device used in the IoT attack. As soon as there is a win, the product disappears. Distributors and retailers must now price in the externality of the risk created by the product. It is far easier to take the product off the shelf. Therefore the manufacturer must either make a better product or die. We don't need government regulation. We just need time. The legal system will do what it is designed to do: assign economic consequences to the right parties.
- deleted 10y ago
  [deleted]
- natch 10y ago
  You say this as if it is just one device used in these attacks. And as if it will be easy to stroll over to the owner's location and determine the supply chain of that one device. And as if it will be easy to collect from the Chinese manufacturer who probably folded last week and reopened under a different name for completely different reasons.
  - philiphodgen 10y ago
    This is why tort lawyers sue distributors and retailers in the USA. They are here and they have insurance. Once there is a court decision, a rational seller (oh, Amazon hypothetically) will understand that selling fly-by-night small manufacturer items is fraught with peril. The offending items disappear from the marketplace.
    - AnthonyMouse 10y ago
      More likely the marketplace itself moves overseas. People use Amazon because they sell the stuff they want. If they stop selling it, the buyers go somewhere else. Amazon is a website. It could as easily be a website hosted out of China. Also, it sounds like you don't mind if the effect of your proposal is to destroy things like Etsy. And eBay.
linsomniac 10y ago
DNS seems ripe for revolution! Yesterday it felt obvious that we are treating DNS data as too ephemeral. I am not intimately familiar with the implementations in BIND and others, but it seems like when we hit the TTL we just throw away the data. Usually, that works fine. But, in the case of the origin servers not responding, yesterday I was wishing that it would just give me back the stale data rather than giving me nothing. I'll admit the impact on me was somewhat limited. Around 10am Mountain I was trying to install some Atom.io modules and couldn't reach that site or a github download URL. I had some success with using 4.2.2.2 (8.8.8.8 was not answering the names). Using a stale cached result probably wouldn't have helped for atom.io though, I hadn't been there in a while and this was querying my own local name servers. Do I want my name servers keeping weeks old stale data around? Probably not in RAM, but saving old names to disk sounds like it would require a lot of IOPS for a big provider. But I do know I'd visited the sites I was trying to hit within the last few weeks, since I couldn't reach the authoritative servers it'd be nice to have tried the last IP I had for them. Of course, I ran into this about an hour after I rebooted my entire dev/staging infrastructure to fix the Linux kernel privilege escalation issue, so my caches were cold. Sure would be nice if my server could "ask around" if it can't talk to an authoritative server. "Hey Google, hey Comcast, hey Level-3, do you know this name?" That's effectively what I did by changing my resolv.conf. But if you start asking around too widely, you ideally probably want to have some signature to verify the data you are getting. Seems like the new norm might be listing authoritative DNS servers from multiple big providers (Dyn and Route53) and having to keep them in sync? Then you lose some of the advance features... Funny aside: One of the sites I run uses Distil in front of it to protect against content scrapers. Months ago I was working with their support about getting a health checker set up in Route53 to test the full paper path through distil and fail over to our backup site if anything on the primary paper path didn't work. Distil assured me that their services were so resilient that we shouldn't worry about them being down. Guess what the only part of our infrastructure was that was impacted by this? :-)
elcct 10y ago
Did they mitigate the attack or attack has stopped? I am asking, because update is lacking the details about said mitigation.

71 points by sajal83 10y ago

deleted 10y ago

[deleted]

nucotano 10y ago

I'm so looking forward at IPv6, the death of NAT, and billions of IoT devices with all ports exposed to the world :-)

skarap 10y ago
A brave new world!
heypete 10y ago
NAT != stateful firewall.
acomjean 10y ago
Arent most IOT devices behind a router and thus unexposed directly to the internet (excepting routers)? This part of these attacks confuses me.
- sajal83 10y ago
  Compromised routers can be used to compromise devices behind it. Also many devices (like IP cameras) usually have port forwarding to allow the users to access it from outside.
- kijin 10y ago
  Many devices use UPNP to automatically punch a hole through the NAT and expose their ports to the world.
- dendory 10y ago
  Pretty much every router is sold with UPnP turned on. Many IOT devices use UPnP to open their interface to the world.
- tyingq 10y ago
  This is an interesting approach to get past NAT: https://thehackerblog.com/sonar-a-framework-for-scanning-and-exploiting-internal-hosts-with-a-webpage/ https://thehackerblog.com/sonar-a-framework-for-scanning-and...
Yuioup 10y ago
/s
walk3r 10y ago
Most CPE's running IPv6 will be following RFC 6092. Everything is blocked apart from ICMPv6 basicly.
- nucotano 10y ago
  >Most CPE's running IPv6 will be following RFC 6092. It's pretty naive to think that any CPE will be following any kind of norm or rule.

ComodoHacker 10y ago

This attack looks like another probing into critical internet infrastructure Bruce Schneier had talked about. Who's next?

pyvpx 10y ago
because a number of high profile sites decided to put all their DNS eggs in a single, vulnerable basket does not make Dyn "critical internet infrastructure" nor this volumetric DDoS attack a "probe"
kukx 10y ago
You suggest it's probing. Then how would the full scale attack look like? Also, how does probing help the perpetrators, doesn't it lead to better defenses in the future?
- Taek 10y ago
  These attacks seem to be getting sophisticated faster than defenses are being thrown up. It might lead to better defenses in 5 or 10 years, but the Internet at large is built out of infrastructure is difficult and slow to upgrade. And is everyone scrambling for a solution? Not really... Most are taking the mentality "well sucks to by Dyn but it's not affecting me so I don't need to respond". Especially because the vulnerabilities being exploited right now seem to by systemic. It's not like patching a zero day. "Uh... we've got 45,000,000 un-upgradeable IOT devices from dozens of different manufacturers executing a DoS attack". You can't fix that the same way you fix a privilege escalation bug in the linux kernel. If the attackers end up finding the Internet's equivalent to a jugular, there might not be much we can do about it. BGP isn't going to be replaced in the next 10 years, it's here whether we like it or not. DNS is also not going to be replaced in the next 10 years. And neither are the major centralized internet exchange points. Any vulnerabilities an attacker can find related to the fundamental design of those things are going to remain vulnerabilities for many years. If the attacker can get good at exploiting them, we are in trouble.
vonklaus 10y ago
Not sure what the disagreement is, seems logical. The attack knocked out service and presumably attackers recorded the failure threshold and mitigation attempt. Then recalibrated again to confirm/get more data and pushed another high vol attack.

TekMol 10y ago

I am sure the DDoS problem is something that the free market will sort out. The individual players will make it costly for the other players to send problems their way. I expect a chain of "charge the next node for resource usage" to evolve. If this chain will go all the way to the end user, I don't know. If it will, then end users will probably start using routers that feature restrictions / monitoring / control of outbound traffic. So you know your coffee machine will not be able to use up too much resources. Just like you have a fuse for the power line.

Kenji 10y ago
I am not sure why this is being downvoted. I think it's an interesting perspective. After all, it's in pretty much every internet user's and provider's interest to have a working internet.
- tyingq 10y ago
  I think it's being downvoted because it's not a great strategy to address a distributed denial of service. In a DDOS, you have a high number of attack sources, none of them using a great amount of bandwidth on their own. The bandwidth use isn't notable until you get pretty close to the destination. Thus, the costs are (mostly) borne by the victim. Basically it doesn't put the pain where it needs to go.
  - rlpb 10y ago
    If the transaction cost can become insignificantly low, then money could flow in the direction of the victim. Each attack source could end up making a small contribution to the victim's bandwidth bill. I'm not sure economic incentives will lead to this happening. There's also the difficulty in attributing traffic to the person who requested it ("is this a request, so we should bill the packet source, or is this a reply, so we should bill the packet destination?"). But in theory, I don't see the DDoS pattern (a high number of attack sources) making any difference to this proposal.
    - 10y ago

nik736 10y ago

Is there any info on what volume (how much bandwidth) the DDoS attacks were?

vonklaus 10y ago
Curious as well. I didnt see any in the status here nor the original.

honksillet 10y ago

So is it retaliation for cutting Assange's internet, retaliation for threatening Russia with a cyber attack or both?

leephillips 10y ago
https://twitter.com/wikileaks/status/789574436219449345 https://twitter.com/wikileaks/status/789574436219449345

mjec 10y ago

Pardon my ignorance, but why don't companies run their own nameservers? I get why you don't want to run email - it's highly reputation driven. But as far as I can tell, running nameservers is no harder than running webservers or DB servers. HA is potentially even easier, because the system was designed that way from day zero. I'm not suggesting I'd run one for my personal website, but twitter and github are already managing distributed networks for this. What are the services Dyn and others provide that are so invaluable?

ropiku 10y ago
It's not that hard but DynDNS can offer a much higher performance, reliable and advanced service. They use anycast with a lot more servers than it's practical for each company to manage. They also offer advanced georouting, failover, etc. In the face of a DDoS I'm not sure a custom nameserver network would do much better than a company who does that for a living. The only advantage is that attacks would have to target individual services (which did happen other times).
Yodoshi 10y ago
Latency, Ops, Cost, specialised features like latency based routing (nearest datacentre to the user making a request). It comes down to the same reasons as someone using the cloud or a cdn, why spend more running it yourself (staff, equipment etc) instead getting someone who's who job it is to run that specific piece of software to the absolute best of their ability. It's just not a core competency of almost all companies.
deleted 10y ago
[deleted]
majke 10y ago
The complex DNS products exist for a reason. For one they can do really good geo-routing. This makes your services go faster for a global audience. Then, some of the big companies use multiple CDN's. You might want to use one CDN provider in Asia and another in Europe. Furthermore, you may want to select CDN not only on geo-routing dimension, but on arbitrary criteria. Imagine that you had a fixed budget for, say, cloufront, and wanted to route to them as much as you could, but never exceed your budget. Modern DNS services allow all this complex of scenarios. Furthermore, running your own DNS infrastructure is far from obvious these days. In May 2015 I gave this talk on defending DNS from DDoS: https://idea.popcount.org/2015-05-11-oarc---dealing-with-dns-packet-floods/ https://idea.popcount.org/2015-05-11-oarc---dealing-with-dns... Draw your own conclusions, but I'd say that running your own DNS makes you _more_ exposed to DDoS and extortion than using someone else's DNS infrastructure.

DyslexicAtheist 10y ago

you can watch BGP routes changing (as we speak) here https://stat.ripe.net/widget/bgplay#w.resource=208.78.70.16 https://stat.ripe.net/widget/bgplay#w.resource=208.78.70.16

amq 10y ago

Did Dyn mitigate the attack, or did it stop by itself?

zer0gravity 10y ago

So I'm wondering, what are the implications of this ? Someone is controlling a powerful enough botnet to do this. How powerfull is it really ? What else can it do ? Was this just a message or a test ? Or both ? What would happen if they would point it to google's nameservers ? What should we expect next ?

tyingq 10y ago
From this krebsonsecurity.com post: https://krebsonsecurity.com/2016/10/source-code-for-iot-botnet-mirai-released/ https://krebsonsecurity.com/2016/10/source-code-for-iot-botn... According to research from security firm Level3 Communications, the Bashlight botnet currently currently is responsible for enslaving nearly a million IoT devices and is in direct competition with botnets based on Mirai. “Both [are] going after the same IoT device exposure and, in a lot of cases, the same devices,” So, roughly 1 million IoT devices. And the source code for Mirai is freely available, so there's a DIY kit for anyone to use.
neom 10y ago
It's complicated because the internet is a complicated(and I'm no expert)- but, it's not good, but it's not insurmountable. The big issues today are: lots of network attached compute, lots of types of traffic, highly highly distributed network. There are numerous ways of mitigating DDoS, today a lot of it is via BGP route announcement[1] - although we've seen folks using BGP in questionable ways to mitigate DDoS recently[2]. As more and more of the internet becomes software defined (like SD WAN sees large global rollout)[3] more and more granular but non-disruptive control will be enabled. To answer your question about google, their DNS is probably resilient enough to deal with a pretty huge attack but who knows really -- In my dream world we get some good neural networks built and deployed to the edge watching for unusual traffic patterns and disrupt them. It used to be biggest pipes win, I don't know that will continue to exclusively be the case. [1] http://www.enterprisenetworkingplanet.com/netsp/article.php/3615896/Networking-101-Understanding-BGP-Routing.htm http://www.enterprisenetworkingplanet.com/netsp/article.php/... [2] https://www.youtube.com/watch?v=LFJzu0AFDpU https://www.youtube.com/watch?v=LFJzu0AFDpU (Dyn Engineer gave the talk) [3 ]http://www.rcrwireless.com/20160408/telecom-software/using-sd-wan-combat-ddos-aggressive-attacks-tag2 http://www.rcrwireless.com/20160408/telecom-software/using-s...

djhworld 10y ago

I think it's about time they updated http://dyn.com/ddos/ http://dyn.com/ddos/ It was an unprecedented attack, sure, but I'm not sure how their sales guys are going to spin this.

peter303 10y ago

The movie Fantasia Sorcerers Apprentice was right! All the the household objects will rise and overthrow their masters!

philiphodgen 10y ago

The free market will solve this problem with an existing and efficient tool. Tort law. A few wins in court will do the trick. Here is how. A victim of a DDoS attack sues manufacturers, distributors, and retailers of that product for selling a defective product--the IoT device used in the IoT attack. As soon as there is a win, the product disappears. Distributors and retailers must now price in the externality of the risk created by the product. It is far easier to take the product off the shelf. Therefore the manufacturer must either make a better product or die. We don't need government regulation. We just need time. The legal system will do what it is designed to do: assign economic consequences to the right parties.

deleted 10y ago
[deleted]
natch 10y ago
You say this as if it is just one device used in these attacks. And as if it will be easy to stroll over to the owner's location and determine the supply chain of that one device. And as if it will be easy to collect from the Chinese manufacturer who probably folded last week and reopened under a different name for completely different reasons.
- philiphodgen 10y ago
  This is why tort lawyers sue distributors and retailers in the USA. They are here and they have insurance. Once there is a court decision, a rational seller (oh, Amazon hypothetically) will understand that selling fly-by-night small manufacturer items is fraught with peril. The offending items disappear from the marketplace.
  - AnthonyMouse 10y ago
    More likely the marketplace itself moves overseas. People use Amazon because they sell the stuff they want. If they stop selling it, the buyers go somewhere else. Amazon is a website. It could as easily be a website hosted out of China. Also, it sounds like you don't mind if the effect of your proposal is to destroy things like Etsy. And eBay.

linsomniac 10y ago

DNS seems ripe for revolution! Yesterday it felt obvious that we are treating DNS data as too ephemeral. I am not intimately familiar with the implementations in BIND and others, but it seems like when we hit the TTL we just throw away the data. Usually, that works fine. But, in the case of the origin servers not responding, yesterday I was wishing that it would just give me back the stale data rather than giving me nothing. I'll admit the impact on me was somewhat limited. Around 10am Mountain I was trying to install some Atom.io modules and couldn't reach that site or a github download URL. I had some success with using 4.2.2.2 (8.8.8.8 was not answering the names). Using a stale cached result probably wouldn't have helped for atom.io though, I hadn't been there in a while and this was querying my own local name servers. Do I want my name servers keeping weeks old stale data around? Probably not in RAM, but saving old names to disk sounds like it would require a lot of IOPS for a big provider. But I do know I'd visited the sites I was trying to hit within the last few weeks, since I couldn't reach the authoritative servers it'd be nice to have tried the last IP I had for them. Of course, I ran into this about an hour after I rebooted my entire dev/staging infrastructure to fix the Linux kernel privilege escalation issue, so my caches were cold. Sure would be nice if my server could "ask around" if it can't talk to an authoritative server. "Hey Google, hey Comcast, hey Level-3, do you know this name?" That's effectively what I did by changing my resolv.conf. But if you start asking around too widely, you ideally probably want to have some signature to verify the data you are getting. Seems like the new norm might be listing authoritative DNS servers from multiple big providers (Dyn and Route53) and having to keep them in sync? Then you lose some of the advance features... Funny aside: One of the sites I run uses Distil in front of it to protect against content scrapers. Months ago I was working with their support about getting a health checker set up in Route53 to test the full paper path through distil and fail over to our backup site if anything on the primary paper path didn't work. Distil assured me that their services were so resilient that we shouldn't worry about them being down. Guess what the only part of our infrastructure was that was impacted by this? :-)

elcct 10y ago

Did they mitigate the attack or attack has stopped? I am asking, because update is lacking the details about said mitigation.