5 ms·
All this talk about redundancy, real-time apps, scalable architecture and and a "simple" DDOS against DNS architecture brings half of the internet down. Honest
by 6d6b73 10y ago
All this talk about redundancy, real-time apps, scalable architecture and and a "simple" DDOS against DNS architecture brings half of the internet down.
Honestly did nobody think about having a spare dns at some other company? or even backup dns server exactly for a situation like that?
- ljosa 10y agoThe TTL for the glue records of a .com domain is 48 hours, so even if you have Route53 set up and ready to go, it takes a long time to switch the zone away from Dyn. We switched from Dyn to Rout53 a few weeks ago. It took about 12 hours before half of the traffic had shifted over.
- 6d6b73 10y agoThat's the reason to have your DNS at at least two different companies, working in tandem. In a case where one is down, your Unicorn Corp doesn't go down with it.
- St-Clock 10y agoHow does that work in practice? Even if I set NS records pointing to two different DNS providers, I don't think a DNS client would automatically switch and retry if one is too slow to respond/times out.
- 6d6b73 10y agoWhy not? That's the whole idea behind having more than one NS server isn't it?
- acdha 10y agoMost DNS resolvers will automatically try each NS record until they get a response. That might be your ISP rather than your iPhone but that's an old practice going back to when the internet was even less reliable because some random Sun box under someone's desk failed. Modern web browsers will also do the same thing if a query returns multiple A records and they get a connection error.
- acdha 10y agoExactly: there's nothing wrong with only using one provider if you're not willing to pay for two services but if you can't afford downtime you really need active diversity all the way down. Route53 uses a bunch of different top-level domains for the same reason – if someone does manage to take the .com servers offline you'll be glad .co.uk is run by a separate organization.
- tuke 10y agoN.b., unless things have changed, I don't think Dyn (for one) allows secondary DNS. Maybe that has changed. {?}
- bhaak 10y agoFrom where I'm looking at the internet (central Europe), I don't notice anything. Maybe your internet on the other side of the Atlantic is broken, ours seems to be working fine. ;-) Edit: Looks like the eastern part of the USA is affected: https://cloudharmony.com/status-for-dyn https://cloudharmony.com/status-for-dyn
- tzaman 10y agoNot really, I can't access our production servers which are in US east. Can't access Intercom with which we provide customer support. Our clients are mailing us that payment provider doesn't work either. So we're losing money while being in central EU.
- a3n 10y ago> Maybe your internet on the other side of the Atlantic is broken, ours seems to be working fine. ;-) Works on my continent.
- 551199 10y agoI'm in Europe now and worked fine something like 30 min ago. Now facing similar issues reported in original post.
- ibjhbrown 10y agoThings are down here on the west coast too...
- pamvegas 10y agoI'm in Las Vegas and I haven't had access to several sites all day. I don't think this is limited to eastern U.S.
- planteen 10y agoYep same issue in Denver
- xXCrusherXxFTW 10y agoSame Im in Illinios and I can't play destiny which is made by bungie which is down for some fucking reason
- rotauserch 10y agothey need a round robin dns that is geographically dispersed
- mancerayder 10y agoIf ONLY they had 10X developers ... EDIT: joking aside, the issue with multiple DNS providers is primarily (in my experience at the company I'm at having investigated this in the past) intelligent DNS entries. Example, 'return these A records, in this order, based on the number of requests, roughly balanced'. There's no universal standard, just common aspects. DNS Provider A has one set of features, names for returning A records in some way, based on some weighted averaging, provider B has a different mechanism. So as an infrastructure person you have to: 1. Investigate provider A and B features for intelligent DNS (it's not even universally called intelligent DNS!), and mentally parse the commonalities and differences 2. Create a custom mechanism to keep them in sync internally. So you hope that A and B have an API for maintaining the records. 3. Ensure that when someone in your org wants to make an update, A and B update at the same time in the same way 4. APIs don't change.
- charles98102 10y ago+1
- dsl 10y agoIn my experience Dyn pushes this functionality very hard during customer on boarding, effectively as a lock in for their platform.
- mancerayder 10y agoWell I really think those features are useful, though. They let you get some powerful de-facto load balancing and HA and other complex 'features' without any new hardware or much complexity. They give you some easy creative freedom in designing a redundant infrastructure, give you some geo-aware DNS the likes of which I remember paying Akamai a lot of money for years go, etc. But the lock-in problem is absolutely right. I'm also familiar with the aggressiveness of that sales team. I prefer another provider and they were trying to solicit our business by specifically calling out our provider as amenable to a DDOS attack which had occurred. Someone with a simpler setup with standard DNS features will find it much easier to use multiple providers, of course.
- BlackGuyCoding 10y agoagreed 100%. This kind of thing shouldn't happen and this widespread. It's like no one is preparing for worse case scenarios until AFTER something tragic happens. On the West Coast and I just lost twitter/soundcloud/github - 9:40 am PST
- karmajunkie 10y agoloss of twitter, soundcloud, and github is what passes for tragic these days?
- exolymph 10y agoTry PayPal. That's gonna be a big issue for everyone who uses their payment processing.
- oxtoacart 10y agoAs a programmer, the loss of GitHub is a pretty big deal in terms of work productivity, and as a maintainer of production systems that rely on assets from GitHub, it's a pretty big availability issue.
- karmajunkie 10y agoI mean, i get it, its a big deal to us. But tragic? Syria is tragic. Github is just above annoying.
- waterhouse 10y agoLet's say that it's tragic for the people whose job it is to maintain those services.
- sphinx65 10y agoYeah, as if a "spare DNS" has anything to do with solving the problem. You could get into your own infrastructure, but your customers are still screwed.
- xtracto 10y agoMaybe it is time to widely adopt namecoin ?
- blazinDC 10y agoYes!! For sure, after such an attack it will force the internet into rethinking their DNS strategies. Why commit to having a single point of weakness? For those interested: https://namecoin.org/ https://namecoin.org/
- 123456789101112 10y ago..and they thought war happens only with guns and tanks. don't the know, digital is the new platform to kill each other?