6 ms·
Amazon goes down; takes S3, Salesforce, Target, and others with them.
- Sam_Odio 17y agoIt looks like a DNS issue, since while http://amazon.com http://amazon.com isn't working for me http://72.21.207.65 http://72.21.207.65 is (kind of). EDIT: "No A records were found for amazon.com" http://www.zoneedit.com/lookup.html?host=amazon.com&type=A&server=&forward=Look+it+up http://www.zoneedit.com/lookup.html?host=amazon.com&type...
- ggrot 17y agoI wonder how much money amazon loses per minute 2 days before christmas. Ouch.
- alain94040 17y agoAt $20B annual, assuming Christmas is a big chunk, I'm guessing $2B (10%) in the last two weeks, 2 minutes would then be worth $200,000? I think my 10% number is too low, so maybe $500K for two minutes? But this is not profit, just revenue.
- wglb 17y agoDo you think it might catch up once it is available again? It is a pretty serious shopping destination, and possibly people retry.
- robryan 17y agoIt really depends on downtime, if it was more than say an hour a lot of people would probably buy elsewhere, small amount of downtime you wouldn't think would effect sales though.
- rlpb 17y agoI don't think that Amazon itself will lose much, since they're well-known enough that most people who fail to reach them will retry later. I think the small fry using S3 and EC2 will be the ones who are actually hit by this.
- cookiecaper 17y agoI don't know, I don't think it'd be as much as 4 - 7 days before Christmas. Most people know that it's too late to order from Amazon or any other online-only store by the night of Dec. 23. It's probably more money than they'd lose normally, but maybe not that significant? I have no data to back that up, it's pure speculation.
- aw3c2 17y agoNothing. But they probably don't earn much.
- justinsb 17y agoIt is DNS. If you put the EC2/S3 address into /etc/hosts, the services work fine. Affecting lots of other big websites as well apparently (target, salesforce) because they all outsource to UltraDNS
- shaddi 17y agoNANOG chatter confirming it is an issue with UltraDNS. Seems to be west coast related. EDIT: Potentially a DOS attack. From NANOG: "We have some DNS providing type customers (not UltraDNS) receiving a few million packets/sec of UDP/53 DoS traffic, starting at about the same time as the UltraDNS problems. No clue if it's related, but it certainly sounds suspicious. :)"
- aristus 17y agoI'm very surprised that they rely on a single vendor. But I guess DNS is one of those things you don't think about until it fails.
- metachor 17y agoThe point of using UltraDNS is that it they provide fast "real-time" failover of DNS routing. This is used, for example, for fault tolerance where the IP address responding to a domain name might change due to failure scenarios or load balancing (where a different server is now primary responder to the domain name, maybe located in a different data center or country). Infrastructure-wise, UltraDNS is kind of like the Akami of DNS, instead of content distribution.
- cperciva 17y agoAnd people told me I was crazy for hard-coding IP addresses for s3.amazonaws.com and sdb.amazonaws.com into the Tarsnap code... :-)
- justinsb 17y agoAwesome ... was that for security or for redundancy? Did you have a solution for what would happen if the IP address changed?
- cperciva 17y agoMostly security -- DNS is one of the worst offenders when it comes to protocols with security problems, both in terms of protocol issues and bugs in implementations. Amazon doesn't move endpoints very often, and I round-robin requests to several endpoints; so in the rare cases an AWS endpoint changes I see a slight decrease in Tarsnap performance and can make the Tarsnap server stop using the dead endpoint before any users are likely to notice.
- esja 17y agoHmmm... should I trust my backups to generic software vendor X, or to someone who clearly takes nothing for granted... tough choice. Signing up now.
- akl 17y agoYou are crazy for doing that - one instance of downtime doesn't justify ignoring all the advantages DNS brings.
- cperciva 17y agoWhen I made that design decision I wasn't considering the possibility of DNS outages at all; I was just thinking in terms of "there's a huge number of places between me and ultradns where someone could insert a spoofed DNS response".
- justinsb 17y agoI thought DNS was supposed to try backups servers automatically... any DNS experts able to explain what's going on? Some of the ultradns servers are returning (correct) values, others simply not responding.
- lsc 17y agoyeah, uh, if you are smart, you have a secondary DNS provider. But that really requires you managing it yourself. the problem was that many people outsource, which usually means going with only one provider. (now, ultradns does have a good setup, they probably aren't a bad choice for a provider, but having only one is just plain stupid.)
- tinio 17y agoIt looks to be back up now.
- notmyname 17y agosurely they mean "Amazon goes down and takes the Internet with it"
- slig 17y agoOnly if TC was hosted there.
- newhouseb 17y agoWhy wouldn't such a company run their own name servers? I understand it's "yet another thing to maintain," but I've set up bind before... didn't seem that bad.
- rbranson 17y agoIt's relatively trivial to outsource. It's one of those services that's easy to measure, quantify, and manage (from an outsourced perspective). There's also a bit more to it then that. The Anycast routing can be quite difficult to setup and maintain. It's virtually useless outside of a very small set of protocols (DNS being one of them), so it wouldn't make sense for Amazon to bring that kind of talent in-house for something like DNS.
- lsc 17y agothe mistake was outsourcing to only one provider. it's easy enough to setup a BIND slave elsewhere that automatically transfers the zone from your primary provider.
- kierank 17y agoThe Anycast routing can be quite difficult to setup and maintain. Anycast's not particularly difficult to setup and maintain.
- evgen 17y agoThis is one of those services that a dedicated provider can sometimes do better than internal IT. Ultradns, for example, has secure secondaries Colocated with large ISPs so you get some good protection against cache poisoning attacks. Everything they do you could do yourself, but it would cost you a lot more than what they charge. (full disclosure: I am a customer and until this evening I was quite happy with their service and reliability)
- jseifer 17y agoIt doesn't say in the TC article and I can't really tell from this thread. How long was Amazon actually down?
- boredguy8 17y agoNever let truth get in the way of a good headline.
- artagnon 17y agoHaha.. true. What else can we expect from Techcrunch?
- tybris 17y agoInternet can be so fragile.
- mattiss 17y agoWhen Rackspace goes down and takes TechCrunch with them, the whole INTERNET goes down. When Amazon goes down, Salesforce, Target, and others...