15 ms·
Netflix is Down
- yoda_sl 14y agoThe net says it is mostly on the East coast but I am in California just a few miles from Netflix HQ and it is down for me on my iPad. I should try on the Apple TV since they mentioned that it is working on other devices. I will be curious to hear the post mortem of the outage from both AWS and Netflix. The timing I am sure doesn't help with Christmas Eve around the corner and probably many engineers not reachable quickly.
- yoda_sl 14y agoInteresting to see that the outage is in fact all over the place as the comments below are showing: http://www.isitdownrightnow.com/netflix.com.html http://www.isitdownrightnow.com/netflix.com.html
- protomyth 14y agoApple TV in St. Paul, MN failed so I don't think the east coast thing is true. Parents are not happy campers. I get the feeling all iOS devices are include.
- amartya916 14y agoIf Netflix can deactivate clients based on device size; that'd be pretty neat. People on tablets–Netflix on my Nexus 7 is out–would usually be watching stuff alone (kids may huddle around an iPad though) and for Christmas, a movie on a large screen is probably more important to keep going. Makes me think that apps should consider using Push notifications to tell consumers about issues; e.g. instead of a cryptic error number; say something like "Netflix servers are overloaded at the moment, NetFlix may not work on your mobile device". Not sure if that will piss off people more, but something like that instead of an error number would probably be better.
- bavidar 14y agoyou would think they would have learned to build redundancy into these types of systems.
- pixie_ 14y agoThey have been working hard on this very issue.. http://techblog.netflix.com/2012/07/chaos-monkey-released-into-wild.html http://techblog.netflix.com/2012/07/chaos-monkey-released-in... It'll be interesting to see what failed when they put out a statement.
- yoda_sl 14y agoUsually Netflix do well when AWS has an outage... At least in the past that was the case, so it really pick my curiosity on what is the difference with this outage vs the others. In addition it will be interesting to know if the timing of it -aka just before XMas- did affect the responsiveness of the engineers/team for both Amazon and Netflix.
- zrail 14y agoThe difference is that elastic load balancers are the things that actually implement the redundancy in AWS, and if one goes down and another can't be immediately started in it's place, no traffic will be getting to the instances that sit behind it. I believe AWS has been working on a solution to this, but thus far haven't released anything.
- pbiggar 14y agoInteresting that Heroku rode this one out. Any Heroku engineers want to tell us your LB setup? Do you use ELB?
- zrail 14y agoHeroku did not ride this one out. I have an app right now behind a failed ELB. https://status.heroku.com/incidents/479 https://status.heroku.com/incidents/479 Edit: Make that two apps.
- criswell 14y agoSomeone's Christmas Eve just went south. Godspeed.
- zrail 14y agoSeriously. Instead of getting angry at Amazon I'm going to wish them a happy Christmas Eve and hope their collective pagers go quiet soon.
- chuhnk 14y agohttp://status.aws.amazon.com/ http://status.aws.amazon.com/ Amazon CloudSearch (N. Virginia) - CloudSearch issues "6:00 PM PST We are continuing to investigate the issue. Domain creation and indexing operations continue to be unavailable. Changes to existing domains may have severe delays in being processed." Amazon Elastic Compute Cloud (N. Virginia) - Elastic Load Balancer issues "5:49 PM PST We continue to work on resolving issues with the Elastic Load Balancing Service in the US-EAST-1 region. Traffic for some ELBs are currently experiencing significant levels of traffic loss."
- jongold 14y agoI hope y'all have Home Alone on DVD…
- antidoh 14y agoAmazon movie rental over Roku is cutting in and out, and I can't watch Netflix (over Roku) at all. Colorado.
- jpdevereaux 14y agoHow is anyone supposed to enjoy the holidays with their families if they aren't consuming media?
- rhizome 14y ago1) some families like to watch stuff together 2) not all netflix subscribers have families
- jpdevereaux 14y agoFair enough. I for one enjoy expressing anti-television sentiments, with and without company.
- staunch 14y agohttp://www.theonion.com/articles/area-man-constantly-mentioning-he-doesnt-own-a-tel,429/ http://www.theonion.com/articles/area-man-constantly-mention...
- deleted 14y ago[deleted]
- rhizome 14y agoGood holidays to you, then.
- jpdevereaux 14y agoYou too! Sorry if my above absurdism comes off as offensive.
- marshray 14y agoWe're fresh out of chestnuts here, but thankfully Halo 4 matchmaking is online.
- smoyer 14y agoSoon it will be news when the N. Virginia services are actually up! I don't want to be too hard on Amazon because what they've built is pretty amazing, but I really have to wonder what's different about the N. Virginia site ... And why they're not having similar failures at the other sites.
- bgentry 14y agous-east-1 is much, much larger than the other regions. Many more services use it, which is why when it has issues you actually notice the effects.
- campnic 14y agoBased on some rough second hand estimation [1] it is an order of magnitude larger then all other AZs and is about as big as them all combined. [1]: https://huanliu.wordpress.com/2012/03/13/amazon-data-center-size/ https://huanliu.wordpress.com/2012/03/13/amazon-data-center-...
- lelandbatey 14y agoIf I remember correctly, the N. Virginia site is the default for EC2 instances, and is much much larger than the other availability zones. I've even heard that it's bigger than all the other zones combined. However, I'd love for someone to corroborate my statement.
- cperciva 14y agoThat's certainly true for the public IP spaces. It's possible that the other regions have a higher proportion of instances in VPC which don't have public IP addresses, or a higher proportion of larger instance types, though.
- arkem 14y agoA pedantic note: In the title "EVE" should capitalized as "Eve"
- sigzero 14y agoI just got finished watching something on Netflix?
- Raphael 14y agoYou don't seem to sure of yourself.
- toomuchtodo 14y agoYou'd think Netflix would learn by now to move back to their own gear. EDIT: Downvote away; its practically dogma on HN to use AWS. How much downtime are people willing to tolerate for a "superior" technology? Sure, Amazon AWS has some great ideas and tech, but you might as well give up if your business depends on EBS in us-east-1 at all
- jrockway 14y agoThe gear isn't the problem: the dependency on a single data center is the problem. It requires a lot of software engineering effort to maintain a service that works when a data center suddenly goes away. To be fault-tolerant, Netflix has to do this engineering regardless of whether or not they own the servers. But if they use Amazon, they don't have to actually fix the servers when they break, freeing up engineering resources to fix the software. (Why doesn't Amazon offer transparent replication? Because the price of replication is unbelievably high and most people can't believe how high it is. If you want to write to 12 replicas around the world, budget five seconds for your transaction to complete. Compare this to a MySQL database on an SSD that can do millions of writes on the same entity group in the same amount of time. This doesn't even include the cost: 12x the storage cost, and 12x the bandwidth cost for data going from your frontend to all of the backends.)
- moe 14y agoYou're blowing this a little out of proportion. Replicating a few petabytes of static videos is not rocket science nor cost prohibitive for a company the size of Netflix. Nor is engineering a system that can withstand a datacenter outage. Especially one as trivial as Netflix which is largely read-only. Thousands of systems of higher complexity are engineered to that standard, many of them are much larger than netflix. You rarely hear of Google outages, or iTunes, or Youtube, or [insert six dozen other popular brands], do you? Yes it can happen to the best of them, but the EC2 outages are really piling up lately.
- pixl97 14y agoYou rarely hear of Google outages http://techcrunch.com/2012/12/10/gmail-experiences-a-widespread-outage-most-users-affected/ http://techcrunch.com/2012/12/10/gmail-experiences-a-widespr... ,or iTunes http://appleinsider.com/articles/12/11/19/itunes-match-down-as-apples-icloud-issues-continue http://appleinsider.com/articles/12/11/19/itunes-match-down-... ,or Youtube http://abcnews.go.com/blogs/technology/2012/10/youtube-goes-down-not-googles-best-day/ http://abcnews.go.com/blogs/technology/2012/10/youtube-goes-... Everyone goes down, but EC2 has been buggy as hell from what I can tell. And before everyone says Netflix should `just` move or add more bandwidth... http://www.nbcnews.com/technology/technolog/netflix-uses-32-7-percent-internet-bandwidth-119517 http://www.nbcnews.com/technology/technolog/netflix-uses-32-... They represent a huuuge amount of bandwidth usage!
- 31reasons 14y agoWhat is the purpose of Xmas Eve if you spend it watching TV just like any other eve. Just saying.
- buzzkillr2 14y agoDoing normal things, but with family.
- dusing 14y agoA lot of people (including my family) watch christmas movies and gorge ourselves on food and drink.
- antidoh 14y agoFor many of us, watching a movie on TV is anything but just another evening. Everyone is different, and no one needs to justify their time to anyone.
- muyuu 14y agoAre they insured against a contingency of this calibre? Does Amazon provide/offer such an insurance?
- zrail 14y agoAmazon EC2 has a Service Level Agreement (SLA)[1] that guarantees 99.95% uptime in a rolling 365 day period, and provides for service credits. Of course, a huge customer like Netflix can and probably did negotiate their own SLA terms. [1]: http://aws.amazon.com/ec2-sla/ http://aws.amazon.com/ec2-sla/
- muyuu 14y agoI was wondering if they'd pay damages and how would that be calculated. I don't think free service for a while would cut it with a contingency of this sort. Maybe someone has first-hand information of their contract (or any other big player) and can answer to this publicly.
- moe 14y agoI don't know the Netflix contract but I've been involved with a few big (telco) SLAs. The penalties are usually calculated with a points-system and a multiplier that raises according to the duration/impact of an outage. E.g. the first 15 minutes of an outage may cost 1 point per minute, 15-60 minutes 2 points, and so on. You also have multipliers for the severity (partial or full outage, customer impact, affected countries, etc.), time-of-day, and so on. Collected points may then later be traded in for dollars or a nice lawsuit. Corporate lawyers love to go nuts on these things, an enterprise SLA can easily span a hundred pages of legalese.
- gurch101 14y agoHaving been in a similar position in the past, I actually feel bad for all the people at amazon and netflix that'll need to work late tonight...
- davidf18 14y agoClearly the work of "The Grinch who Stole Christmas...."
- cowkingdeluxe 14y agoNetflix is down because Amazon is down, so we had to watch shows on Amazon (prime) instead. Funny how that works!
- sjs382 14y agoI had the same experience. When watching movies with my SO tonight, I explained that Netflix uses AWS and that AWS was experiencing problems. We started watching a movie on Amazon prime and she asked "So I guess Amazon Video isn't using AWS?"
- antidoh 14y agoI tried to watch a movie rented from Amazon over Roku last night (not Prime) and it was unwatchable. It would get stuck on some scene for seconds to minutes, occasionally letting loose a bit of audio. The movie was Inception. Maybe there's some irony in there.
- toddh 14y agoThis really sucks for the engineers. Not a fun way to spend xmas at all.
- arkonaut 14y agoIf you're netflix, I'm sure you have staff/engineers at the office 24/7 during this time of year specifically. At least, I would hope they did.
- lucian303 14y agoCloud with a single point of failure. Merry XMas!