4 ms·
A Closer Look At The Christmas Eve Outage
- ChuckMcM 14y agoChaos Monkey fights back :-) I wonder if there is a way for the CM team to do ELB outages. At some point you entertain the idea that Amazon goes offline but probabilistically is that similar to a 9.0 magnitude quake and a 30M tsunami on the same day?
- lucian303 14y agoCorrect. Probabilistically, I'd say it's the same as long as that earthquake is under the ocean. Yeah you'll get 30M tsunami with a 9.0mag earthquake (most likely, I'm no geologist). So yeah, AWS going down should be something _every_ company that runs its services on AWS including providers like Heroku should take into account as far as their architecture goes. It's not a matter of if, it's a matter of when. You _will_ have downtime. Period.
- lucian303 14y ago"The Netflix Web site remained up throughout the incident, supporting sign up of new customers and streaming to Macs and PCs, although at times with higher latency and a likelihood of needing to retry. Over-all streaming playback via Macs and PCs was only slightly reduced from normal levels." This is simply false. I tried Netflix streaming on both my Macs and my Roku and neither worked. The site may have been up, but the streaming was down for Macs (and any computers in general I assume), not just TV boxes like Roku.
- saurik 14y agoI continued to be able to use Netflix through the problem from my Mac, however it would take multiple minutes for the stream to connect (it would get stuck at like 7%--or maybe it was actually 0%--while buffering, but with patience it would work).
- lucian303 14y agoYeah, I tried that but it didn't work for me.
- Steko 14y agoA small number of users unable to stream out of several hundred thousand might reasonably be described as "slightly reduced from normal levels".
- lucian303 14y agoTrue, but false claims in a post-mortem from Netflix make me wonder if I should retain the service. What they should be offering is credit to those affected. Their excuse that people don't watch too many streams on XMas eve is lame even by big company standards and makes no sense. My family was spending time together and we wanted to wind down with a movie. It was close to midnight (EST).
- pixie_ 14y agoI really don't like the excuse laden and sugar coated style of this postmortem.
- lucian303 14y agoSeriously. Especially after all the Chaos monkey articles. Their Chaos monkey doesn't have enough Chaos apparently or is too stupid to hit load balancers? Shit, that's the first thing I'd knock out!
- Soulsbane 14y agohttp://dvd.netflix.com/ http://dvd.netflix.com/ is down right now and has been every time I've checked today. "We're Sorry The Netflix site is temporarily unavailable. Our engineers are working hard to bring the site back up as quickly as possible." Of course it's not a priority over the streaming portion I'm sure for obvious reasons.
- Bill_Dimm 14y agoIt's up for me for at least a little while now, but I think it was down for me all afternoon (Eastern timezone). This supposedly only impacted "some" of the DVD customers. http://www.bloomberg.com/news/2012-12-31/netflix-says-some-customers-can-t-access-dvd-portion-of-website.html?cmpid=yhoo http://www.bloomberg.com/news/2012-12-31/netflix-says-some-c...
- lucian303 14y agoThis and their re-hauled (for awhile now) UI to a horrible mess that is almost impenetrable even by seasoned UI/UX developers is most likely due to a shortage of good engineers willing to go down to Los Gatos every day. I told them, one day a week, but big companies like this never learn till they're on the brink of collapse and it's too late. coughBlockbustercough
- jkat 14y agoCurrently in a battle with management about whether to launch on AWS or not. They want to, engineers don't. Engineers are largely driven by raw CPU performance. Management, well, they seem to be thinking "no one ever gets fired for picking..." Anyways, when you read all the effort Netflix has put into their cloud architecture (1) and the hiccups they (and others) have...I just don't know what hope our small team of 5 has of success. It seems like, to succeed on the cloud, you really need to build your app specifically for it (which we haven't done!) 1 - http://techblog.netflix.com/2010/12/5-lessons-weve-learned-using-aws.html http://techblog.netflix.com/2010/12/5-lessons-weve-learned-u...
- raverbashing 14y ago"engineers don't. Engineers are largely driven by raw CPU performance" There are several reasons to not use AWS. CPU is not one of them. Especially when you have the choice EC2 offers (though at a price) But unless you're doing something very, very CPU intensive (like doing heavy math like CFD, integer programming, etc) this is irrelevant. My bet is that you aren't.
- lucian303 14y agoCPU is a limiting factor for a lot of applications. And EC2's offerings are paltry until you get into the really expensive VMs. But it's not just CPU, it's the overall performance. Engineers want to run on hardware for many more reasons than CPU. And for most apps it's the rational choice. After all, you can build your own data center for the price you pay to rent some virtual machines. tl;dr: With cloud services you pay thousands a month for _renting software_.
- Negitivefrags 14y agoReally? I was just using aws to run some load testing bots against our game servers and found the CPUs even in High CPU instances pitiful for the price. The worst part about it is that the performance is so variable between instances. Some could comfortably run 1300 bots while others were slowing down so much after 750 that they started dropping bots. Because of that, it's easier to just run 750 on each which is wasting resources. My desktop PC can easily run 4000.
- plasma 14y agoWhy is CPU performance on EC2 so terrible compared to dedicated servers?