6 ms·

It's not clear they learned the simplest and most import ant fact: You have to be able to migrate your production traffic away from whole regions.

dangrossman 14y ago
That's something they just started talking about last year, when everything was still in a single region. http://techblog.netflix.com/2011/04/lessons-netflix-learned-from-aws-outage.html http://techblog.netflix.com/2011/04/lessons-netflix-learned-... Though their post today makes it sound like they could have (and maybe did) fail over to an entirely different region, but their mechanism for doing so isn't automatic and took longer than expected.
- paulsutter 14y ago
  It's literally shocking to me that responsible adults would "just start talking" about basic resiliency, like failing over to a different region, for a major commercial service. There are ways to do it that take seconds (we do that at Quantcast using anycast), there are ways that take minutes (using DNS failover, which is readily available), and there are ways that basically take forever (Netflix guys - feel free to contact me). It's pretty clear they were aware of the problem, and if they had even the simplest and most basic DNS control, they could have moved people over minutes after realizing the problem.
  - maukdaddy 14y ago
    I'm sure it didn't literally shock you.
    - paulsutter 14y ago
      Lit·er·al·ly/ˈlitərəlē/ Adverb: 1. In a literal manner or sense; exactly: "the driver took it literally when asked to go straight over the traffic circle". 2. Used to acknowledge that something is not literally true but is used for emphasis or to express strong feeling.
      - sofal 14y ago
        Yes, the second definition was added because people lazily try to leverage the first definition to intensify the exaggeration of their figurative statements. The problem is that the second definition now masks the first (because it is essentially a devolved form of the first and actually relies on the first definition), making it difficult in many cases to get across the meaning of the first definition without resorting to other words. I see you've embraced this unfortunate evolution. Some of us are still mourning it.
        aw3c2 14y ago
        Language evolves and changes. Please do not bitch about that in unrelated discussions on HN.
    - deleted 14y ago
      [deleted]
    - dionidium 14y ago
      It's called hyperbole and it's perfectly acceptable usage and we all -- every single one of us! -- knew exactly what he meant and dear god am I tired of people using this smug and useless retort.
  - dangrossman 14y ago
    Didn't we just have a discussion of this type of comment today? [1] Being able to move /millions of requests per second/ (this was a Friday night!) on /thousands of servers/ [2] to a different /thousands of servers/ involves more than just routing requests to the right IP. Like having a separate set of /thousands of servers/ capable of handling that much failover, for one. Plus, if you had read the link, what they just started talking about a year after moving to AWS was moving across regions; they're already set up to fail entire availability zones over to other availability zones in the same region. Neither architecture at scale nor employee experience magically appears from nowhere. It's not even been two years since they moved from a data center to running the site on AWS. Considering how much they've had to learn, and the kinds of tools they've had to build for themselves to manage their kind of scale (read their tech blog!), I think the cheap insults are unwarranted. 1: http://news.ycombinator.com/item?id=4208134 http://news.ycombinator.com/item?id=4208134 2: http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html http://techblog.netflix.com/2012/02/fault-tolerance-in-high-...
    - paulsutter 14y ago
      My comment is far from a cheap insult. At Quantcast we handle 300,000 requests a second in 14 different cities, and we can failover between cities in 6 seconds. And we're doing a lot more than just log a pixel on each request, just to be clear. It makes maintenance and upgrades really easy. It was a bit of work, but we're always up. Even a child knows that when the power fails, you can't turn on the TV. This isn't specialized technical knowledge. EDIT: I'm not going to keep responding to your comment below. I'm certain that if I were involved in the design of Netflix' infrastructure, they would be able to survive problems that affect whole regions. (AND I DONT SEE THAT NETFLIX NEEDS 6 SECOND FAILOVER, THERE ARE MUCH EASIER WAYS TO DO IT IN MINUTES). EDIT: My repeated, emphatic comments are intended to serve a purpose. Everyone should be aware that this is a real problem and you need to plan for it, and it's pretty clear from the discussions on HN that people are surprised by the Amazon downtime. I personally think Amazon does a fantastic job as it is, and Amazon's reliability issues are not an excuse for the downtime of their customers.
      - breidh 14y ago
  - Xorlev 14y ago
    It's literally shocking to me that you'd be belligerent about it. That's not how adults solve problems. AWS doesn't support multicast or anycast, and the AWS EC2 control planes were so hosed it was impossible to recover in any meaningful way. Certainly, the fault was both AWS and Netflix, but both are learning from their mistakes.
askbill 14y ago
It sounds like they detected it but since so many components failed at once, the action was to page someone to have a look. I'm not convinced they can't actually do this, but rather were gun-shy about programmatically pulling the trigger. Back to AWS, the control plane failures are concerning, killing the ability to deploy new resources. It's expensive to have this kind of capacity pre-deployed and I'm sure it bit many folks.
jonny_eh 14y ago
I think this story from the front page of HN is relevant: http://pilif.github.com/2012/07/armchair-scientists/ http://pilif.github.com/2012/07/armchair-scientists/
- paulsutter 14y ago
  I'm not an armchair scientist, look at my bio. I was personally the driver behind the resiliency that we have at Quantcast, and if we had done anything less we would not have succeeded. I'm happy to help anyone who is serious about uptime, just email me. There's a reason that I post under my actual name, and there's a reason that I make my contact information avilable.
  - NDizzle 14y ago
    You may not be an armchair scientist but you are sure acting like a huge douchebag in this thread. Chill out.
  - ams6110 14y ago
    Based on your tone here, I don't think you could really "help" anyone. Maybe I'm wrong, but you're coming off like a complete a-hole
  - deleted 14y ago
    [deleted]
adrianco 14y ago
Netflix is designed to run on two out of three availability zones in a region. There are tens of TB of customer data triple replicated in that region, which has off-region archive but we don't live replicate the data intensive data sources. We also have the Europe region which does live replicate things like membership (since all members are global members of Netflix). In this case we had some bugs, we should have had a two minute increase in error rate as a third of the clients retried, then the dead instances would have been out of traffic. That's what happened in the previous power outage, where fewer instances went down, and it didn't trigger this bug.
jedberg 14y ago
Paul, I'm sure your skills are top notch, but even you have to admit that the problem you solved -- a globally available write only system -- is a totally different problem than the one Netflix is solving, which is a read/write workload. Also, as someone who was a user of your globally available service, I can tell you that while it might have been UP all the time, it certainly had no problems losing data all the time too. Some months there was just simply no data for reddit at all, even though we were sending the service more than a billion data points. So we could sit here and sling insults all day, or you can operate under the assumption I do -- that each of is a competent engineer who works for a business that has to make decisions and tradeoffs between costs and reliability.
- justinsb 14y ago
  Something constructive I'd like to hear from Netflix is a bit about what it is all those cloud instances actually do :-) My understanding: Streaming is offloaded to a CDN. The content & user databases change very slowly and thus are very amenable to caching. 'Current position' syncing is very non-critical and so can be done with e.g. writes to independent Redis instances (or even memcache - it really isn't very important!) Ratings, recommendations and the queue seem like the tricky ones, and while I don't think the throughput is particularly high, because they are all per-user this is "trivially shardable" if you do outgrow a single SQL database. The big question for me is understanding why streaming breaks, because that should all be served from systems that use read-only data? (where read-only = cacheable for at least a day without significant negative consequences) I think a better understanding of Netflix's technical challenges would serve everyone well here!
  - bcrescimanno 14y ago
    Read here: http://techblog.netflix.com/2012/04/netflix-recommendations-beyond-5-stars.html http://techblog.netflix.com/2012/04/netflix-recommendations-... In parts one and two, it's explained that just about every list generated for users is an up-to-date set of recommendations. You point out yourself that recommendations is one area where caching it's truly viable (at least, not in a traditional sense). I'm definitely not the right person to go into all the details (nor do I think such a discussion would be prudent on HackerNews)--but I wanted to weigh in quickly that there's a lot of stuff served that goes way beyond the notion of a "static" content that's trivially cached.

dangrossman 14y ago

That's something they just started talking about last year, when everything was still in a single region. http://techblog.netflix.com/2011/04/lessons-netflix-learned-from-aws-outage.html http://techblog.netflix.com/2011/04/lessons-netflix-learned-... Though their post today makes it sound like they could have (and maybe did) fail over to an entirely different region, but their mechanism for doing so isn't automatic and took longer than expected.

paulsutter 14y ago
It's literally shocking to me that responsible adults would "just start talking" about basic resiliency, like failing over to a different region, for a major commercial service. There are ways to do it that take seconds (we do that at Quantcast using anycast), there are ways that take minutes (using DNS failover, which is readily available), and there are ways that basically take forever (Netflix guys - feel free to contact me). It's pretty clear they were aware of the problem, and if they had even the simplest and most basic DNS control, they could have moved people over minutes after realizing the problem.
- maukdaddy 14y ago
  I'm sure it didn't literally shock you.
  - paulsutter 14y ago
    Lit·er·al·ly/ˈlitərəlē/ Adverb: 1. In a literal manner or sense; exactly: "the driver took it literally when asked to go straight over the traffic circle". 2. Used to acknowledge that something is not literally true but is used for emphasis or to express strong feeling.
    - sofal 14y ago
      Yes, the second definition was added because people lazily try to leverage the first definition to intensify the exaggeration of their figurative statements. The problem is that the second definition now masks the first (because it is essentially a devolved form of the first and actually relies on the first definition), making it difficult in many cases to get across the meaning of the first definition without resorting to other words. I see you've embraced this unfortunate evolution. Some of us are still mourning it.
      - aw3c2 14y ago
        Language evolves and changes. Please do not bitch about that in unrelated discussions on HN.
  - deleted 14y ago
    [deleted]
  - dionidium 14y ago
    It's called hyperbole and it's perfectly acceptable usage and we all -- every single one of us! -- knew exactly what he meant and dear god am I tired of people using this smug and useless retort.
- dangrossman 14y ago
  Didn't we just have a discussion of this type of comment today? [1] Being able to move /millions of requests per second/ (this was a Friday night!) on /thousands of servers/ [2] to a different /thousands of servers/ involves more than just routing requests to the right IP. Like having a separate set of /thousands of servers/ capable of handling that much failover, for one. Plus, if you had read the link, what they just started talking about a year after moving to AWS was moving across regions; they're already set up to fail entire availability zones over to other availability zones in the same region. Neither architecture at scale nor employee experience magically appears from nowhere. It's not even been two years since they moved from a data center to running the site on AWS. Considering how much they've had to learn, and the kinds of tools they've had to build for themselves to manage their kind of scale (read their tech blog!), I think the cheap insults are unwarranted. 1: http://news.ycombinator.com/item?id=4208134 http://news.ycombinator.com/item?id=4208134 2: http://techblog.netflix.com/2012/02/fault-tolerance-in-high-volume.html http://techblog.netflix.com/2012/02/fault-tolerance-in-high-...
  - paulsutter 14y ago
    My comment is far from a cheap insult. At Quantcast we handle 300,000 requests a second in 14 different cities, and we can failover between cities in 6 seconds. And we're doing a lot more than just log a pixel on each request, just to be clear. It makes maintenance and upgrades really easy. It was a bit of work, but we're always up. Even a child knows that when the power fails, you can't turn on the TV. This isn't specialized technical knowledge. EDIT: I'm not going to keep responding to your comment below. I'm certain that if I were involved in the design of Netflix' infrastructure, they would be able to survive problems that affect whole regions. (AND I DONT SEE THAT NETFLIX NEEDS 6 SECOND FAILOVER, THERE ARE MUCH EASIER WAYS TO DO IT IN MINUTES). EDIT: My repeated, emphatic comments are intended to serve a purpose. Everyone should be aware that this is a real problem and you need to plan for it, and it's pretty clear from the discussions on HN that people are surprised by the Amazon downtime. I personally think Amazon does a fantastic job as it is, and Amazon's reliability issues are not an excuse for the downtime of their customers.
    - breidh 14y ago
- Xorlev 14y ago
  It's literally shocking to me that you'd be belligerent about it. That's not how adults solve problems. AWS doesn't support multicast or anycast, and the AWS EC2 control planes were so hosed it was impossible to recover in any meaningful way. Certainly, the fault was both AWS and Netflix, but both are learning from their mistakes.

askbill 14y ago

It sounds like they detected it but since so many components failed at once, the action was to page someone to have a look. I'm not convinced they can't actually do this, but rather were gun-shy about programmatically pulling the trigger. Back to AWS, the control plane failures are concerning, killing the ability to deploy new resources. It's expensive to have this kind of capacity pre-deployed and I'm sure it bit many folks.

jonny_eh 14y ago

I think this story from the front page of HN is relevant: http://pilif.github.com/2012/07/armchair-scientists/ http://pilif.github.com/2012/07/armchair-scientists/

paulsutter 14y ago
I'm not an armchair scientist, look at my bio. I was personally the driver behind the resiliency that we have at Quantcast, and if we had done anything less we would not have succeeded. I'm happy to help anyone who is serious about uptime, just email me. There's a reason that I post under my actual name, and there's a reason that I make my contact information avilable.
- NDizzle 14y ago
  You may not be an armchair scientist but you are sure acting like a huge douchebag in this thread. Chill out.
- ams6110 14y ago
  Based on your tone here, I don't think you could really "help" anyone. Maybe I'm wrong, but you're coming off like a complete a-hole
- deleted 14y ago
  [deleted]

adrianco 14y ago

Netflix is designed to run on two out of three availability zones in a region. There are tens of TB of customer data triple replicated in that region, which has off-region archive but we don't live replicate the data intensive data sources. We also have the Europe region which does live replicate things like membership (since all members are global members of Netflix). In this case we had some bugs, we should have had a two minute increase in error rate as a third of the clients retried, then the dead instances would have been out of traffic. That's what happened in the previous power outage, where fewer instances went down, and it didn't trigger this bug.

jedberg 14y ago

Paul, I'm sure your skills are top notch, but even you have to admit that the problem you solved -- a globally available write only system -- is a totally different problem than the one Netflix is solving, which is a read/write workload. Also, as someone who was a user of your globally available service, I can tell you that while it might have been UP all the time, it certainly had no problems losing data all the time too. Some months there was just simply no data for reddit at all, even though we were sending the service more than a billion data points. So we could sit here and sling insults all day, or you can operate under the assumption I do -- that each of is a competent engineer who works for a business that has to make decisions and tradeoffs between costs and reliability.

justinsb 14y ago
Something constructive I'd like to hear from Netflix is a bit about what it is all those cloud instances actually do :-) My understanding: Streaming is offloaded to a CDN. The content & user databases change very slowly and thus are very amenable to caching. 'Current position' syncing is very non-critical and so can be done with e.g. writes to independent Redis instances (or even memcache - it really isn't very important!) Ratings, recommendations and the queue seem like the tricky ones, and while I don't think the throughput is particularly high, because they are all per-user this is "trivially shardable" if you do outgrow a single SQL database. The big question for me is understanding why streaming breaks, because that should all be served from systems that use read-only data? (where read-only = cacheable for at least a day without significant negative consequences) I think a better understanding of Netflix's technical challenges would serve everyone well here!
- bcrescimanno 14y ago
  Read here: http://techblog.netflix.com/2012/04/netflix-recommendations-beyond-5-stars.html http://techblog.netflix.com/2012/04/netflix-recommendations-... In parts one and two, it's explained that just about every list generated for users is an up-to-date set of recommendations. You point out yourself that recommendations is one area where caching it's truly viable (at least, not in a traditional sense). I'm definitely not the right person to go into all the details (nor do I think such a discussion would be prudent on HackerNews)--but I wanted to weigh in quickly that there's a lot of stuff served that goes way beyond the notion of a "static" content that's trivially cached.