4 ms·

Really looking forward to read about what happened. I know they have great resiliency measures in place, I'd like to know if they worked, in what measure, how t

Really looking forward to read about what happened. I know they have great resiliency measures in place, I'd like to know if they worked, in what measure, how they failed or how they weren't enough. Or if there was anything they could have done.

johansch 12y ago
Great resilience measures? Like depending on AWS that's seemingly going down like one full day per year? I guess what I'm trying to say is that one the one hand i see netflix having lots out outages, on the other hand being celebrated as some sort of thought leader on resilicency/high availability. Good tech/open source marketing goes a long way in certain crowds, it seems. It's not like doing lots of stateless transactions/distributing read-only content is a really hard problem by now. Akamai has done it well for ages. Yet Netflix while being celebrated primarily for their tech marketing in this area is struggling.
- pat2man 12y ago
  Yes, those types of measures.
- fletchowns 12y ago
  https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey
- AYBABTME 12y ago
  Great resilience measure, in spite of their choice to use AWS. Any infrastructure will involve downtime, be it your own hardware or some cloud provider. Probabilities will vary, but they're always there. Despite their hosting solution, Netflix is doing (or showing) extra effort in dealing with the reality of failures. I'm interested in that bit. Sure it might be a good idea to diversify their cloud providers, although I'm not sure this might be involved here, since AWS seems to be doing fine at this time.
- abalone 12y ago
  According to this audit EC2 had just 2.14 total hours downtime last year, the least of any major provider by region. https://cloudharmony.com/status-1year-group-by-regions https://cloudharmony.com/status-1year-group-by-regions This is HN.. You shouldn't use dodgy language like "seemingly" to avoid doing homework and citing sources.
  - johansch 12y ago
    Then there's S3, and xyz, etc.
    - abalone 12y ago
      S3 had 2.81 hours downtime, according to the report I just linked to. Homework...
    - thezilch 12y ago
      xyz? S3 is on the same link. Even if they are additive -- assume the S3 downtime doesn't overlap with EC2 -- they are still at less than a day, half day, quarter day...
      - chenshu_ivory 12y ago
        What does S3 mean? Seems you guys are working for troubleshooting.
        thezilch 12y ago
        S3 is an AWS product: http://aws.amazon.com/s3/ http://aws.amazon.com/s3/ johansch is trying to paint AWS -- the culmination of its products (EC2, S3, "xyz", etc) -- as something that is down for day(s) a year. We don't (have to) work for these providers to know the real truth and shut down such FUD.
- mikeyouse 12y ago
  During the evening hours, Netflix represents over 35% of all internet traffic in the US. That is an insane demand to be able to service, but they've done extremely well in my opinion. Much of their tech is open-sourced leading to much better practices industry-wide and they're transparent when things go wrong. Here's an example of their preemptive work to deal with the loss of AWS/ELB regions: http://techblog.netflix.com/2013/12/active-active-for-multi-regional.html http://techblog.netflix.com/2013/12/active-active-for-multi-... An an example of their post-mortem after deploying Node.js with a bad handler: http://techblog.netflix.com/2014/11/nodejs-in-flames.html http://techblog.netflix.com/2014/11/nodejs-in-flames.html Is there any other service with a library well over a petabyte in size that delivers multiple terabits/second worldwide to millions of customers with better reliability? Maybe YouTube with the backing of a company worth $300B more than Netflix?
  - johansch 12y ago
    Again you are confusing openness with availability. I do agree they are open. I don't really see the connection between their openness and their performance though.
    - mikeyouse 12y ago
      I think my last paragraph was pertinent to your question, is there anyone providing the volume of traffic to millions of worldwide customers comparable to Netflix who is more reliable?

johansch 12y ago

Great resilience measures? Like depending on AWS that's seemingly going down like one full day per year? I guess what I'm trying to say is that one the one hand i see netflix having lots out outages, on the other hand being celebrated as some sort of thought leader on resilicency/high availability. Good tech/open source marketing goes a long way in certain crowds, it seems. It's not like doing lots of stateless transactions/distributing read-only content is a really hard problem by now. Akamai has done it well for ages. Yet Netflix while being celebrated primarily for their tech marketing in this area is struggling.

pat2man 12y ago
Yes, those types of measures.
fletchowns 12y ago
https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey
AYBABTME 12y ago
Great resilience measure, in spite of their choice to use AWS. Any infrastructure will involve downtime, be it your own hardware or some cloud provider. Probabilities will vary, but they're always there. Despite their hosting solution, Netflix is doing (or showing) extra effort in dealing with the reality of failures. I'm interested in that bit. Sure it might be a good idea to diversify their cloud providers, although I'm not sure this might be involved here, since AWS seems to be doing fine at this time.
abalone 12y ago
According to this audit EC2 had just 2.14 total hours downtime last year, the least of any major provider by region. https://cloudharmony.com/status-1year-group-by-regions https://cloudharmony.com/status-1year-group-by-regions This is HN.. You shouldn't use dodgy language like "seemingly" to avoid doing homework and citing sources.
- johansch 12y ago
  Then there's S3, and xyz, etc.
  - abalone 12y ago
    S3 had 2.81 hours downtime, according to the report I just linked to. Homework...
  - thezilch 12y ago
    xyz? S3 is on the same link. Even if they are additive -- assume the S3 downtime doesn't overlap with EC2 -- they are still at less than a day, half day, quarter day...
    - chenshu_ivory 12y ago
      What does S3 mean? Seems you guys are working for troubleshooting.
      - thezilch 12y ago
        S3 is an AWS product: http://aws.amazon.com/s3/ http://aws.amazon.com/s3/ johansch is trying to paint AWS -- the culmination of its products (EC2, S3, "xyz", etc) -- as something that is down for day(s) a year. We don't (have to) work for these providers to know the real truth and shut down such FUD.
mikeyouse 12y ago
During the evening hours, Netflix represents over 35% of all internet traffic in the US. That is an insane demand to be able to service, but they've done extremely well in my opinion. Much of their tech is open-sourced leading to much better practices industry-wide and they're transparent when things go wrong. Here's an example of their preemptive work to deal with the loss of AWS/ELB regions: http://techblog.netflix.com/2013/12/active-active-for-multi-regional.html http://techblog.netflix.com/2013/12/active-active-for-multi-... An an example of their post-mortem after deploying Node.js with a bad handler: http://techblog.netflix.com/2014/11/nodejs-in-flames.html http://techblog.netflix.com/2014/11/nodejs-in-flames.html Is there any other service with a library well over a petabyte in size that delivers multiple terabits/second worldwide to millions of customers with better reliability? Maybe YouTube with the backing of a company worth $300B more than Netflix?
- johansch 12y ago
  Again you are confusing openness with availability. I do agree they are open. I don't really see the connection between their openness and their performance though.
  - mikeyouse 12y ago
    I think my last paragraph was pertinent to your question, is there anyone providing the volume of traffic to millions of worldwide customers comparable to Netflix who is more reliable?