4 ms·

Netflix was experiencing issues

37 points by jameshk 12y ago

nattaylor 12y ago
Failed to load resource: the server responded with a status of 503 (Service Unavailable: Back-end server is at capacity) https://support.netflix.com/en/rest/views/popularbytopcat.jsonp?args[0]=131&limit=3&callback=jQuery1830988649561535567_1423010486399&_=1423010486506 https://support.netflix.com/en/rest/views/popularbytopcat.js...
- nattaylor 12y ago
  From their Twitter: https://twitter.com/Netflixhelps/status/562760983790485504 https://twitter.com/Netflixhelps/status/562760983790485504 We're aware some members are experiencing technical difficulties on the web site in all regions. We hope to resolve this asap.
Aldo_MX 12y ago
Failed to load resource: the server responded with a status of 503 (Service Unavailable: Back-end server is at capacity)
jameshk 12y ago
I'm watching House of Cards right now, but it took a few tries for it to start and not throw an error.
AYBABTME 12y ago
Really looking forward to read about what happened. I know they have great resiliency measures in place, I'd like to know if they worked, in what measure, how they failed or how they weren't enough. Or if there was anything they could have done.
- johansch 12y ago
  Great resilience measures? Like depending on AWS that's seemingly going down like one full day per year? I guess what I'm trying to say is that one the one hand i see netflix having lots out outages, on the other hand being celebrated as some sort of thought leader on resilicency/high availability. Good tech/open source marketing goes a long way in certain crowds, it seems. It's not like doing lots of stateless transactions/distributing read-only content is a really hard problem by now. Akamai has done it well for ages. Yet Netflix while being celebrated primarily for their tech marketing in this area is struggling.
  - pat2man 12y ago
    Yes, those types of measures.
  - fletchowns 12y ago
    https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey
  - AYBABTME 12y ago
    Great resilience measure, in spite of their choice to use AWS. Any infrastructure will involve downtime, be it your own hardware or some cloud provider. Probabilities will vary, but they're always there. Despite their hosting solution, Netflix is doing (or showing) extra effort in dealing with the reality of failures. I'm interested in that bit. Sure it might be a good idea to diversify their cloud providers, although I'm not sure this might be involved here, since AWS seems to be doing fine at this time.
  - abalone 12y ago
    According to this audit EC2 had just 2.14 total hours downtime last year, the least of any major provider by region. https://cloudharmony.com/status-1year-group-by-regions https://cloudharmony.com/status-1year-group-by-regions This is HN.. You shouldn't use dodgy language like "seemingly" to avoid doing homework and citing sources.
    - johansch 12y ago
      Then there's S3, and xyz, etc.
      - abalone 12y ago
        S3 had 2.81 hours downtime, according to the report I just linked to. Homework...
      - thezilch 12y ago
        xyz? S3 is on the same link. Even if they are additive -- assume the S3 downtime doesn't overlap with EC2 -- they are still at less than a day, half day, quarter day...
jijojv 12y ago
AWS status is all green so this is a first. Monkey gone wrong ?
dragonshed 12y ago
Worked fine from my iPhone over LTE but not Wii over wifi. And apparently it started right before my son's designated Sesame Street time. Boo.
- rodgerd 12y ago
  > And apparently it started right before my son's designated Sesame Street time. Something to be said for local copies of programs that really matter.
  - FroshKiller 12y ago
    Something something offline viewing?
ericfrederich 12y ago
Chaos monkey strikes?
thezilch 12y ago
Planet of the Apes? Eager for the postmortem.
wodenokoto 12y ago
i have been getting the issues warning and a limited movie selection on and off for days now.
mhuffman 12y ago
Rise of the simian army? I suspected netflix was hiding the "good" AI somewhere (because they haven't been using it on my recommendations).
ahcox 12y ago
Podcast where Adrian Cockcroft goes into the philosophy and technology behind scaling at Netflix: http://www.se-radio.net/2014/12/episode-216-adrian-cockcroft-on-the-modern-cloud-based-platform/ http://www.se-radio.net/2014/12/episode-216-adrian-cockcroft...

37 points by jameshk 12y ago

Failed to load resource: the server responded with a status of 503 (Service Unavailable: Back-end server is at capacity) https://support.netflix.com/en/rest/views/popularbytopcat.jsonp?args[0]=131&limit=3&callback=jQuery1830988649561535567_1423010486399&_=1423010486506 https://support.netflix.com/en/rest/views/popularbytopcat.js...

nattaylor 12y ago
From their Twitter: https://twitter.com/Netflixhelps/status/562760983790485504 https://twitter.com/Netflixhelps/status/562760983790485504 We're aware some members are experiencing technical difficulties on the web site in all regions. We hope to resolve this asap.

Aldo_MX 12y ago

Failed to load resource: the server responded with a status of 503 (Service Unavailable: Back-end server is at capacity)

jameshk 12y ago

I'm watching House of Cards right now, but it took a few tries for it to start and not throw an error.

AYBABTME 12y ago

Really looking forward to read about what happened. I know they have great resiliency measures in place, I'd like to know if they worked, in what measure, how they failed or how they weren't enough. Or if there was anything they could have done.

johansch 12y ago
Great resilience measures? Like depending on AWS that's seemingly going down like one full day per year? I guess what I'm trying to say is that one the one hand i see netflix having lots out outages, on the other hand being celebrated as some sort of thought leader on resilicency/high availability. Good tech/open source marketing goes a long way in certain crowds, it seems. It's not like doing lots of stateless transactions/distributing read-only content is a really hard problem by now. Akamai has done it well for ages. Yet Netflix while being celebrated primarily for their tech marketing in this area is struggling.
- pat2man 12y ago
  Yes, those types of measures.
- fletchowns 12y ago
  https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey https://github.com/Netflix/SimianArmy/wiki/Chaos-Monkey
- AYBABTME 12y ago
  Great resilience measure, in spite of their choice to use AWS. Any infrastructure will involve downtime, be it your own hardware or some cloud provider. Probabilities will vary, but they're always there. Despite their hosting solution, Netflix is doing (or showing) extra effort in dealing with the reality of failures. I'm interested in that bit. Sure it might be a good idea to diversify their cloud providers, although I'm not sure this might be involved here, since AWS seems to be doing fine at this time.
- abalone 12y ago
  According to this audit EC2 had just 2.14 total hours downtime last year, the least of any major provider by region. https://cloudharmony.com/status-1year-group-by-regions https://cloudharmony.com/status-1year-group-by-regions This is HN.. You shouldn't use dodgy language like "seemingly" to avoid doing homework and citing sources.
  - johansch 12y ago
    Then there's S3, and xyz, etc.
    - abalone 12y ago
      S3 had 2.81 hours downtime, according to the report I just linked to. Homework...
    - thezilch 12y ago
      xyz? S3 is on the same link. Even if they are additive -- assume the S3 downtime doesn't overlap with EC2 -- they are still at less than a day, half day, quarter day...

jijojv 12y ago

AWS status is all green so this is a first. Monkey gone wrong ?

dragonshed 12y ago

Worked fine from my iPhone over LTE but not Wii over wifi. And apparently it started right before my son's designated Sesame Street time. Boo.

rodgerd 12y ago
> And apparently it started right before my son's designated Sesame Street time. Something to be said for local copies of programs that really matter.
- FroshKiller 12y ago
  Something something offline viewing?

ericfrederich 12y ago

Chaos monkey strikes?

thezilch 12y ago

Planet of the Apes? Eager for the postmortem.

wodenokoto 12y ago

i have been getting the issues warning and a limited movie selection on and off for days now.

mhuffman 12y ago

Rise of the simian army? I suspected netflix was hiding the "good" AI somewhere (because they haven't been using it on my recommendations).

ahcox 12y ago

Podcast where Adrian Cockcroft goes into the philosophy and technology behind scaling at Netflix: http://www.se-radio.net/2014/12/episode-216-adrian-cockcroft-on-the-modern-cloud-based-platform/ http://www.se-radio.net/2014/12/episode-216-adrian-cockcroft...