21 ms·
Netflix suffers first massive global outage
- dnate 8y agoI hope they release a post mortem about this. Would be very interesting.
- weliketocode 8y agoI was just about to reply with the exact same message. Netflix has spent A LOT of time and money on their infrastructure, expounding their engineering views, proclaiming best practices, and how they use redundancy/sharding/insert best practices and major buzzwords in order to combat the chance of this happening. I'm a huge fan of Netflix and their approaches to infrastructure, and I'm almost certain that they'll have some very interesting conclusions from this.
- Operyl 8y agoNothing is bullet proof. Even resilient systems like Amazon’s S3 are subject to human error, it’s not like Netflix being down is the end of the world.
- circa 8y agoTrue but you know people will act like it is.
- perfmode 8y agotrue. netflix is probably the world’s least critical system at global scale.
- Rjevski 8y agoAd networks would be the least critical. Netflix at least provides content some people actually want. I have yet to hear someone say "I want to see ads".
- vageli 8y ago> Ad networks would be the least critical. Netflix at least provides content some people actually want. I have yet to hear someone say "I want to see ads". That's actually a commonly-heard phrase in the US around Superbowl time.
- mike-cardwell 8y agoWell, when the alternative is watching American football...
- ihsw2 8y ago> watching American football Are you referring to the 17 minutes per hour of actual gameplay, the 21 minutes of rambling commentary/sports celebrity gossip, or the 22 minutes of ad spots?
- mkirklions 8y agoUpon recent years of dissapointment, I no longer watch superbowl ads. I think there was a massive marketing campaign to get people to pay attention to superbowl ads claiming they are funny. They've just turned into ads in recent years.
- glenneroo 8y agoAnd if the curiosity gets the better of you, numerous people upload them all to YouTube, often bundled together in one long clip (unfortunately too often with meager attempts at witty commentary).
- mrep 8y agoI thought the amazon alexa one this year was really funny: https://www.youtube.com/watch?v=J6-8DQALGt4 https://www.youtube.com/watch?v=J6-8DQALGt4
- quakenul 8y agoCertainly not, considering how heavily businesses rely on ad networks as a source of revenue (both by selling ad space but even more so by buying ad space and advertising their products)
- beberlei 8y agoconsider the increase in child birth rate by this outage. lives will be forever changed :-)
- awalton 8y agoCan't test for everything, but seeing this is the most significant outage in ages, I'd say they're doing a damned good job. I mean, after all, the internet's true natural predator still lurks, its yellow CAT steel present across all of the nation... I speak of course, of Backhoes. (1): https://www.wired.com/2006/01/the-backhoe-a-real-cyberthreat/ https://www.wired.com/2006/01/the-backhoe-a-real-cyberthreat... (2): https://it.slashdot.org/story/06/01/19/1643215/the-backhoe-the-internets-natural-enemy https://it.slashdot.org/story/06/01/19/1643215/the-backhoe-t... (3): https://www.urbandictionary.com/define.php?term=Fiber%20Seeking%20Backhoe https://www.urbandictionary.com/define.php?term=Fiber%20Seek...
- baud147258 8y agoI heard a story from the construction company (not int US) where my brother is working: on a construction site, they had to dig in an area with fiber. So they called the telco, aking them to disconnect the cables so they could start digging. But the telco was late and a few days later, fed up by the delay (and associated costs), the site manager said "start digging, we'll deal with the fallout". So the backhoe start digging and cut the cables. The telco were quick to send a guy after that...
- hinkley 8y agoI knew a guy who was the head of networking for a university. His biggest rant was that the fine structure for cutting a line made it cheaper to cross their fingers than to wait for someone to mark the exact location of the cables so they could dig.
- paulie_a 8y agoOr also known has backhoe fade
- sgift 8y agoYou make it sound as if good engineering practices somehow should guarantee no outages ever. Best practices are risk minimization, not risk elimination.
- hinkley 8y agoThe worst outage my company ever had was due to bad cost cutting decisions. My company ended up having an equipment failure that took out both production servers and out internal support system at the same time. Lots of pale faces that day. What has two thumbs and bureaucratic bullshit that put runbooks and deployment tools on production hardware? This guy. My bet is whatever cause this problem, someone got a raise last year for doing it.
- deleted 8y ago[deleted]
- danschumann 8y agoI thought they have a button in their office which shuts down a server, as a means of testing. Maybe someone brought in their 3 year old and he just kept hitting it.
- wyldfire 8y agoThey have a "Chaos Monkey" [1] feature that is intended to bring down individual nodes. "Exposing engineers to failures more frequently incentivizes them to build resilient services." If Chaos Monkey had been responsible for setting off a global outage, I could imagine business leaders getting cold feet about using a tool like this. In traditional companies, anyways, they'd never have seen the benefit of it and after only hearing the costs, they'd probably be livid that a widespread outage had been caused by something like this. [1] https://github.com/Netflix/chaosmonkey https://github.com/Netflix/chaosmonkey
- bpicolo 8y ago> they'd probably be livid that a widespread outage had been caused by something like this Widespread outage is exactly what something like chaos monkey is intended to help prevent. Even if this were chaos monkey induced (seems very unlikely), they've gained far more stability by having it than they would have lost in a single outage
- deelowe 8y agoThat's extremely speculative and impossible to quantify. There are plenty of counter examples where companies have impressive uptimes and don't resort to such extreme testing. Then again, attributing the outage to these chaos services is also extremely speculative at this point.
- mlthoughts2018 8y agoThis seems like basically the same principle as dropout from neural networks. One could make an Arrested Development episode where George Sr. and Lucille hear about this concept and decide to apply it to their children. (Edit: the Arrested Development <-> Netflix tie in didn't even occur to me until after writing the comment.)
- gwbas1c 8y agoIronically, the article isn't loading for me.
- a012 8y agoHN hugs of death? It seems like the article is also down.
- nmg 8y agoNet Neutrality's been dead for less than 24 hours. Tin foil hats aside, is it realistic to consider a relationship between these two events?
- nvr219 8y agoNot realistic at all.
- pritambaral 8y agoI think you should expand on why you think so. Here are my reasons: 1. This is a global outage. Net Neutrality is down in US only. 2. The pattern of failure is not consistent with American ISPs past attempts. They've wanted to make Netflix less attractive compared to their services, not outright broken. Subtlety is better because outright breakage brings too much attention.
- dagenix 8y agoNo
- LogicX 8y agoGoogle cache: https://webcache.googleusercontent.com/search?q=cache:https%3A%2F%2Fnews.alphastreet.com%2Fnetflix-suffers-first-massive-global-outage%2F https://webcache.googleusercontent.com/search?q=cache:https%...
- Retroity 8y agoHere's an alternative article from CNBC since the linked article is having trouble loading: https://www.cnbc.com/2018/06/12/netflix-down-streaming-service-says-it-fixed-problem-that-caused-outage.html https://www.cnbc.com/2018/06/12/netflix-down-streaming-servi...
- lyjackal 8y agohttps://webcache.googleusercontent.com/search?q=cache:mKL4IqFNOO8J:https://news.alphastreet.com/netflix-suffers-first-massive-global-outage/ https://webcache.googleusercontent.com/search?q=cache:mKL4Iq...
- jack6e 8y agoAt PyCon in Cleveland this year Amjith Ramanujam did a presentation on "how Netflix does failovers in 7 minutes flat" [0]. Worth a watch/listen for anyone interested in what their response may have looked like. Now I'm curious to read a post-mortem from them and see whether their procedures worked as expected (it sounds like they were down longer than 7 minutes) or where they encountered unexpected issues. [0] https://www.youtube.com/watch?v=iQI56-up3Yk https://www.youtube.com/watch?v=iQI56-up3Yk
- mikece 8y agoDid someone at NFLX point Chaos Monkey at something they shouldn't have or was this an "honest failure?"
- john58 8y agoHere's the link to AMP page of the article: https://news.alphastreet.com/netflix-suffers-first-massive-global-outage/amp/ https://news.alphastreet.com/netflix-suffers-first-massive-g...
- paulgpetty 8y agoIt might not seem like this is a big deal. For now that’s probably right; but Netflix has aspirations to do nightly news and if that’s intended to be live or near real-time we should be comparing their uptime with broadcast TV. When was the last time ABC or Fox went off air? Again, not right now, but at some point this sort of outage becomes critical and will probably deserve more scrutiny. I’m impressed by how reliable Netflix is, hopefully this just pushes them harder. And maybe even prepares them for a less neutral internet.
- Retric 8y agoDepends on what you're measuring. Local stations regularly go off air. Sure, some stations outside the range of your antenna might work but effectively their just down. For national events they will fail over to the local stations and keep broadcasting, but if the content is not there they are just down.
- Scarbutt 8y agoThat's like blaming Netflix when your local ISP is down.
- e40 8y agoMy dad has directv and they regularly have issues. He wasn't able to get NBA games for weeks, due to some software glitch (the entire channel was unavailable to him, for unknown reasons).
- arjunvijay 8y agoI wonder why they have not given an official statement on why it happened. Probably just a one-off case.
- filereaper 8y agoI'm looking forward to the post-mortem of this (hopefully by someone like Dave Hahn) Hopefully it wasn't a Hurricane's Butterfly kind of deal...
- pc86 8y agoWhat do you mean by Hurricane's Butterfly? I'm familiar with the "if a butterfly flaps its wings" thing but not sure what you're referring to in this context is all.
- filereaper 8y agoApologies, was referring to bcantrill's tech talk. [1] Similar idea, you're in a hurricane but it may have been caused by an insignificant butterfly far awaya. [1] https://www.janestreet.com/tech-talks/hurricanes-butterfly/ https://www.janestreet.com/tech-talks/hurricanes-butterfly/
- thermodynthrway 8y agoI'm not surprised. I've tried to get Netflix's OSS tools like Spinnaker running and it's a total nightmare with how many interdependent services need to run. It took me days to get running and was never reliable. I think they drank a little too much koolaid. Microservices + async DB updates = hell. After working on such projects, I respect the wisdom of Google making all of their data stores immediately consistent. A monolith isn't bad at all if it's organized and built to scale horizontally. Move your consistency concerns back to the database where they belong, not between services. It's a joke that monoliths are bad when you look at every popular operating system kernel. These things are multi million line binary blobs in languages that aren't friendly to mistakes and they run EVERYTHING
- jedberg 8y ago> I've tried to get Netflix's OSS tools like Spinnaker running and it's a total nightmare You may want to check out Armory Spinnaker[0]. It's a commercial version of Spinnaker that takes care of all the hard bits of setting it up. [0] https://www.armory.io https://www.armory.io Disclaimer: I'm an investor in Armory
- danielodio 8y agoHey @thermodynthrway -- CEO of Armory here. You can give our enterprise distribution a try at http://go.Armory.io/install http://go.Armory.io/install We'd love to hear how it goes for you!
- Redsquare 8y agoBit drastic considering this was their first major global outage...
- TotempaaltJ 8y agoSeeing as this is their first major outage, and Netflix-the-streaming-service has existed since 2007, I'd say don't know what you're talking about.
- 8y ago
- poundtown 8y agocool to see people totally having to face their addiction. my kids were flopping around on the floor like fish out of water.
- pofilat 8y agoInstagram, GitLab, Netflix all down at the same time. Coincidence?
- breakingcups 8y agoAdd StackOverflow to that list.