6 ms·

I thought they have a button in their office which shuts down a server, as a means of testing. Maybe someone brought in their 3 year old and he just kept hitti

I thought they have a button in their office which shuts down a server, as a means of testing. Maybe someone brought in their 3 year old and he just kept hitting it.

wyldfire 8y ago
They have a "Chaos Monkey" [1] feature that is intended to bring down individual nodes. "Exposing engineers to failures more frequently incentivizes them to build resilient services." If Chaos Monkey had been responsible for setting off a global outage, I could imagine business leaders getting cold feet about using a tool like this. In traditional companies, anyways, they'd never have seen the benefit of it and after only hearing the costs, they'd probably be livid that a widespread outage had been caused by something like this. [1] https://github.com/Netflix/chaosmonkey https://github.com/Netflix/chaosmonkey
- bpicolo 8y ago
  > they'd probably be livid that a widespread outage had been caused by something like this Widespread outage is exactly what something like chaos monkey is intended to help prevent. Even if this were chaos monkey induced (seems very unlikely), they've gained far more stability by having it than they would have lost in a single outage
  - deelowe 8y ago
    That's extremely speculative and impossible to quantify. There are plenty of counter examples where companies have impressive uptimes and don't resort to such extreme testing. Then again, attributing the outage to these chaos services is also extremely speculative at this point.
- mlthoughts2018 8y ago
  This seems like basically the same principle as dropout from neural networks. One could make an Arrested Development episode where George Sr. and Lucille hear about this concept and decide to apply it to their children. (Edit: the Arrested Development <-> Netflix tie in didn't even occur to me until after writing the comment.)
- some_account 8y ago
  It's possible that executes would think that way but only because they are not too technical. Every company should have a chaos monkey to make sure stability is at the top of the list always.
  - nwah1 8y ago
    Every large company building mission critical systems, maybe. Resilient systems take longer to build, and thus are more costly. It is not always wise to spend that much resources on this for, say, a startup. After a startup is crushing it, then maybe they can start rewriting any brittle monolithic systems into something more resilient.
  - noir_lord 8y ago
    > Every company should have a chaos monkey. Big companies do though they call then interns.
- squeaky-clean 8y ago
  They've actually upgraded to the "Simian Army" now. [0] Which among others includes the "Chaos Gorilla": > Chaos Gorilla is similar to Chaos Monkey, but simulates an outage of an entire Amazon availability zone. https://medium.com/netflix-techblog/the-netflix-simian-army-16e57fbab116 https://medium.com/netflix-techblog/the-netflix-simian-army-...
  - iamforreal 8y ago
    Isn't there like a Kong for an entire region as well?
    - chx 8y ago
      Yes, that's in the 2015 blog post https://medium.com/netflix-techblog/chaos-engineering-upgraded-878d341f15fa https://medium.com/netflix-techblog/chaos-engineering-upgrad...

I thought they have a button in their office which shuts down a server, as a means of testing. Maybe someone brought in their 3 year old and he just kept hitting it.

wyldfire 8y ago

They have a "Chaos Monkey" [1] feature that is intended to bring down individual nodes. "Exposing engineers to failures more frequently incentivizes them to build resilient services." If Chaos Monkey had been responsible for setting off a global outage, I could imagine business leaders getting cold feet about using a tool like this. In traditional companies, anyways, they'd never have seen the benefit of it and after only hearing the costs, they'd probably be livid that a widespread outage had been caused by something like this. [1] https://github.com/Netflix/chaosmonkey https://github.com/Netflix/chaosmonkey

bpicolo 8y ago
> they'd probably be livid that a widespread outage had been caused by something like this Widespread outage is exactly what something like chaos monkey is intended to help prevent. Even if this were chaos monkey induced (seems very unlikely), they've gained far more stability by having it than they would have lost in a single outage
- deelowe 8y ago
  That's extremely speculative and impossible to quantify. There are plenty of counter examples where companies have impressive uptimes and don't resort to such extreme testing. Then again, attributing the outage to these chaos services is also extremely speculative at this point.
mlthoughts2018 8y ago
This seems like basically the same principle as dropout from neural networks. One could make an Arrested Development episode where George Sr. and Lucille hear about this concept and decide to apply it to their children. (Edit: the Arrested Development <-> Netflix tie in didn't even occur to me until after writing the comment.)
some_account 8y ago
It's possible that executes would think that way but only because they are not too technical. Every company should have a chaos monkey to make sure stability is at the top of the list always.
- nwah1 8y ago
  Every large company building mission critical systems, maybe. Resilient systems take longer to build, and thus are more costly. It is not always wise to spend that much resources on this for, say, a startup. After a startup is crushing it, then maybe they can start rewriting any brittle monolithic systems into something more resilient.
- noir_lord 8y ago
  > Every company should have a chaos monkey. Big companies do though they call then interns.
squeaky-clean 8y ago
They've actually upgraded to the "Simian Army" now. [0] Which among others includes the "Chaos Gorilla": > Chaos Gorilla is similar to Chaos Monkey, but simulates an outage of an entire Amazon availability zone. https://medium.com/netflix-techblog/the-netflix-simian-army-16e57fbab116 https://medium.com/netflix-techblog/the-netflix-simian-army-...
- iamforreal 8y ago
  Isn't there like a Kong for an entire region as well?
  - chx 8y ago
    Yes, that's in the 2015 blog post https://medium.com/netflix-techblog/chaos-engineering-upgraded-878d341f15fa https://medium.com/netflix-techblog/chaos-engineering-upgrad...