4 ms·

I posted this piece of news to my team Slack at work, and a colleague of mine wrote: "we don't need chaos monkey, we have developers for that". While being fun

by iagooar 10y ago

I posted this piece of news to my team Slack at work, and a colleague of mine wrote: "we don't need chaos monkey, we have developers for that". While being funny, it also holds a lot of truth. I guess that Netflix can hire really top-notch devs who do not accidentally force downtime to their software.

Thaxll 10y ago
Netflix had a worldwide outage few weeks ago that last hours so yes they need those kind of tools :)
agentgt 10y ago
The joke is funny but it is actually shockingly difficult to make nodes kill themselves instead of doing something for more malignant which is the typical case for bugs. That is a zombie just like in real life can be worse than a dead corpse (since we are being funny and all). Chaos Monkey shoots two in the head. I have had issues killing errant JVMs and Rackspace nodes (yes sadly we are still on Rackspace). I can understand why 2.0 is much more focused given the plethora of monitoring solutions.
asadjb 10y ago
That rings so true. But on the flip side, it might be that because Netflix has a chaos monkey and all it's services need to be resilient to failures, accidental downtime isn't that big of an issue to them. Their developers can still make the same mistakes that we do, but their architecture is better designed to handle that. Just a thought.
jeremiep 10y ago
The difference is scale. Only a handful of companies run at Netflix scale. I wouldn't trust developers to do what Chaos Monkey does at such a scale, no matter how good you think they are.
- deleted 10y ago
  [deleted]
CorvusCrypto 10y ago
I don't think even hiring top-notch devs would make this reality completely go away. Tools like this would likely put them in check too. And yeah same response on our internal chat here basically x)
aaronblohowiak 10y ago
We have different tools for different kinds of failures. Chaos Monkey helps ensure that you are resilient to single instance failure. Kong helps ensure that we are resilient to region failure. Most of the developer-induced pain (which is most frequent source of pain) happens at the service level -- a bad code push that somehow made it through canary, accidentally doing something you shouldn't, misconfiguring something, etc. For tolerating service-level failures, we use different tools that minimize the fallout of the failure injection. Specifically, FIT (and the soon to be revealed ChAP.) These tools allow us to be more surgical in our injection of failure and tie that into our telemetry solutions. We only inject failures we expect to be resilient to. Sadly, that is a subset of the failures that people cause ;)
kevan 10y ago
Developers cause software downtime, but they usually don't cause infrastructure downtime, which is what Chaos Monkey does. Cloud VMs have failure rates somewhere around 1-2% depending on who you ask. This is low enough that you can ignore it most of the time, but it'll come back and bite you hard later. Chaos Monkey artificially forces that failure rate high enough that you'll notice problems immediately and fix them before they become too engrained into your architecture.
avitzurel 10y ago
From my experience, this is naive. It's funny, but really naive. The bigger your application stack is (micro-services, API calls, network calls), the more failure you would need to test out and there's no way to "trust" developers to do it themselves. Also, Netflix hires a lot of top-notch developers and their infrastructure is pretty awesome. [I don't work @ Netflix, just a devops dude :)]

by iagooar 10y ago

Thaxll 10y ago

Netflix had a worldwide outage few weeks ago that last hours so yes they need those kind of tools :)

agentgt 10y ago

The joke is funny but it is actually shockingly difficult to make nodes kill themselves instead of doing something for more malignant which is the typical case for bugs. That is a zombie just like in real life can be worse than a dead corpse (since we are being funny and all). Chaos Monkey shoots two in the head. I have had issues killing errant JVMs and Rackspace nodes (yes sadly we are still on Rackspace). I can understand why 2.0 is much more focused given the plethora of monitoring solutions.

asadjb 10y ago

That rings so true. But on the flip side, it might be that because Netflix has a chaos monkey and all it's services need to be resilient to failures, accidental downtime isn't that big of an issue to them. Their developers can still make the same mistakes that we do, but their architecture is better designed to handle that. Just a thought.

jeremiep 10y ago

The difference is scale. Only a handful of companies run at Netflix scale. I wouldn't trust developers to do what Chaos Monkey does at such a scale, no matter how good you think they are.

deleted 10y ago
[deleted]

CorvusCrypto 10y ago

I don't think even hiring top-notch devs would make this reality completely go away. Tools like this would likely put them in check too. And yeah same response on our internal chat here basically x)

aaronblohowiak 10y ago

We have different tools for different kinds of failures. Chaos Monkey helps ensure that you are resilient to single instance failure. Kong helps ensure that we are resilient to region failure. Most of the developer-induced pain (which is most frequent source of pain) happens at the service level -- a bad code push that somehow made it through canary, accidentally doing something you shouldn't, misconfiguring something, etc. For tolerating service-level failures, we use different tools that minimize the fallout of the failure injection. Specifically, FIT (and the soon to be revealed ChAP.) These tools allow us to be more surgical in our injection of failure and tie that into our telemetry solutions. We only inject failures we expect to be resilient to. Sadly, that is a subset of the failures that people cause ;)

kevan 10y ago

Developers cause software downtime, but they usually don't cause infrastructure downtime, which is what Chaos Monkey does. Cloud VMs have failure rates somewhere around 1-2% depending on who you ask. This is low enough that you can ignore it most of the time, but it'll come back and bite you hard later. Chaos Monkey artificially forces that failure rate high enough that you'll notice problems immediately and fix them before they become too engrained into your architecture.

avitzurel 10y ago

From my experience, this is naive. It's funny, but really naive. The bigger your application stack is (micro-services, API calls, network calls), the more failure you would need to test out and there's no way to "trust" developers to do it themselves. Also, Netflix hires a lot of top-notch developers and their infrastructure is pretty awesome. [I don't work @ Netflix, just a devops dude :)]