4 ms·

Chaos Engineering comes into play here. Deliberately break your platform to see if everything works as expected.

input_sh 6y ago
To add to that, Netflix has open sourced a tool called Chaos Monkey which randomly terminates production instances so that you can be sure that your overall system is resilient to instance failures.
- brazzy 6y ago
  I don't think that would have ever provoked this error mode, though.
  - adrianN 6y ago
    It might have prevented this failure by killing stale HAProxy instances...
  - harpratap 6y ago
    It would have. Their haproxy-server-state-management also comes under the scope of Chaos testing. Which was failing but they never set up alerts for it. So they could have seen that their chaos engine killed this process but they never got notified.
ex_amazon_sde 6y ago
> Chaos Engineering ...or just "availability testing" or simply "testing". Yet, testing systemic failure modes at scale is way more tricky than shutting down some VMs or some network devices. For example: saturating the uplink bandwidth on a whole datacenter.
cranekam 6y ago
But how, exactly? Sure, the chaos monkey could kill haproxy-server-state-management but that wouldn't uncover the bug in question — it'd just demonstrate that without it running HA proxy's view of the world goes stale, which anyone would expect. Triggering the bug would require reducing the number of HA proxy slots below the number of webapps running for many hours. This is clearly something chaos engineering could do but IMO it's highly unlikely anyone would think to do this. If they thought of this they would also have thought about adding tests that caught such as issue long before the code went into production. In my experience chaos engineering is often only as good as the amount of thought put into the things it does. Killing processes here and there can be useful but it often won't expose the kind of when-the-stars-align issues that take down infrastructures. It looks like a classic lack of monitoring, as the article says. Alerting on webapps > slots, early exits, or differing views of the number of webapps up would have likely caught this.
- harpratap 6y ago
  > Sure, the chaos monkey could kill haproxy-server-state-management but that wouldn't uncover the bug in question No it won't. But it would uncover their missing alerts for a critical platform component. Their issue was exacerbated by the fact that state-management kept failing for nearly 12 hours and no one noticed.
  - cranekam 6y ago
    Maybe, though again I would expect an alert for the process being dead would also not uncover this particular issue. It's possible someone would notice that there was insufficient alerting then go back to add something which would have caught this but it's far from certain. OTOH it's also possible that at this stage in the game, when the code and systems are EOL, that something like chaos testing is the last chance to catch this problem. I'm not totally against chaos testing. I just haven't seen it done well and think it's actually pretty hard to pull off (particularly the non-technical aspect of convincing people it's okay to let this thing go mad). I'd love to see how effective it was within Netflix.
zomglings 6y ago
To catch this particular issue, they would have had to overprovision webapp instances beyond their N*M limit, take down a subset of the originally provisioned webapp instances and reprovision new ones in order to have observed this particular issue with their HAProxy setup. It seems like it would take a superhuman level of foresight to catch this scenario for chaos engineering.

To add to that, Netflix has open sourced a tool called Chaos Monkey which randomly terminates production instances so that you can be sure that your overall system is resilient to instance failures.

brazzy 6y ago
I don't think that would have ever provoked this error mode, though.
- adrianN 6y ago
  It might have prevented this failure by killing stale HAProxy instances...
- harpratap 6y ago
  It would have. Their haproxy-server-state-management also comes under the scope of Chaos testing. Which was failing but they never set up alerts for it. So they could have seen that their chaos engine killed this process but they never got notified.

ex_amazon_sde 6y ago

> Chaos Engineering ...or just "availability testing" or simply "testing". Yet, testing systemic failure modes at scale is way more tricky than shutting down some VMs or some network devices. For example: saturating the uplink bandwidth on a whole datacenter.

cranekam 6y ago

But how, exactly? Sure, the chaos monkey could kill haproxy-server-state-management but that wouldn't uncover the bug in question — it'd just demonstrate that without it running HA proxy's view of the world goes stale, which anyone would expect. Triggering the bug would require reducing the number of HA proxy slots below the number of webapps running for many hours. This is clearly something chaos engineering could do but IMO it's highly unlikely anyone would think to do this. If they thought of this they would also have thought about adding tests that caught such as issue long before the code went into production. In my experience chaos engineering is often only as good as the amount of thought put into the things it does. Killing processes here and there can be useful but it often won't expose the kind of when-the-stars-align issues that take down infrastructures. It looks like a classic lack of monitoring, as the article says. Alerting on webapps > slots, early exits, or differing views of the number of webapps up would have likely caught this.

harpratap 6y ago
> Sure, the chaos monkey could kill haproxy-server-state-management but that wouldn't uncover the bug in question No it won't. But it would uncover their missing alerts for a critical platform component. Their issue was exacerbated by the fact that state-management kept failing for nearly 12 hours and no one noticed.
- cranekam 6y ago
  Maybe, though again I would expect an alert for the process being dead would also not uncover this particular issue. It's possible someone would notice that there was insufficient alerting then go back to add something which would have caught this but it's far from certain. OTOH it's also possible that at this stage in the game, when the code and systems are EOL, that something like chaos testing is the last chance to catch this problem. I'm not totally against chaos testing. I just haven't seen it done well and think it's actually pretty hard to pull off (particularly the non-technical aspect of convincing people it's okay to let this thing go mad). I'd love to see how effective it was within Netflix.

zomglings 6y ago

To catch this particular issue, they would have had to overprovision webapp instances beyond their N*M limit, take down a subset of the originally provisioned webapp instances and reprovision new ones in order to have observed this particular issue with their HAProxy setup. It seems like it would take a superhuman level of foresight to catch this scenario for chaos engineering.