4 ms·
> Sure, the chaos monkey could kill haproxy-server-state-management but that wouldn't uncover the bug in question No it won't. But it would uncover their missi
by harpratap 6y ago
> Sure, the chaos monkey could kill haproxy-server-state-management but that wouldn't uncover the bug in question
No it won't. But it would uncover their missing alerts for a critical platform component. Their issue was exacerbated by the fact that state-management kept failing for nearly 12 hours and no one noticed.
- cranekam 6y agoMaybe, though again I would expect an alert for the process being dead would also not uncover this particular issue. It's possible someone would notice that there was insufficient alerting then go back to add something which would have caught this but it's far from certain. OTOH it's also possible that at this stage in the game, when the code and systems are EOL, that something like chaos testing is the last chance to catch this problem. I'm not totally against chaos testing. I just haven't seen it done well and think it's actually pretty hard to pull off (particularly the non-technical aspect of convincing people it's okay to let this thing go mad). I'd love to see how effective it was within Netflix.