4 ms·
Interesting that all of the resource burning features have been removed, I wish they had expanded on the reasons why. I always found those to be the most differ
by andrewguenther 10y ago
Interesting that all of the resource burning features have been removed, I wish they had expanded on the reasons why. I always found those to be the most differentiating features of Chaos Monkey. Did they just not get a lot of use internally at Netflix?
- JoachimSchipper 10y agoYes, I wonder that too - complete node failure is the nicest failure that can happen. (See e.g. http://danluu.com/limplock/ http://danluu.com/limplock/, because Dan Luu's site is always excellent.) "We rewrote the service for improved maintainability" seems an important part of this blog post.
- flgr 10y agoI wonder if they have a service that automatically detects all those failures and terminates the entire node instead?
- notyourwork 10y agoI was thinking along these lines as well. Possibly they have other checks and balances in place for less than death situations for servers like high CPU usage and those situations result in another mechanism killing the server which is the same as what this solution provides.
- jpolit 10y agoWe do: http://techblog.netflix.com/2015/07/tracking-down-villains-outlier.html http://techblog.netflix.com/2015/07/tracking-down-villains-o...
- duaneb 10y agoMaybe internally such situations translate to a termination anyway.
- aaronblohowiak 10y agoResource exhaustion manifests as latency or failure. We inject latency and failure using FIT, so we can limit the "blast radius". When you are testing these failure modes, you are more testing the interaction between micro services, and this requires a bit more precision and sophistication. Source: I'm on the Chaos team here at Netflix.
- nawtacawp 10y agoperhaps it had too much liability.