11 ms·
Netflix Chaos Monkey Upgraded
- moondev 10y agoThis is awesome. Since Chaos Monkey now leverages spinnaker you can run it against clouddriver provider. Looking forward to trying this out with kubernetes. I believe spinnaker treats namespaces as regions. Eventually it would be cool to simulate masters or other entire federated clusters going down to test kubernetes scheduling resilience
- ticklemyelmo 10y agoIt looks like it's working too well. The site is unreachable for me.
- netsec_burn 10y agoIt's up.
- dest 10y agodown : Secure Connection Failed The connection to techblog.netflix.com was interrupted while the page was loading.
- fred256 10y agoIt's only available on http, not https.
- gonzoflip 10y agoI am receiving a DNS error when trying to reach the URL C:\Windows\System32>nslookup http://techblog.netflix.com/ http://techblog.netflix.com/ 8.8.8.8 Server: google-public-dns-a.google.com Address: 8.8.8.8 * google-public-dns-a.google.com can't find http://techblog.netflix.com/ http://techblog.netflix.com/: Non-existent domain
- ticklemyelmo 10y agoLooks like some sequence of events convinces Firefox that HSTS is set, and it always rewrites the request as https (which is not supported by the server) after that.
- Thaxll 10y agoEvery time that URL comes up people try to access it from https but the site is only available from http... Fix your Firefox, it's clearly at fault here.
- scrollaway 10y ago"Every time"? If one random guy's complaining about a URL being unreachable and you're already seeing a pattern... is it at all possible the users aren't at fault?
- yalooze 10y agoIn Thaxll's defense, I've seen this a lot with Netflix blog posts on HN.
- corobo 10y agoTo be fair it does happen frequently enough. I wouldn't say every time but often on Netflix posts. A bit of a sampling for you: https://news.ycombinator.com/item?id=12269411 https://news.ycombinator.com/item?id=12269411 https://news.ycombinator.com/item?id=12217900 https://news.ycombinator.com/item?id=12217900 https://news.ycombinator.com/item?id=12038367 https://news.ycombinator.com/item?id=12038367 https://news.ycombinator.com/item?id=11771714 https://news.ycombinator.com/item?id=11771714
- niftich 10y agoIt's NoScript's new-ish defaults at fault. See https://news.ycombinator.com/item?id=12256720 https://news.ycombinator.com/item?id=12256720
- e1g 10y agoAnother useful tool is https://github.com/gaia-adm/pumba https://github.com/gaia-adm/pumba - like ChaosMonkey, but just for Docker containers. The coolest part for us was emulating networking problems between containers (packet loss, unavailability etc).
- jackweirdy 10y agoI'd love to see this used in the Jepsen tests
- haneefmubarak 10y agoAFAIR Jepsen is only meant to test the CAP aspects of a system, not how it behaves otherwise.
- gkop 10y agoParent may have meant that it would be cool to see a write up on how this tool can be used to simulate network conditions between nodes in a Jepsen test using only containers on a single docker host.
- jackweirdy 10y agoYep, that's what I meant.
- brunoqc 10y agoWhat do you use for high availability with Docker?
- e1g 10y agoHA containers for us means smart orchestration tools. We did not want to lock ourselves into Docker-only infrastructure (even now rkt is a very compelling alternative), and wanted an orchestrator/scheduler that is focused entirely on that job. Outside of Swarm, Mesos & co appeared too intrusive, and Nomad is quite narrow in what it does. So we picked Kubernetes and are very happy with it.
- yalooze 10y agoI wonder what the reasoning was for having version 2 only terminate instances (vs burning up CPU, taking disks offline, etc.)? I assume it's something to do with what Chaos Monkey is NOT trying to solve (ie. eating up CPU is caught elsewhere by another system and out of scope for Chaos Monkey now). Just trying to think it through...
- jsingleton 10y agoI would assume that terminating is easy via the AWS API, whereas some of the other things need a process on the instance. You shouldn't really be connecting to boxes directly over SSH if you do DevOps correctly, so maybe they blocked port 22 to enforce this.
- moondev 10y agoEverything goes through Spinnaker now, which in turn supports all clouddriver provides including aws, gcp, azure and kubernetes. Resource limits should be set by instance type. It's more of an application level thing than infrastructure which is what chaos monkey is supposed to simulate
- benhoyt 10y agoWhat do you mean by "do devops correctly" to avoid SSH on boxes? (I'm a developer, not devops.)
- devonkim 10y ago"Devops" has bazillions of meanings, but avoiding (human) ssh to production boxes is a generally sound principle these days because our infrastructures are becoming harder to understand by poking at boxes one or two at a time now even for forensic analysis.
- dberg 10y agoSo logging in to a server to check a logfile (assuming i dont or cant do centralized logging) is considered anti-devops ? Edit: Sorry responded to wrong parent, sigh.
- jsingleton 10y agoHas anyone else deployed a Chaos Monkey in production? I can imagine it would be a tough sell to the CEO. :)
- klapinat0r 10y agoHow so? The benefits are worth it, and I doubt any CEO will be argue against having fault tolerant code :) You catch bugs, and no one says you can't run Chaos Monkey in staging or a similar environment if it really is a tough sell.
- takeda 10y agoAgreed, it should be hard to explain benefits even to non technical people. It's like doing a fire drill, if you do it frequently when the actual fire happens you will know what to do. Similarly with infrastructure, it might not be good handling rare events, but once these events are not rare you will learn to handle them. The biggest issue IMO is explaining need to make things more resilient. Actually the technical people (mainly developers) might be the biggest obstacle, because it adds more work for them (with no visible benefit to them, because when application fails it's ops who get woken up).
- birdman3131 10y agoThe drawbacks of potentially causing downtime and therefore having the potential to drive away customers as well as obtain an image of unreliability can be much more damaging than not using it in the first place. Customer image means quite a bit.
- solatic 10y agoChaos Monkey is sort of like Advanced Continuous Deployment. Most shops are still struggling with the basics. You cant even think of trying to sell this running to the C-level until you've proven that you can at least walk (automated deployment and rollback). I remember reading years and years ago about bandit algorithms... this kind of ops work is at a level that's found only in a few different companies.
- aaronblohowiak 10y ago
- andrewguenther 10y agoInteresting that all of the resource burning features have been removed, I wish they had expanded on the reasons why. I always found those to be the most differentiating features of Chaos Monkey. Did they just not get a lot of use internally at Netflix?
- JoachimSchipper 10y agoYes, I wonder that too - complete node failure is the nicest failure that can happen. (See e.g. http://danluu.com/limplock/ http://danluu.com/limplock/, because Dan Luu's site is always excellent.) "We rewrote the service for improved maintainability" seems an important part of this blog post.
- flgr 10y agoI wonder if they have a service that automatically detects all those failures and terminates the entire node instead?
- notyourwork 10y agoI was thinking along these lines as well. Possibly they have other checks and balances in place for less than death situations for servers like high CPU usage and those situations result in another mechanism killing the server which is the same as what this solution provides.
- jpolit 10y agoWe do: http://techblog.netflix.com/2015/07/tracking-down-villains-outlier.html http://techblog.netflix.com/2015/07/tracking-down-villains-o...
- duaneb 10y agoMaybe internally such situations translate to a termination anyway.
- aaronblohowiak 10y agoResource exhaustion manifests as latency or failure. We inject latency and failure using FIT, so we can limit the "blast radius". When you are testing these failure modes, you are more testing the interaction between micro services, and this requires a bit more precision and sophistication. Source: I'm on the Chaos team here at Netflix.
- tomcart 10y agoFirst, a shameless plug for an alternative implementation: https://github.com/BBC/chaos-lambda https://github.com/BBC/chaos-lambda Seems unfortunate that it requires the coupling with spinnaker - although i can see how it helps with the cluster definition features. Edit: I'll add that we've been using the original chaos monkey and chaos lambda extensively in production for some time with very few problems.
- Beltiras 10y agoEvery time this is in the news I get a feeling of awe for operations teams having confidence enough to deploy this. I've usually been in small teams with the feature mill factor turned up way too high.
- justinhj 10y agoI plan to run on it on staging while running load test suites. Not brave enough to run it in production yet
- aaronblohowiak 10y agoIf you don't use a tool like this, entropy will take care of taking your machines down for you. Only then, it won't be a regularly rehearsed part of "normal operations" so you might find yourself up creek without a paddle.
- tormeh 10y agoYeah, sure, but then it won't be the fault of whoever (me?) thought using Chaos Monkey in production was a good idea. Is it good for the organization? Yes. Good for the guy pushing it? Very possibly very not.
- deleted 10y ago[deleted]
- deleted 10y ago[deleted]
- iagooar 10y agoI posted this piece of news to my team Slack at work, and a colleague of mine wrote: "we don't need chaos monkey, we have developers for that". While being funny, it also holds a lot of truth. I guess that Netflix can hire really top-notch devs who do not accidentally force downtime to their software.
- Thaxll 10y agoNetflix had a worldwide outage few weeks ago that last hours so yes they need those kind of tools :)
- agentgt 10y agoThe joke is funny but it is actually shockingly difficult to make nodes kill themselves instead of doing something for more malignant which is the typical case for bugs. That is a zombie just like in real life can be worse than a dead corpse (since we are being funny and all). Chaos Monkey shoots two in the head. I have had issues killing errant JVMs and Rackspace nodes (yes sadly we are still on Rackspace). I can understand why 2.0 is much more focused given the plethora of monitoring solutions.
- asadjb 10y agoThat rings so true. But on the flip side, it might be that because Netflix has a chaos monkey and all it's services need to be resilient to failures, accidental downtime isn't that big of an issue to them. Their developers can still make the same mistakes that we do, but their architecture is better designed to handle that. Just a thought.
- jeremiep 10y agoThe difference is scale. Only a handful of companies run at Netflix scale. I wouldn't trust developers to do what Chaos Monkey does at such a scale, no matter how good you think they are.
- deleted 10y ago[deleted]
- 10y ago
- CorvusCrypto 10y agoFollowing in line with the jokes, it's like they looked at my company's production environment and said "Let's make that a tool!"
- deleted 10y ago[deleted]
- cpeterso 10y agoBeyond external termination services like Chaos Monkey, what are good examples of software that purposely increase internal nondeterminism or failure injection in production? Go has race detector mode, but it is an optional debug feature with a performance cost. The Linux kernel's jiffy clock starts counting from -5 minutes so drivers must handle clock rollover correctly because it's not a uncommon "once every 48 days" event. Firefox has a chaos debug mode that does things like randomize thread priorities and simulate short socket reads, but that has performance costs.
- dsl 10y agoGo intentionally introduces randomness when reading maps so developers don't write code dependent on order.
- cpeterso 10y agoIronically, some Go developers began depending on the randomized map ordering and were surprised when it changed! :) runtime: hashmap iterator start position not random enough #8688 https://github.com/golang/go/issues/8688 https://github.com/golang/go/issues/8688
- caf 10y agoThe Linux kernel also recently added a debugging option that when enabled, instead of just probing a device it performs a probe / remove / probe sequence to ensure that device removal works.
- stgnet 10y agoInspired by chaos monkey, I introduced malloc chaos mechanism into our codebase: https://reviewboard.asterisk.org/r/4463/ https://reviewboard.asterisk.org/r/4463/ Although designed originally to catch places where malloc failure wasn't being handled, it can also be used to randomly trigger other off-nominal portions of the code that might not otherwise be tested.
- kpwagner 10y agoHoly fuck that is some small font size. And the paragraphs aren't in paragraph tags... just hanging out between <div>'s with some <br>'s to keep them company.
- rlau26 10y ago"Chaos Monkey even periodically terminates itself." Whoa, that's meta.
- brianwawok 10y agoJust wait till it starts killing developers to test your bus factor.
- caf 10y agoI can see that Chaos Monkey adds selective pressure to ensure that systems evolve into a state where they can handle unexpected server outages. But isn't there a danger that it also encourages maladaptions that come to rely on being regularly restarted by the Chaos Monkey? I'm particularly thinking that you might evolve a lot of resource leaks that go unnoticed so long as Chaos Monkey is on the job.