6 ms·
Principles of Chaos Engineering
- taneq 9y agoThis sounds like the kind of thing that would be very powerful if built in from the start, but an absolute nightmare to try and retrofit to an existing system. It reminds me of the Amazon "everything must be a network service" architecture. Painful to institute but with great payoffs later in terms of robustness and scalability.
- ealexhudson 9y agoI know what you mean here, but I have to disagree. I see constant over-engineering in teams, and one of the main reasons for this is copying patterns like this way too early in the life of the business. I see this as a version of Monte Carlo testing - I'm not totally sold on doing it in production, but certainly if you have a load that's difficult to replicate in scale, then this is the type of thing you should be doing. But I don't think anyone should be doing this from the start. This is something that you evolve into. Sure, it's easier to bring it in if you planned it from the start - but as an engineer, you cannot proceed on that basis. You know and understand stuff has to change as you grow, and the art of it is making (more often than not) good calls on architecture to allow that.
- empath75 9y agoWe started our current project running chaos monkey from almost day one, and a few weeks ago, someone fucked up a tag enforcement script on our dev aws account and started shutting down every single instance, while all of our asgs kept restarting them. So basically every single service from consul to kafka to our microservices was down to a few or sometimes even no instances. Because we had already been dealing with chaos monkey, everything recovered by itself within a few minutes of the script stopping, and we barely had any alerts from any services going down. Also, when aws was pushing our meltdown patches, we just ignored all the notifications because we didn’t care about instances being shut down, because they go down randomly all the time. It’s a pain in the ass at first to deal with, but it definitely forces you to fix a lot of potential problems unless you want to spend all your time manually fixing stuff every day. Even just aside from chaos monkey, we build new base amis and push out rolling updates daily. That’ll also quickly suss out problems with not having version numbers locked, etc.
- drdrey 9y agoWhat you can do from the start that will help you down the line is to use a common RPC framework/library across the board, especially if it supports failure injection. It's a shame when you can't easily do chaos testing on a system because it chose to use a different RPC system.
- philipov 9y agoI don't like the name. It's sounds too much like it's the Engineering of Chaotic Systems, but there is no reference to Chaos Theory: https://en.wikipedia.org/wiki/Chaos_theory https://en.wikipedia.org/wiki/Chaos_theory Are they using the formal definition of chaos, or just a colloquialism?
- wglb 9y agoI agree. The article is not clear and I think it uses the term “chaos” where the word “error” might be better suited.
- drdrey 9y agoIn this context, "chaos" means roughly "injecting failures/latency"
- stochastic_monk 9y agoResilient Engineering, then, would be a less buzz-wordy alternative that expresses what they mean.
- wglb 9y agoI would call that "noise" rather than chaos.
- alextheparrot 9y agoThey are using a dictionary definition, I believe. It is “Engineering of Chaos”, which is to describe how computer systems, especially when distributed, can have near random failure states, resulting in what appears to be a chaotic system. Network partitioning, hard drives failing, bits randomly flipping because of physics would all be examples of such failure states. The engineering comes in when we try to simulate such situations and create processes for hardening applications against those types of failure.
- jimnotgym 9y agoI'm not sure I can agree. > Are they using the formal definition of chaos, or just a colloquialism? Is it a colloquialism to use a word in its normal English sense? Chaos as a word precedes Chaos Theory by some thousands of years. If I write a theory and take over an existing word in everyday use, it seems a bit much to accuse every one else of colloquiallism when they use an existing but less strict definition?
- xg15 9y agoVery considerate that they mentioned negative impact for users in the last point. I wonder though how well that is measured and controlled in real-world applications of this design style.
- kostarelo 9y agoI too wonder if any of these experiments actually affected real customers.
- zzzcpan 9y agoNot affecting real customers is kind of the point of controlled experiments.
- drdrey 9y agoYou can very much impact real customers, but the idea is that you should be able to only apply the experiment to a small subset of customers and abort it in case something goes wrong. For instance, you may want to check that the authorization service can function without access to the A/B testing service, so you cut the network connection between them. If authoritazion errors start rising, you stop the experiment and investigate why this happened. You may also find that clients did not retry properly on unexpected errors from the authorization service (e.g. http 500 error codes)
- falcor84 9y agoThis is what an SLA is for, no? If you have otherwise upheld your targets, you can use the remaining slack at the end of the period to run larger experiments such as these, for a long term benefit.
- extrapickles 9y agoThey have certainly impacted real customers. The idea is that ideally these outages can be easily stopped, and your engineers know exactly what the chaos monkey did rather than just seeing an uptick in errors. Basically you are creating a fault at a time of your choosing to prevent the same fault from occurring at the worest possible time.
- kostarelo 9y agoimproper fallback settings when a service is unavailable; retry storms from improperly tuned timeouts; outages when a downstream dependency receives too much traffic; cascading failures when a single point of failure crashes; I don't see how one couldn't replicate those in an environment other than the production. They all involve bringing down complete services and sending some unexpected high load to other services.
- drdrey 9y agoYou can start finding and fixing issues in non prod environments, but eventually you have to run this in prod if you want to find the good stuff. Nothing is going to replicate the particular configuration, load, mix of requests and client behavior that you have in prod
- antoncohen 9y agoThere is no mention of Netflix on the site, but the term Chaos Engineering, and the popularization of the technique, seem to come from Netflix. The Chaos Monkey README even links to this site. https://github.com/Netflix/chaosmonkey https://github.com/Netflix/chaosmonkey https://medium.com/netflix-techblog/chaos-engineering-upgraded-878d341f15fa https://medium.com/netflix-techblog/chaos-engineering-upgrad... http://www.oreilly.com/webops-perf/free/chaos-engineering.csp http://www.oreilly.com/webops-perf/free/chaos-engineering.cs... https://en.wikipedia.org/wiki/Chaos_Monkey https://en.wikipedia.org/wiki/Chaos_Monkey
- rapnie 9y agoi think you are indeed correct on stating that.. the chaos monkey, cool naming :)
- nbraga 9y agoA related Ask HN (from 2016) that I submitted: [Ask HN: Who uses Chaos Monkey in production?](https://news.ycombinator.com/item?id=12749597 https://news.ycombinator.com/item?id=12749597). A couple of interesting responses there.
- voidmain 9y agoIt says something about the engineering maturity of the industry that destructive testing in production environments is among the more sophisticated practices.
- CharlesW 9y agoAssuming perfect systems that never fail seems more immature, no?
- brandonbloom 9y agoThat the industry is finally maturing in to a field that capitalizes on the unique properties of its medium?
- darkerside 9y agoSeems analogous to load testing a physical structure. At first, that probably seemed unnecessarily risky, but in retrospect, you want to build robustly enough that even fairly excessive/destructive loads have no real impact.
- tormeh 9y agoYeah, when hardware failures are expected, simulating hardware failures is no different from simulating trucks driving over a bridge.
- platinumrad 9y agoWhy do you think this is one of the "more sophisticated" testing practices?
- mseebach 9y agoThe chaos monkey is pointedly unsophisticated, but a system that can reliably survive its attack is likely to be very sophisticated. Airline pilots are effectually faced with a chaos monkey in their recurring simulator trainings and as a result have a very sophisticated skill set (indeed, one most of them will never actually need, but train for anyway, for obvious reasons). Also, I take issue with it being “destructive”. Compute resources are effectively free on the margin (at least at the scale that concerns us here) and ephemeral. Killing such a process doesn’t meaningfully destroy anything of value.
- projectileboy 9y agoI’m not affiliated with Netflix, but the negative tone of some of the comments here is hilarious. Would the critics care to share how their traffic/uptime ratio compares to Netflix’s? On weekends, Netflix is responsible for about 50% of the packets on the Internet. On Earth. Netflix is down how often? But please, do share how you once replicated a MySQL database from prod to a staging environment.
- user5994461 9y agoThey're distributing a couple of static videos. It's not rocket science.
- JshWright 9y agoWell, if you add up the various encodings they offer, I think it's like 8 static videos...
- yeukhon 9y agoCooking is not rocket science, but can you cook like a world-class chef? I don’t know about you, but I can’t. There’s science in cooking. Professional work is rocket science hard.
- alayne 9y agoNetflix developed new software architectures to support its products. It's more rocket science than most of what people work on in the industry.
- dang 9y agoPlease don't post shallow dismissals of other people's work. That's emblematic of what we don't want here. https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- marknadal 9y agoChaos Monkey and Kyle Kingsbury's work on the Jensen tests inspired us to make this (https://github.com/gundb/panic-server https://github.com/gundb/panic-server), which lets us simulate all sorts of failure cases. It should be extremely useful to others if they enjoy stuff like Chaos Monkey, etc.!
- vvdcect 9y agoSo did this come about based off Chaos Theory? Then the main goal of chaos engineering would be to "create order out of chaos"?