5 ms·

Chaos Monkey Guide for Engineers

195 points by vinnyglennon 8y ago

deleted 8y ago
[deleted]
empath75 8y ago
Older versions of chaos monkey didn’t require spinnaker and still work perfectly fine. Also, if you’re using this, the right time to start doing it is in development, account wide. It’s mildly obnoxious for devs to lose stuff like Jenkins servers and bastions, but you really quickly find single points of failure and start to engineer around it. Usually making your stuff resilient isn’t that much work, but it’s hard to know what you need to do until you test it. If you’re randomly terminating everything in dev and testing and staging, you should already be battle hardened by the time you get into production.
expertentipp 8y ago
Honestly I've never worked in an environment and with stack mature and scalable enough, with roles within the team so clearly divided, for chaos monkey to be of any use. Hectic deployments, "it's not my job" attitude, feature creep, and a litany of minor errors and misconfigurations were taking its place.
- jedberg 8y ago
  Perhaps if you had introduced Chaos Monkey, those other problems would have gotten solved as a solution to the Chaos (tm). :)
  - marcosdumay 8y ago
    If the GP introduced a Chaos Monkey, he would probably get fired by the same people that are not fixing those problems.
    - tomrod 8y ago
      Corporations are groups of people, policies, and technologies. Of these, people are the hardest to work with.
  - delusional 8y ago
    I think it was Netflix engineer at a conference that once said that you shouldn't run chaos monkey if you expect it to fail. If you don't think it will work, then go fix those places that you know to be rotten first. Your comment was probably in jest, but it provided a nice platform for mine :)
    - jedberg 8y ago
      It was indeed in jest. And there’s a good chance I was the Netflix engineer who told you that. To be fair though when we launched it at Netflix we knew it would break things. But we also knew we had the corporate will to deal with it.
      - twblalock 8y ago
        > But we also knew we had the corporate will to deal with it. That's really the key. If you don't have that, it's not going to work out in the long term.
  - karlkatzke 8y ago
    No. That’s entirely not the way it works. Chaos Engineering works in companies with significant scale and sufficient maturity, and the purpose is to discover design or implementation failures in distributed systems before they happen during big rushes where you’re hopefully making lots of profit. If you lack sufficient maturity or you are not large enough, you will actually lose benefits if you try to implement chaos engineering. It’s like teaching a toddler to use a propane torch. You just don’t do it, or everything you love in the world will burn.
    - karlkatzke 8y ago
      Self reply to realize that I shouldn’t lecture my grandma on how to suck eggs.
      - jedberg 8y ago
        Maybe I should have added a disclaimer with the :) at the end. Thanks for the laugh though. Your phrasing was awesome.
    - empath75 8y ago
      I’m telling you from experience that’s wrong. We did it at a project with just 15 or 20 developers, starting from day one of development. Literally the first thing we installed was chaos monkey before doing anything else, even before getting Jenkins running.
- Swizec 8y ago
  Yeah I don't need a chaos monkey I've got myself.
tayo42 8y ago
something that i dont think gets talked about or addressed enough are partial failures. a whole instance dying is pretty much a solved problem and i think most people are aware of it and how to deal with it. if its not addressed people are just taking their chances at this point. a server being removed is an on and off kind of thing. something sporadic latency or corrupted data can cause some surprising and unpredictable issues.
- jedberg 8y ago
  Chaos Monkey does that too. We used to have a separate monkey called Latency Monkey that induced network latency, because as you aptly point out, detecting if something is down is a lot easier than detecting if it is slow or intermittently down. Netflix has since incorporated those failure modes into chaos monkey: https://github.com/Netflix/SimianArmy/wiki/The-Chaos-Monkey-Army https://github.com/Netflix/SimianArmy/wiki/The-Chaos-Monkey-...
zeeshan121 8y ago
GET LATEST MONTHLY COUPONS AND PROMO CODES AT kouponkabla.com
cuadraman 8y ago
cool logo
deleted 8y ago
[deleted]
smartbit 8y ago
What is a recommended Chaos Monkey guide/method on Kubernetes iow without running Spinnaker?
- dankohn1 8y ago
  Here are several chaos engineering efforts that work with Kubernetes: https://landscape.cncf.io/category=chaos-engineering&format=card-mode&grouping=category https://landscape.cncf.io/category=chaos-engineering&format=...
NHQ 8y ago
I built a similar, much more generic thing, to test networks with various failure modes [0]. It creates proxies for tcp/http/APIs, and handles those streams however you want—cutoff, slow down, dropout, or DIY handlers. Everything can be configured w/JSON, routed by API endpoints, and modulated live through an admin API. Process clustering supported. 0. https://github.com/NHQ/netmorphic-1 https://github.com/NHQ/netmorphic-1
deleted 8y ago
[deleted]
elizabeth_olu 8y ago
guys pls read my blog https://www.elizabethblog.com.ng https://www.elizabethblog.com.ng and also dont forget to comment. Thanks
yavor-atanasov 8y ago
At the BBC a few years ago we completely replaced our Chaos Monkey setup with a "chaos-lambda". It's extremely simple to setup and saves us a chunk of cash in terms of compute and maintenance. I'm sure Chaos Monkey is more featureful, but a lot of times you don't really need these features. If you just need something that ticks on a cron and kills instances at random across your estate and you have a good amount AWS accounts, have a look. https://github.com/bbc/chaos-lambda https://github.com/bbc/chaos-lambda It's always good to see this "chaos culture" being promoted as it drives best engineering practices in terms of architecting, testing and monitoring resilient systems. However it's also interesting to see how at times this space gets inflated, packaged and sold as this big complicated thing that requires "chaos engineers" to implement (almost like "agile" got inflated into a stand alone industry :)). It's just a set of good practices that engineers can do to improve critical systems.

195 points by vinnyglennon 8y ago

deleted 8y ago

[deleted]

Older versions of chaos monkey didn’t require spinnaker and still work perfectly fine. Also, if you’re using this, the right time to start doing it is in development, account wide. It’s mildly obnoxious for devs to lose stuff like Jenkins servers and bastions, but you really quickly find single points of failure and start to engineer around it. Usually making your stuff resilient isn’t that much work, but it’s hard to know what you need to do until you test it. If you’re randomly terminating everything in dev and testing and staging, you should already be battle hardened by the time you get into production.

expertentipp 8y ago

Honestly I've never worked in an environment and with stack mature and scalable enough, with roles within the team so clearly divided, for chaos monkey to be of any use. Hectic deployments, "it's not my job" attitude, feature creep, and a litany of minor errors and misconfigurations were taking its place.

jedberg 8y ago
Perhaps if you had introduced Chaos Monkey, those other problems would have gotten solved as a solution to the Chaos (tm). :)
- marcosdumay 8y ago
  If the GP introduced a Chaos Monkey, he would probably get fired by the same people that are not fixing those problems.
  - tomrod 8y ago
    Corporations are groups of people, policies, and technologies. Of these, people are the hardest to work with.
- delusional 8y ago
  I think it was Netflix engineer at a conference that once said that you shouldn't run chaos monkey if you expect it to fail. If you don't think it will work, then go fix those places that you know to be rotten first. Your comment was probably in jest, but it provided a nice platform for mine :)
  - jedberg 8y ago
    It was indeed in jest. And there’s a good chance I was the Netflix engineer who told you that. To be fair though when we launched it at Netflix we knew it would break things. But we also knew we had the corporate will to deal with it.
    - twblalock 8y ago
      > But we also knew we had the corporate will to deal with it. That's really the key. If you don't have that, it's not going to work out in the long term.
- karlkatzke 8y ago
  No. That’s entirely not the way it works. Chaos Engineering works in companies with significant scale and sufficient maturity, and the purpose is to discover design or implementation failures in distributed systems before they happen during big rushes where you’re hopefully making lots of profit. If you lack sufficient maturity or you are not large enough, you will actually lose benefits if you try to implement chaos engineering. It’s like teaching a toddler to use a propane torch. You just don’t do it, or everything you love in the world will burn.
  - karlkatzke 8y ago
    Self reply to realize that I shouldn’t lecture my grandma on how to suck eggs.
    - jedberg 8y ago
      Maybe I should have added a disclaimer with the :) at the end. Thanks for the laugh though. Your phrasing was awesome.
  - empath75 8y ago
    I’m telling you from experience that’s wrong. We did it at a project with just 15 or 20 developers, starting from day one of development. Literally the first thing we installed was chaos monkey before doing anything else, even before getting Jenkins running.
Swizec 8y ago
Yeah I don't need a chaos monkey I've got myself.

tayo42 8y ago

something that i dont think gets talked about or addressed enough are partial failures. a whole instance dying is pretty much a solved problem and i think most people are aware of it and how to deal with it. if its not addressed people are just taking their chances at this point. a server being removed is an on and off kind of thing. something sporadic latency or corrupted data can cause some surprising and unpredictable issues.

jedberg 8y ago
Chaos Monkey does that too. We used to have a separate monkey called Latency Monkey that induced network latency, because as you aptly point out, detecting if something is down is a lot easier than detecting if it is slow or intermittently down. Netflix has since incorporated those failure modes into chaos monkey: https://github.com/Netflix/SimianArmy/wiki/The-Chaos-Monkey-Army https://github.com/Netflix/SimianArmy/wiki/The-Chaos-Monkey-...

zeeshan121 8y ago

GET LATEST MONTHLY COUPONS AND PROMO CODES AT kouponkabla.com

cuadraman 8y ago

cool logo

deleted 8y ago

[deleted]

smartbit 8y ago

What is a recommended Chaos Monkey guide/method on Kubernetes iow without running Spinnaker?

dankohn1 8y ago
Here are several chaos engineering efforts that work with Kubernetes: https://landscape.cncf.io/category=chaos-engineering&format=card-mode&grouping=category https://landscape.cncf.io/category=chaos-engineering&format=...

NHQ 8y ago

I built a similar, much more generic thing, to test networks with various failure modes [0]. It creates proxies for tcp/http/APIs, and handles those streams however you want—cutoff, slow down, dropout, or DIY handlers. Everything can be configured w/JSON, routed by API endpoints, and modulated live through an admin API. Process clustering supported. 0. https://github.com/NHQ/netmorphic-1 https://github.com/NHQ/netmorphic-1

deleted 8y ago

[deleted]

elizabeth_olu 8y ago

guys pls read my blog https://www.elizabethblog.com.ng https://www.elizabethblog.com.ng and also dont forget to comment. Thanks

yavor-atanasov 8y ago

At the BBC a few years ago we completely replaced our Chaos Monkey setup with a "chaos-lambda". It's extremely simple to setup and saves us a chunk of cash in terms of compute and maintenance. I'm sure Chaos Monkey is more featureful, but a lot of times you don't really need these features. If you just need something that ticks on a cron and kills instances at random across your estate and you have a good amount AWS accounts, have a look. https://github.com/bbc/chaos-lambda https://github.com/bbc/chaos-lambda It's always good to see this "chaos culture" being promoted as it drives best engineering practices in terms of architecting, testing and monitoring resilient systems. However it's also interesting to see how at times this space gets inflated, packaged and sold as this big complicated thing that requires "chaos engineers" to implement (almost like "agile" got inflated into a stand alone industry :)). It's just a set of good practices that engineers can do to improve critical systems.