3 ms·
I have not used it, but I have heard this is a very useful tool https://github.com/Netflix/chaosmonkey https://github.com/Netflix/chaosmonkey
by jtms 7y ago
I have not used it, but I have heard this is a very useful tool https://github.com/Netflix/chaosmonkey https://github.com/Netflix/chaosmonkey
- colemannugent 7y agoBasically it randomly kills prod nodes and forces engineers to consider this when designing their services. This is one of those "so stupid it's genius" ideas, kinda like mailing DVDs rather than having people to go to a movie store.
- poooogles 7y agoSure, often at a super initial stage the people running the test can just do this manually though. Don't be put off by having to set up the whole suite, a lot of the value from Chaos Engineering can be achieved from randomly removing bits of infrastructure manually (a d6 and a lookup table works fine). The value comes from the learnings as a result of infrastructure being terminated.
- polskibus 7y agoWhat would be advised in the following situation if one wanted to follow chaos engineering principles: - there's a service that needs config data from a DB on another node to initialize itself to become useful - should the service die if it doesn't have connection to the DB on startup (so that the error propagates), or should it start and perform retry indefinitely until DB connection is set? Until that happens it sends back error code to its consumers.
- 8note 7y agoI don't think what to do during the failure case is part of chaos engineering. identifying that it needs to do something, and whether it does it or not is part of chaos engineering. eg by turning off the DB for a bit and seeing what it does