5 ms·
I want a Chaos Monkey, too! Actually, that was my first reaction, but after thinking for a moment, that isn't really a reliable way to test. If you make chang
by wccrawford 16y ago
I want a Chaos Monkey, too!
Actually, that was my first reaction, but after thinking for a moment, that isn't really a reliable way to test. If you make changes to something, you don't know for sure if the chaos monkey hit while you were testing a certain thing or not. Proper unit tests would seem to be a lot more useful.
- qjz 16y agoIf Chaos Monkey hits during a test, you just run the test again. Multiple tests are a good thing, after all, and the failure might expose a weakness you wouldn't have seen otherwise.
- mickeyben 16y agoHow would you write unit tests to test against services not responding and routing issues between your different instances ?
- jdludlow 16y agoThat's what mock objects are for. You mock the object that would normally call the remote system, have it artificially fail, and use that to test clients of that object.
- samuel 16y agoSo you only test those cases you have thought can fail(which is compulsory to test anyway). The purpose of the Monkey Chaos is to trigger situations you haven't dreamed about. OK, it ain't perfect. Things usually doesn't fail in a uniformaly distributed way and you can't be sure you'll see a different kind of fail on production, but it sounds useful(yeah, and pretty cool) nonetheless.
- joevandyk 16y agoThe problem is "artificially fail". You need to ensure that the systems work in production even after real failures.
- jdludlow 16y agoWhy are people replying to me as if I ever said that Chaos Monkey was a bad idea? I was responding to a question about how to unit test remote system failures. Obviously that's not the only testing you're going to do.
- ghshephard 16y agoSQLite [1] uses a test harness to do this: "SQLite responds gracefully to memory allocation failures and disk I/O errors. Transactions are ACID even if interrupted by system crashes or power failures. All of this is verified by the automated tests using special test harnesses which simulate system failures. " http://www.sqlite.org/about.html http://www.sqlite.org/about.html
- joevandyk 16y agoIt's not meant to be a reliable way to test. Unit tests wouldn't help the problems that Chaos Monkey is meant to solve. Chaos Monkey is meant to force apps to go into failure modes "in the real world". If they see anything unexpected happening after a service is killed or degraded, they can investigate. It sure beats having a service go down unexpectedly for the first time six months ago and not have tested scenario in production.
- SpikeGronim 16y agoUnit tests and fault injection tests catch a different but overlapping set of errors. Chaos Monkey is a terrific idea and I commend Netflix for implementing it early on. If you're not willing to randomly kill hosts then you're not confident that your distributed system really works. This excellent paper[1] by James Hamilton (then MS, now AWS) recommends never doing clean shutdowns on applications - just kill them. Unless you have a lot of persistent state to manage this is a great idea. If it takes hours to migrate multiple TB off your storage host, maybe not such a good idea. 1. http://www.usenix.org/event/lisa07/tech/full_papers/hamilton/hamilton_html/ http://www.usenix.org/event/lisa07/tech/full_papers/hamilton...