3 ms·

Their "Chaos Monkey" approach reminds me of an excellent paper on "Crash Only Software": http://goo.gl/dqDII http://goo.gl/dqDII The best way to test the uncom

Their "Chaos Monkey" approach reminds me of an excellent paper on "Crash Only Software": http://goo.gl/dqDII http://goo.gl/dqDII The best way to test the uncommon case is to make it more common.

squidsoup 16y ago
It would be fantastic to see Chaos Monkey open-sourced, but it's probably a very domain specific app that only works in the context of their infrastructure. Some great ideas to be gleamed from the paper you've provided - thanks!
- gfodor 16y ago
  It'd be incredibly easy to implement a v1, a small shell script. AWS has "security groups", which generally are used to split up the roles of machines. (They're used for firewall rules, but in practice having these serve as role metadata is quite useful as well.) Chaos Monkey is just a script that runs "ec2-terminate-instance" commands on node in certain security groups at a certain rate, and boots machines at the same rate. Of course, the devil is in the details, but at a high level this wouldn't be too specific to the domain the infrastructure is built around.
  - somic 16y ago
    The complexity of a system probably in use by Netflix comes from the number of instances and - most importantly - probably the number of accounts that they have. I strongly doubt all those instances they have are running on a single AWS account. Agree that implementing similar system for a single AWS account is not going to be very difficult.
apu 16y ago
Interesting paper, but for future reference, please post non-shortened links here. Thanks!
aw3c2 16y ago
Please do not use URL shorteners when they are not needed. They are obfuscating. Actual URL is http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.90.9585&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.90.... and that is a direct link to a PDF.
- jemfinch 16y ago
  When pg implements proper hyperlinking that doesn't clutter up my text with an ugly URL, I'll stop using a shortener. I understand the positives and negatives of using a shortener, but the positive impact on readability seems to outweigh the negatives when I'm confronted with an ugly link longer than the text I use to introduce it. Sorry if that offends you. I should have noted that my link was a PDF, though. My apologies for that oversight.
  - dpritchett 16y ago
    The HN community has long used footnotes to help with readability [1]. [1] http://www.google.com/search?q=%22[1]%20http%22+site:ycombinator.com&safe=active http://www.google.com/search?q=%22[1]%20http%22+site:ycombin...

Their "Chaos Monkey" approach reminds me of an excellent paper on "Crash Only Software": http://goo.gl/dqDII http://goo.gl/dqDII The best way to test the uncommon case is to make it more common.

squidsoup 16y ago

It would be fantastic to see Chaos Monkey open-sourced, but it's probably a very domain specific app that only works in the context of their infrastructure. Some great ideas to be gleamed from the paper you've provided - thanks!

gfodor 16y ago
It'd be incredibly easy to implement a v1, a small shell script. AWS has "security groups", which generally are used to split up the roles of machines. (They're used for firewall rules, but in practice having these serve as role metadata is quite useful as well.) Chaos Monkey is just a script that runs "ec2-terminate-instance" commands on node in certain security groups at a certain rate, and boots machines at the same rate. Of course, the devil is in the details, but at a high level this wouldn't be too specific to the domain the infrastructure is built around.
- somic 16y ago
  The complexity of a system probably in use by Netflix comes from the number of instances and - most importantly - probably the number of accounts that they have. I strongly doubt all those instances they have are running on a single AWS account. Agree that implementing similar system for a single AWS account is not going to be very difficult.

apu 16y ago

Interesting paper, but for future reference, please post non-shortened links here. Thanks!

aw3c2 16y ago

Please do not use URL shorteners when they are not needed. They are obfuscating. Actual URL is http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.90.9585&rep=rep1&type=pdf http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.90.... and that is a direct link to a PDF.

jemfinch 16y ago
When pg implements proper hyperlinking that doesn't clutter up my text with an ugly URL, I'll stop using a shortener. I understand the positives and negatives of using a shortener, but the positive impact on readability seems to outweigh the negatives when I'm confronted with an ugly link longer than the text I use to introduce it. Sorry if that offends you. I should have noted that my link was a PDF, though. My apologies for that oversight.
- dpritchett 16y ago
  The HN community has long used footnotes to help with readability [1]. [1] http://www.google.com/search?q=%22[1]%20http%22+site:ycombinator.com&safe=active http://www.google.com/search?q=%22[1]%20http%22+site:ycombin...