5 ms·

Or use the Netflix model: Chaos testing in production.

bluecmd 11y ago
Or Google for that matter. DiRT.
toomuchtodo 11y ago
No system is perfect; as you continue to add 9s, the cost increases steeply. Usually its just cheaper to be down for an hour or two, versus architect for the end of times.
- knodi123 11y ago
  > Usually its just cheaper to be down for an hour or two, versus architect for the end of times The opposite of this philosophy was the motivation behind creation of the internet in the first place.
  - toomuchtodo 11y ago
    And yet, we have services who still don't waste the cost on having geographically disperse datacenters.
  - jessaustin 11y ago
    This seems precisely wrong. Some reading: http://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf http://web.mit.edu/Saltzer/www/publications/endtoend/endtoen... https://www.jwz.org/doc/worse-is-better.html https://www.jwz.org/doc/worse-is-better.html [thanks for the hint 'thinkpad20! I don't know what I was thinking.]
    - thinkpad20 11y ago
      Just a note: if you don't indent your links they'll be made clickable by the markup engine, which is convenient in general and especially for those of us on smart phones. :)
    - 3minus1 11y ago
      I think he was referring to ARPANET being a military project whose goal was a system that could survive a nuclear attack or other such calamity.
    - knodi123 11y ago
      It is not precisely wrong, and thanks for tricking me into opening an obscene picture at work, asshole. The internet is designed to be highly fault tolerant, because it was based on an arpanet project to design a network that would NOT go down, even if there was damage to a significant percentage of nodes.
      - jessaustin 11y ago
        The "asshole" in this case is JWZ, [randomly?] switching on the Referer header. Apparently he has a hard-on for HN; he's not the only one, but I won't be linking to his site again. (Although, is that really "obscene"? It doesn't do anything for me?) Try this instead, since Stanford are unlikely to engage in such shenanigans: https://web.stanford.edu/class/cs240/old/sp2014/readings/worse-is-better.html https://web.stanford.edu/class/cs240/old/sp2014/readings/wor... It's funny, my original comment had the links in plaintext so copying-and-pasting was required and Referer wasn't involved. I changed that on request. b^)
  - deleted 11y ago
    [deleted]
aaronblohowiak 11y ago
Part of our Chaos testing in prod is exercising our ability to route traffic around failures of entire regions. jobs.netflix.com

Or Google for that matter. DiRT.

No system is perfect; as you continue to add 9s, the cost increases steeply. Usually its just cheaper to be down for an hour or two, versus architect for the end of times.

knodi123 11y ago
> Usually its just cheaper to be down for an hour or two, versus architect for the end of times The opposite of this philosophy was the motivation behind creation of the internet in the first place.
- toomuchtodo 11y ago
  And yet, we have services who still don't waste the cost on having geographically disperse datacenters.
- jessaustin 11y ago
  This seems precisely wrong. Some reading: http://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf http://web.mit.edu/Saltzer/www/publications/endtoend/endtoen... https://www.jwz.org/doc/worse-is-better.html https://www.jwz.org/doc/worse-is-better.html [thanks for the hint 'thinkpad20! I don't know what I was thinking.]
  - thinkpad20 11y ago
    Just a note: if you don't indent your links they'll be made clickable by the markup engine, which is convenient in general and especially for those of us on smart phones. :)
  - 3minus1 11y ago
    I think he was referring to ARPANET being a military project whose goal was a system that could survive a nuclear attack or other such calamity.
  - knodi123 11y ago
    It is not precisely wrong, and thanks for tricking me into opening an obscene picture at work, asshole. The internet is designed to be highly fault tolerant, because it was based on an arpanet project to design a network that would NOT go down, even if there was damage to a significant percentage of nodes.
    - jessaustin 11y ago
      The "asshole" in this case is JWZ, [randomly?] switching on the Referer header. Apparently he has a hard-on for HN; he's not the only one, but I won't be linking to his site again. (Although, is that really "obscene"? It doesn't do anything for me?) Try this instead, since Stanford are unlikely to engage in such shenanigans: https://web.stanford.edu/class/cs240/old/sp2014/readings/worse-is-better.html https://web.stanford.edu/class/cs240/old/sp2014/readings/wor... It's funny, my original comment had the links in plaintext so copying-and-pasting was required and Referer wasn't involved. I changed that on request. b^)
- deleted 11y ago
  [deleted]

aaronblohowiak 11y ago

Part of our Chaos testing in prod is exercising our ability to route traffic around failures of entire regions. jobs.netflix.com