3 ms·
Five whys
- deleted 19y ago[deleted]
- daniel-cussen 19y agoWhy not just make an NTSB for internet outages?
- wmf 19y agoAn outage at a Web 2.0 site is unlikely to kill anyone, so it doesn't justify government intervention like the NTSB. Laws have been proposed requiring companies to report on privacy breaches, but I don't think they require companies to say why the breach happened, and they don't require an outside audit. The ROC project suggested something similar, although their analogy was not to the NTSB but an internal Ma Bell policy that required a report to be written for all outages. However, most companies don't want to release details about their outages, especially ones that are not customer-visible. When companies do talk about outages (as Joel just did) it's usually as a form of PR damage control. http://roc.cs.berkeley.edu/#pubs http://roc.cs.berkeley.edu/#pubs
- daniel-cussen 19y agoNow I know. Thanks for the link.
- mixmax 19y agoSome years ago a Major Danish bank suffered a black swan event where their entire online financial system went down. It was traced to a bizarre problem in an IBM DB2 database that had never occurred before. In the entire history of DB2. All kinds of experts were flown in from all over the world to fix it, and eventually they succeeded with some loss of data. The result? Customers of the bank were unable to make any transactions for 5 days, and the bank's stock suffered so severely that the company lost almost a billion dollars in market cap within a week. Now they have redundant sytems...
- daniel-cussen 19y agoDid anybody get fired for buying from IBM?
- mixmax 19y agoOf course not :-)
- pius 19y agoNice one. :)
- run4yourlives 19y agoHa! I bet you IBM didn't even get fired.
- deleted 19y ago[deleted]
- eVizitei 19y agoJoel makes a really good point in this article. We cannot prepare for every eventuality, and in fact it simple isn't cost effective to do so. The best we can do is plan for the things that are expected to go wrong, and make permenant fixes (either with technology or with processes) when things we don't expect come up.
- gibsonf1 19y agoWith EC2 available, why be in the hardware as well as the software business? Life seems to short to have to worry about hardware and hardware scaling - especially things like the story of rounding up 100 servers over the weekend to handle demand when you could just deploy as many EC2 instances as you need in a few keystrokes in a few minutes, or this problem of a misconfigured switch.
- spolsky 19y agowhat do you do when Amazon misconfigures their switch?
- gibsonf1 19y agoIf something like that we're to happen, we'd fire up another instance and reroute - in 3 minutes from our desk.
- noonespecial 19y agoOn Amazons "Black Swan" day, there will be on new instances to fire up. Your current instance and all of its data may well be lost. Your little account will be the last thing they're worried about restoring. EC2 is great for startups, but when the time comes, you need a multi-homed, multi-located setup you've done yourself so you know what risks are involved.
- boucher 19y agoWho's to say that Amazon hasn't deployed a "multi-homed, multi-located" setup, or that yours would be any more reliable than theirs? I don't necessarily care about knowing all the potential points of failure, especially if I believe they employ people who know more about the equipment than I do. You could argue that the only reason to build your own data center is if you believe you can do it better than Amazon, if of course you can match your architecture to their system.
- rcoder 19y ago
- curmudgeon 19y agoAn embarassing outage turned into yet another marketing opportunity... WTG, Joel!