11 ms·
The most annoying part of bugs like this, is that we rarely get a good post-mortem. To frequently the media just reports it as "a computer glitch". You would th
by mrweasel 10y ago
The most annoying part of bugs like this, is that we rarely get a good post-mortem. To frequently the media just reports it as "a computer glitch". You would think that with computers being so important to modern day life, we should get better answers.
I'm not suggestion that the bug should be explained in details in the evening news, but at least give some indication of the nature of the problem. They could just refer to a website, if people want the technical details.
When a car has a problem, it's not reported as a "glitch". Imagine the stupidity of a media report claiming "A glitch in the latests Ford models cause several drivers to crash". You don't do that, you explain that it's a fault in the steering, breaks, electric systems, or where ever the issue lies. I think we should reasonably expect the same when errors in computer systems hit the news.
- xixixao 10y agoI'm afraid that's because the target audience has (on average) that level of technical knowledge (in both cases)...
- planetjones 10y agoOne of the main reasons that you don't get a good post mortem is that it's often directly related to human error or a severe defect brought about by a decision to cut costs. There have been several huge outages in UK banking over the past few years and management have a convenient excuse in blaming "legacy systems". While I respect private enterprise, for systems of systemic importance I would like to see some legislation to implement a mandatory and detailed post mortem like you propose.
- mrweasel 10y ago>it's often directly related to human error or a severe defect brought about by a decision to cut costs I wonder if companies do cost-benefit analysis on those kinds of situation. "We saved X amount of £ on cutting costs. Outages as a result of cost cutting Y amount of £". You would think that at least stock holders would insist.
- di4na 10y agoNo because doing that would mean knowing what happened and spending time doing a post mortem.
- pc86 10y agoThis may come as a shock but most people want to do a good job, even if they work for Citi or Delta or Comcast or where ever. Okay, maybe not Comcast.
- di4na 10y agoI never said the contrary. But a tons of these systems are old, and people knowing how they work are long gone. Plus doing a proper post mortem and understanding how it works woudl take a lot of time and money. Things they most of the time to not have budget for.
- oedenfield 10y agoIn a past life I was an IT consultant and good organizations do ask this question. Sometimes you have to ask the question like that to get business leaders (or board members) to pay attention to that "new IT spending".
- user5994461 10y agoI love doing that =D A while ago (when I was in aerospace) there was one senior people who gave us the price of insurance per head. Given that we're talking about 1M€ per head and that a plane can easily have 200 people. I have a hard time finding a situation where I can expect a Return On Investment by doing things poorly and cheaply. We're talking many millions of payout. (And yet, I'd love to have such a situation, it's always good to attract the attention of attendees at conference or student classrooms). Then later I realized that I've made a mistake in my calculation... I'm only accounting for A SINGLE plane failing and killing everyone. In the real world, a bad component would be shipped to a whole batch of planes and they'd fall randomly like flies. That batch-effect drastically increases the cost of failure. So... I really don't see ANY place where there is a ROI on people dying. BAZINGA!!! I'm from Europe and people are expensive there. The trick is to manufacture things for the 3rd world market where people get peanuts when they die (sometimes even nothing) :D But then again... we're back to square one, how could I build parts AND guarantee that they're only shipped to planes in low-cost locations? At that point, it's probably closer to a plane-by-plane targeted sabotage rather than a scheme of doing minimum-shit-to-close-sales. And when doing plane-by-plane there is no economy of scale :( ... Anyway! Just wanted to say that any analysis would clearly reveal that THE PLANE IS CHEAPER WHEN IT FLIES!!! (and your life is truly expensive) so you're fine. You can keep on using airplanes ;) Also, people at work do have a conscience. Once in a while they may forget about the importance of what they do, luckily there is a plane crash happening every few months to remind them that there are lifes at stake, for real (in case it was ever forgotten).
- nkozyra 10y agoAnother big reason is security - the less a nefarious actor knows about a system, the better (at least from the perspective of a large, closed organization). Now we know this is not always true, but even giving out details about schema, data-organization or technologies used could provide more detail than is necessary. We could argue the merits of this approach - I can certainly sympathize with the idea - but security by obscurity is still a rampant ethos in huge corporations. I wouldn't be surprised if TSA also enforced a bit of this as well.
- TuringNYC 10y agoThis makes sense in a vacuum, but it is pretty easy to figure out any decent sized company's tech stack just by browsing their IT group's LinkedIn profiles.
- motxilo 10y agoGood postmortems should be created for human-induced errors or top-down decision to cut costs. Especially for those.
- mseebach 10y agoYou don't need a post-mortem when you already know what the problem was.
- motxilo 10y agoIf you knew what the problem was before the outage happened, and didn't consciously put the effort to prevent it, then oh well, that's professional negligence. Maybe not on you because you may not have the decision power to apply large-scale changes, but definitely somebody within the company. And if you didn't know what caused the outage, and you conducted some type of investigation to get at the heart of the problem, then you need a postmortem. That's one of the main goals of a postmortem: document what the cause of the problem was, not just for you, but for all interested parties. If some human caused the issue, you need to document what happened and what should be done to prevent it from re-occurring. And fix it, of course.
- rpazyaquian 10y agoCan't we all just assume from now on that anything bad that happens is because of some shitty CEO cost-cutting everything to line their pockets? Because 99% of the time that's what's happening.
- mseebach 10y agoDelta being down for four hours is not an issue of "systemic importance". Safety-related issues are already (and rightfully) publicly dissected, but it's hard to see what a post-mortem showing (what is likely to be) the interplay of a couple of subtle and individually benign problems which when hit by the power outage became a big issue will do much except enable a large chorus of armchair experts to pronounce on how this is obviously the fault of evil cost-cutting management/because they didn't use the cloud/because they used the cloud/fixed in the latest version of node.js.
- planetjones 10y agoI did not say that Delta being down was of "systemic importance". However a bank being unable to send or receive payments for a week is of systemic importance; hence why I would like a regulator to decide what is of systemic importance e.g. banking, utilities, telecoms. We are getting more and more where broadband is out, banking is out, etc. and the root cause is never transparent.
- pc86 10y agoIf a bank's systems are down consistently, use a different bank. Where public safety is not a concern, it is not any of your business as a private citizen what the inner workings of a private enterprise are, including the cause of outages even if they affected you directly. That being said, of course I am much more likely to patronize businesses who do give me this type of information. Given a choice between a bank with satisfactory service and amazing transparency, and a bank with great service and no transparency, I will choose the former every time. But government regulators opening up the working of private businesses is not a proportional response to your bill pay being unavailable for a few days.
- digler999 10y agoA ticketing system may not be "safety" critical, but its malfunction can cause tremendous discomfort and financial strain on people's lives. Being stranded far from home, missing an important event, or being put through hours of incredibly stressful situations is inexcusable.
- cbsmith 10y agoFinancial outages get punished by investors anyway, and they all want to keep their competitors from knowing how they operate, so I'm not sure there is a strong interest in getting this right. In the case of airlines though, they have a strong collective interest in avoiding incidents.
- cddotdotslash 10y agoThis will likely never happen since it puts the offending company in the position of having to publicly air their internal problems, whether it's a lack of proper power backup, inadequate staff, etc.
- afarrell 10y agoMost people know what steering, breaks, or car suspensions are and a large chunk of the general population has a mental model of how they work. After the power outage of 2003, the NYISO released a post-mortem [1], but I don't remember CNN reporting on it. This is probably because it takes a significant quantity of effort to turn a detailed report into a 10-minute video. I imagine that a youtube channel that ran post-mortems on large-scale systems would be popular. [1] http://www.nyiso.com/public/webdocs/media_room/press_releases/2005/blackout_rpt_final.pdf http://www.nyiso.com/public/webdocs/media_room/press_release...
- quantumhobbit 10y agoDoes anyone know if a good resource of detailed postmortems even if really old? I love reading these but they are often way too short on details.
- gry 10y agohttps://github.com/danluu/post-mortems https://github.com/danluu/post-mortems
- quantumhobbit 10y agoAwesome. Thanks.
- privateersman 10y agoTechnology seems to be a black box for journalists, including technology journalists. I'm expecting them to start using words like 'boffin', in addition to 'glitch' to show that they have given up trying. That's the language of tabloids (and outlets that do a tip-of-the-hat to tabloids).
- deleted 10y ago[deleted]
- tedmiston 10y agoMarginally better: The official post cites a power outage in Atlanta. > A power outage in Atlanta, which began at approximately 2:30 a.m. ET, has impacted Delta computer systems and operations worldwide, resulting in flight delays. Large-scale cancellations are expected today. All flights en route are operating normally. We are aware that flight status systems, including airport screens, are incorrectly showing flights on time. We apologize to customers who are affected by this issue, and our teams are working to resolve the problem as quickly as possible. Updates will be available on news.delta.com. http://news.delta.com/more-flights-resume-delays-cancels-continue-after-power-outage http://news.delta.com/more-flights-resume-delays-cancels-con...
- hueving 10y agoIt's true that the power outage was technically in Atlanta (because that's where their data center is), but it's misleading because it was only their data center that went down. The utility company claimed no other customers had outages.
- Apofis 10y agoThis is nearly a completely different story. Ace reporting.
- pjmorris 10y agoBy way of contrast, the National Transportation Safety Board does a pretty thorough investigation of aircraft crashes and records results in a public database [0]. The software industry has, so far, managed to avoid this level of scrutiny and liability. The jaded part of me says that it will take a crisis to get software to be taken that seriously, but the crises so far haven't been large enough. [0] http://www.ntsb.gov/_layouts/ntsb.aviation/index.aspx http://www.ntsb.gov/_layouts/ntsb.aviation/index.aspx
- dopamean 10y agoThis is a really interesting point. I think when software starts killing people 100 at a time (like a plane crash) we'll get that kind of oversight.
- pjmorris 10y agoAs the other poster indicates, right now it's more like software wounds by the thousands or millions, and, so far, that's not been enough.
- TuringNYC 10y agoThere have already been crises, it is just that some crises are more subtle (though still quite devastating.) Consider the crisis of the Washington Mutual post-merger system integration. They purchased dozens of banks in the hopes of becoming a super-bank, but the details after the deal closes are often neglected. I was a Dime Savings Bank customer. My beneficiaries (upon death) on the account magically got converted into account holders, though only in the 1099 system, not on the reporting/statement system. We didn't even know until the next year when my brother's student aid was declined -- they noted he had ~$17,000 in his bank account, which was curiously about how much I had. He didn't even have a bank account! Something like this can literally kill a semester/year of college if you arent savvy enough to deal with it. Luckily for us, tens of thousands of other customers had the same issue and the school was inundated with similar cases -- they realized something was amiss. Incidentally, Washington Mutual refused to help -- they noted that the statement did not show my brother's name. They also noted that they cant talk to my brother since he isnt a customer!!!
- PaulAJ 10y agoIts usually a complicated interaction involving old systems, new systems, unusual circumstances and an obscure bug. Delta Airlines was started in 1924, so its a fair bet they've been using computers since at least the 1970s. Environments like that tend to accumulate legacy systems, because every time some new requirement comes along (teletext boxes in travel agents? the Web? Passing data to Homeland Security?) its easier to add a new box on the side instead of re-architecting the existing legacy. So the result, after 4 or 5 decades, is not like a "latest Ford model", its more like Johnny Cash's Cadillac in http://www.metrolyrics.com/one-piece-at-a-time-lyrics-johnny-cash.html http://www.metrolyrics.com/one-piece-at-a-time-lyrics-johnny.... Want to bet that car was stable during an emergency stop in wet conditions? Its easy for us on the outside to say "well, have a backup" or "just replace all the legacy systems", but in the real world its not that simple. Yes, you can have a backup server, but it has to be running exactly the same software and working on a copy of the data. If some unforeseen event corrupts the database, its likely to corrupt the backup replicant in exactly the same way. Also its likely that some of the legacy systems are not designed to support replication and can't have it retrofitted with any reasonable degree of cost and risk. Replacing a single legacy system with another system that does exactly the same thing is expensive, risky, and wins you almost nothing. You have to replace a large chunk of the whole legacy all at once, without disrupting your day to day business. This is hard to do; any project to try this is likely to fail before it really gets started.
- stcredzero 10y agoReplacing a single legacy system with another system that does exactly the same thing is expensive, risky, and wins you almost nothing. You have to replace a large chunk of the whole legacy all at once, without disrupting your day to day business. This is hard to do; any project to try this is likely to fail before it really gets started. The way to do this is to create an automatic translation tool. Furthermore, this translation tool can't be generic. Instead, it has to map the idioms of the legacy project to new code in the idioms of the new environment. Done properly, doing the port at one level of indirection like this, you can absorb all of the changes to the legacy system as they are happening, then instantly switch over when the new system is finally ready.