7 ms·
Southwest Airlines halts flight departures amid technology issue
- NegativeLatency 4y agoIs it really fair to blame it on the technology? After a couple of oopsies it seems like it's more of a management issue than a technology issue. I imagine it's a lot easier to deflect blame by blaming it on computers though.
- raverbashing 4y agoWell yeah, going down the '5-Whys' it ends up being management to blame
- fnordpiglet 4y agoThe flaw with 5 whys (not to say I don’t encourage it!) is they reflect the biases of the author while appearing unbiased and authoritative. So, it depends on who writes the post mortem who gets the blame.
- travisjungroth 4y agoYes, because there’s insufficient room for reality. Causality is this big DAG with tons of incoming edges, going back to the beginning of time. 5 Whys is creating a path 5 edges long. This is often useful, but it’s not “the truth”. If each node has 10 incoming edges, there are 100,000 paths you could take. At least it’s better than root cause analysis, which is walking back until you want to stop and declaring that the root. The real root is the creation of the universe. I mean that sincerely. When you follow definitions of causality, time and the universe, you end up with the Big Bang as the root of your graph. Everything is else is editorializing.
- bentcorner 4y ago"In the beginning the Universe was created. This has made a lot of people very angry and has been widely regarded as a bad move."
- darkerside 4y agoTo make an apple pie from scratch, you must first create the universe, as they say
- AnimalMuppet 4y agoA real 5 whys analysis probably should give a tree. Why did X happen? Because A, B, and C. Why did A happen? Because A1 and A2. And so on.
- drivers99 4y agoFishbone diagram is one way to do that https://asq.org/quality-resources/fishbone https://asq.org/quality-resources/fishbone
- bluGill 4y agoYou don't need to do that because there will be more 5-whys. Just pick whichever of A1, A2 seems like a bigger deal and find a B that seems like the biggest contribution - until you get to some deep root cause you can solve and mitigate. Then you fix that one issue and move one. You WILL see more issues, if they are always in other places you must have got the right one, if not then you will see this again and this time you can go down A2. The important part of the above is by refusing create the whole tree you actually get something solved. Sometimes you will get the right one, sometimes you won't. You can spend a lot of time building a tree of problems, and find tens of thousands of things you can fix - and now you have a larger todo list than you have time to do and likely you don't get anything done. Or analysis paralysis in other words.
- fnordpiglet 4y agoThis is specifically where bias comes in. People selectively prune the tree in ways that are either interesting (in a positive take) or deceptively based on their desire to avoid certain topics. Because the tree can be so large it isn’t always obvious even to the author whether they picked a branch because it was useful or because it served some other purpose - divert focus to a team they want to see behave differently, avoid taking blame for something, avoid being asked to fix something they don’t want to, etc. So, in this case the real question is “will they pick the branch that blames the people that pay and promote them, or take the branches that are of the least resistance.” Something this contentious would be vetted by committee in some way or another and at that point any upward finger pointing would be almost certainly curtailed in favor of a less contentious branch.
- Zigurd 4y agoManagement decision frameworks are often like that, but "5 Whys" is relatively innocuous - too simple to make tendentious. Mainly it is a reminder to dig until you hit bottom when doing root cause analysis.
- tialaramex 4y agoNot always. I read Accident Investigation reports for the transport industries in my countries. The vast majority of the time, management are a factor, for example the management of a bulk freighter haven't allowed enough crew for it to be operated safely, or the management of a airport haven't spent money on the improved hardware that would have prevented a dangerous incident. These reports inevitably have recommendations, many of which are directed to management. But once in a while it really isn't. My "favourite" was a fishing vessel which capsized killing everybody aboard. Why did it get into trouble? Because the entire crew were using heroin. That report has zero recommendations because there's nothing to recommend. It's already both illegal and obviously a bad idea to operate vessels at sea under the influence of drugs or alcohol, so there's nothing to recommend, those guys had only themselves to blame for their deaths.
- shuckles 4y agoYou could say it's management's fault they didn't know entire crews were on heroin. This is one of the challenges of being management: every failure is conceivably a management problem.
- tialaramex 4y agoWhat management? Small boat like that, everybody senior is likely a co-owner.
- nobody9999 4y ago>What management? Small boat like that, everybody senior is likely a co-owner. And probably on the boat when it capsized and high on heroin too.
- bertil 4y agoThe technology is not working because of a lack of investment, that much was obvious at Christmas. Factually describing the outage doesn’t remove responsibility; whoever didn’t take the first crisis seriously enough will face it soon enough. Right now, they need engineers, the FAA, flight controllers, etc. to unstuck them. Then there will be a blameless diagnostic. Then heads will roll and I’d be surprised if the most demotion won’t be a general manager.
- Mountain_Skies 4y agoA few weeks after the first big incident, I applied for a couple of open IT roles at Southwest. Though I was well qualified for all of them, I never heard back from SWA, not even a "Thank you for applying" type of email. Makes me wonder if they thought the first incident was a fluke and they could go about business as normal. Also makes me wonder if they put out a bunch of ghost jobs to appear like they were staffing up. Most companies will at least acknowledge receipt of job applications no matter how unqualified the applicant and Southwest didn't even do that much.
- bluGill 4y agoI would be surprised if they can solve all their issues in 5 months no matter now much focus is put into it.
- dopylitty 4y agoIt really does come down to management. There are ways to make computer systems reliable but private companies seldom have the budget, will, organization, or patience to develop their systems to those specs. In lieu of that management should require business resiliency processes allowing the business to continue manually when the computer systems inevitably fail.
- dragonwriter 4y agoI can’t find the exact quote or its source, but one thing I’ve heard that has stuck with me is (probably unintentionally paraphrased): “Every problem blamed on ‘computer error’ involves at least two human errors, one of which is blaming the computer.”
- xd1936 4y ago"At the source of every error which is blamed on the computer, you will find at least two human errors, one of which is the error of blaming it on the computer." — Tom Glib
- LambdaComplex 4y agoAnd Sidney Dekker will tell you that blaming things on "human error" won't actually help you find the root cause of the problem.
- _8j50 4y agoThere is human error which is an error of an employee and there is process flaw which is managent's problem and while they are humans the flaw is not an error it is incompetence, especially when it happens more than twice.
- givemeethekeys 4y agoApparently the last CEO cut a lot of costs in favor of profitability.
- pixl97 4y agoEverything is working, I don't think we need that IT team then....
- markus_zhang 4y agoThis always triggers me to think that it is really difficult to give a score of what a CEO does. Many times they pumped up stocks and quit before problems they created surfaced yet. They were hailed as heroes and the next guy ate shit.
- lancesells 4y agoI feel this is the same way with US Presidents, although it's not really their choice when it comes to leaving.
- bluGill 4y agoEvery country without regular turn over on top seems to devolve to an evil dictatorship over time. Sometimes kings can be forced to give up power, until they become non-evil just for lack of power, but this takes centuries.
- pjmorris 4y ago"A loss of X dollars is always the responsibility of an executive whose financial responsibility exceeds X dollars." - Weinberg’s 'First Principle of Financial Management' and 'Second Rule of Failure Prevention’ - 'First-Order Measurement', Quality Software Management, Volume 2, Gerald Weinberg, Dorset House Publishing, 1993
- ren_engineer 4y agoit's basically inevitable with how Southwest operates, they trade resiliency for efficiency. Same reason modern supply chains are so fragile, minimal redundancy anywhere. Tech seems to be one of the last places where people understand the value of resiliency, the way these companies run would be like not keeping database backups to save on storage costs. Or not having any excess capacity provisioned to handle traffic surges or hardware failures
- VoodooJuJu 4y ago>How many software engineers does it take to screw in a lightbulb? That's a hardware problem. >How many software engineers does it take to fix broken flight software? That's a management problem. >How do you fix this software problem? This question has been closed for being off-topic.
- adrr 4y agoIt's almost never a technology issue. It's an investment and prioritization issue.
- deleted 4y ago[deleted]
- pgrote 4y agoIt appears to be over except for Dallas Love Field Airport. "FAA lifts nationwide ground stop for Southwest Airlines flights after equipment issues" https://www.cnn.com/travel/article/southwest-airlines-flight-delays/index.html https://www.cnn.com/travel/article/southwest-airlines-flight...
- massysett 4y agoWhy does FAA need to impose a ground stop? Couldn’t the airline decide for itself that it’s not going to have its own planes take off?
- bob1029 4y agoThe way the SA business model is set up makes me wonder if this sort of outcome is inherently baked-in, regardless of the actual technology utilized. At the end of the day, if crew A and plane B are in wildly-unexpected places, no amount of computational prowess can correct that reality. Contingencies can be installed at every level, but then you don't have a viable business anymore. E.g. keeping a backup crew and spare plane on the ready 24/7/365 at every airport just in case is not going to scale. It also still won't deal with the worst scenarios.
- Someone1234 4y agoAt the Christmas outage, they had crews and aircraft in the same location and still weren't able to operate them because their system didn't believe they were in that location, and they had too few staff centrally to update the system with that information via phone. This really is a massive management failure. They couldn't re-route crews and aircraft, they couldn't consolidate ones that were already in the same location, and they couldn't fix it because they under-staffed to an absurd degree. Better technology would have helped a ton. But that isn't, to me, the story here. This is an epic mismanagement story that may ultimately kill Southwest. They cost cut technology for years, then cost cut away their backup staff too.
- bink 4y agoAnd if IIRC they couldn't do all this in a reasonable time because their system still required crew to call them on the phone to report location, availability, and to receive new assignments and that system didn't scale during severe weather.
- dhosek 4y agoAirlines generally treat all flights as part of a matched pair: Home-out and back again. Even with a non-hub-and-spoke flight system like Southwest, this is still the case. Crews and (to a lesser extent planes) don’t wander the earth like Cain, they go out and come back (perhaps with a stop or two along the way). Southwest should be more resilient in some ways in that all of their flights are done with the same type of plane, so they can always swap in a different plane for the task, and within the limits of employment contracts/government regulations, they can also easily swap crews around, as long as they keep that out-and-back scheduling.
- el_benhameen 4y agoTheir “whoopsies, sorry we ruined your Christmas” mea culpa email was full of corporate horseshit and essentially no real changes, so this is pretty unsurprising.
- elromulous 4y agoAm I the only annoyed with the use of the word "technology" to mean anything computer related? That would be like saying turbulence is "an issue with physics". A hammer is a type of technology too. Edit: phrasing/typos
- satellites 4y agoThe more vague and hand-wavy it sounds, the better it is for corporate PR statements.
- markus_zhang 4y agoThis makes think, how can a CEO manage a tech team (headed by CTO) effectively? For sure he himself usually does not know much about the tech involved and even he does grow from the trench he doesn't know all arms. The only way he can do is to trust his CTO and let him pitch changes and innovations and such. But how does a CTO do that? Again he has to rely on his generals to make the judgement. This all sounds like a job impossible to do right. Any idea?
- uejfiweun 4y agoThis might sound dumb, but one interesting thing about Star Trek TNG was that Picard and Riker (captain / first mate) were the two foremost computer experts on the ship. It’s explicitly stated that they have the most knowledge of how the ship’s computer works. Perhaps this was prescient. As tech begins to play an increasingly dominant role, the humans who directly manage that tech will play an increasingly dominant role as well.
- brookst 4y agoThat is interesting, but it requires suspension of disbelief to think that the foremost computer expert will also be the foremost management expert. The aptitudes and personalities required for deep understanding of tech rarely present with those required for effective management of large organizations.
- uejfiweun 4y agoYou make a fair point. Unless, of course, the computers become the best managers of all.
- Domenic_S 4y agoSeems implausible that Riker would know more about the Enterprise's computer than Data.
- knodi 4y agoIn these 3 steps: 1) Gather requirements of what the business expect the systems to do provide which capabilities for the business to operate and grow for the future. 2) Present this to the generals and ask them "how can we get here with which changes or new systems?". 3) Analyze document the proposed changes outline pros and cons. Make recommendation to CEO. Repeat the loop again with action plan and timeline, if the CEO agrees which i'm sure the CEO will have input on where action plan is focused on and timelines. Ask for input from all Cxx and SVPs - repeat loop after this point. Repeat until actionable work items are created and assigned to people to execute on. Monitor execution and keep doing it again and again until all the changes are in.
- kilroy123 4y agoIronically, a Southwest Airline recruiter messaged me today. Maybe that's actually a good sign they've realized they need to take their tech stack seriously?
- bink 4y agoUnless you're an attorney.
- adrr 4y agoWhat tech stack? They make their pilots call in to find out where the planes/pilots are.
- halfjoking 4y agoGrandparent should pitch he can fix it in 2 weeks for $1 million. Use vocode.dev to take the pilot calls. Use bot to ask the relevant questions, and get transcriptions. Send transcriptions to ChatGPT to formalize into well-defined json updates to the database. It's a 2 week project, max.
- ratg13 4y agoIf the transcription messes up that flight might not fly .. which means the airline is out a ton of money trying to make alternative arrangements for a flight full of people. Your invention could bankrupt the airline in a matter of months.
- halfjoking 4y agoThat means you can upsell them after they deploy. Have the bot read back the formalized updates to confirm if you want more accuracy. And don't forget premium voices. I'd argue using General Adama's voice from Battlestar Galactica leads to less mistakes. This is Battlestar actual, what's your status? Who won't give accurate information when they hear that?
- adrr 4y ago
- deleted 4y ago[deleted]
- deleted 4y ago[deleted]
- hardtail29 4y agoHey! Do us all a favor and don't post links to stories hosted on sites that won't let you read them until you subscribe. Here is a link to the AP News story which you can read freely as long as you don't mind the ads. https://apnews.com/article/southwest-flights-grounded-faa-fa702529c286c44b45102d8128088deb?utm_source=homepage&utm_medium=TopNews&utm_campaign=position_07 https://apnews.com/article/southwest-flights-grounded-faa-fa...
- elSidCampeador 4y agohttps://archive.is/HWoER https://archive.is/HWoER
- NickBusey 4y agoFrom the HN FAQ https://news.ycombinator.com/newsfaq.html https://news.ycombinator.com/newsfaq.html > Are paywalls ok? It's ok to post stories from sites with paywalls that have workarounds. In comments, it's ok to ask how to read an article and to help other users do so. But please don't post complaints about paywalls. Those are off topic.
- kencausey 4y agohttps://archive.is/HWoER https://archive.is/HWoER
- game_the0ry 4y agoThey'll take tax payer bailouts, buy back stock, cut staff, and pay record bonuses to executives. But fuck investing in the business.
- Nifty3929 4y agoPerhaps we shouldn't do those bailouts...
- game_the0ry 4y agoI concur
- MavisBacon 4y agoI’m in the middle of dealing with this as I’m not a proud nor rich man and always keep crawling back to Southwest. Flying Nashville > Phoenix > Oakland right now. As far as I can tell they just delayed all flights by 30 mins which set things back in place somehow
- sixbrx 4y agoMy wife and I had a recent bad experience on SouthWest, where we boarded the plane at Love Field in Dallas and then were made to wait 2.5 hours on the tarmac, because their IT system had a different passenger count than what the crew counted. After several attempts at counting again, they ended up having to go to each passenger row by row with a manifest in hand, and verify the passenger boarding pass and driver's license. I think aside from their IT system problems, this situation was made worse because as a passenger you don't have an assigned seat, so it's not obvious when someone is missing or a seat has two or more contending passengers.