38 ms·

How Southwest Airlines melted down

283 points by wallflower 4y ago

nthitz 4y ago
https://archive.ph/J3pFF https://archive.ph/J3pFF
francisofascii 4y ago
So they are blaming SkySolver software. The article says it is off-the-shelf software? But in other news reports, they make it sound like it was developed in-house.
- phpisthebest 4y ago
  Alot of enterprise software is both, it is more akin to a "super framework". Where you start with the base system that provides commons functions but then it extendable. Most ERP platforms are like this. over the decades it is not uncommon for little of the original platform to be used. My org.s ERP is like that, we use probably 10% of the commercial code, and 90% of functions are custom in house written code.
  - ghaff 4y ago
    Even a SaaS poster-child like Salesforce is like that--between partners and in-house development of various sorts. I think it may be the largest tech conference these days and it's not because everyone buys a standard Salesforce subscription.
  - partdavid 4y ago
    Often, even more complicated: the ERP software vendor is one party; then there's the consulting VAR who does the customer's "customizations", and then the in-house team which does the "administration" and some "configuration". The VAR may or may not have a continuing relationship and of course the three groups have ample opportunity for finger-pointing; and the "administration" team is often under-resourced and doesn't really have the right skills to continue or maintain the work. It's worse than a framework because most frameworks are designed for people to do some form of software engineering with. But with some "configurable" tool, you end up with something that you need to do software engineering but it doesn't have the tools, power or affordances to do so. I've done some Jira consulting and worked in MDM and I'm of the firm belief that "buy vs. build" isn't the real dichotomy: it's "build" vs. "buy and build", and the effort of building on top of the platform is usually underestimated and undervalued.
    - phpisthebest 4y ago
      We have a team of in house devs that do all of our customization but yea out-sourcing is still all the rage with many other companies VP's looking for that next bonus... Companies are allergic to headcount
- kube-system 4y ago
  If it's anything like an ERP, then that sounds about right. There's the out of box part that does the most generic things, and then heavy customization to integrate and customize it to fit the particular business.
- dragonwriter 4y ago
  > The article says it is off-the-shelf software? But in other news reports, they make it sound like it was developed in-house. Working in enterprise, there is a lot of “modifiable off-the-shelf” software; ideally, this provides basic functionality out of the box, but also the alignment to custom business needs of in-house software. (Often, it seems like it combines the up-front and ongoing external licensing cost of COTS software, with the internal development and maintenance costs of in-house software, and the combined problems of both.)
- chris_wot 4y ago
  It probably was - massive customizations would have had to have been done to make it fit for purpose. Good luck replacing it, on an operational airline that can’t shutdown. How you would do this is to me, just mind-boggling.
  - hellcats 4y ago
    AA does shut down for about 5-10 mins to run their solver and upload new assignments after a major event.
- Beltalowda 4y ago
  > So they are blaming SkySolver software. The article says it is off-the-shelf software? The article actually says "SkySolver, an off-the-shelf piece of software that Southwest has customized and updated".
- billsac 4y ago
  So what year was it introduced? Is GE still in the software bix? Who knew.
colechristensen 4y ago
There really should be laws or regulations that fine airlines for this kind of behavior and compensate passengers. Reliability failures that have that much impact on people need more consequences than just seeking another airline next time.
- toomuchtodo 4y ago
  There are, but more importantly, as critical infrastructure you should have to demonstrate a BCDR (business continuity disaster recovery) plan exists and that you can perform it from a cold start and under extreme system stress.
- luma 4y ago
  One would hope that the market could work this out, and when people went looking for options, would keep this in mind. Instead what people do is find the lowest fare possible, all other considerations be damned. It’s not like this is the first time this has happened to SW.
  - munificent 4y ago
    > One would hope that the market could work this out Southwest did lose 10% of its market value overnight, so it seems the market is punishing them. Given how much executive compensation is in stock, a lot of senior level folks at Southwest are likely feeling some pain (or, least, as much pain as any very rich person ever feels). > Instead what people do is find the lowest fare possible, all other considerations be damned. That's an over-simplification. Yes, passengers are very price sensitive. But whenever I talk to friends and family about travel, it's become increasingly clear that they factor predicted quality of service in to their choice. I don't fly United and try to avoid American. I'm certain that many many people will hesitate to fly on Southwest after this. People aren't stupid and no one wants to roll the dice on an entire vacation just to save twenty bucks.
    - Spivak 4y ago
      And people will buy the dip, the news cycle will blow over and in a year the market won’t remember this at all.
lowbloodsugar 4y ago
They rely on telephones???
- phpisthebest 4y ago
  They relied on hand written notes for baggage until 2017... 2017... This is going to be a prime example CIO's can use as what happens when a company fails to invest in IT, treating as a cost center to alway gut budgets from Somewhere there are Sysadmins, and Devs at southwest just face palming at the years and years they have beggged for resources to improve things
  - eightysixfour 4y ago
    It isn’t just tech problems that drive this kind of stuff, SWA couldn’t get a deal worked out with AMFA to use bag scanners until 2016. > Southwest said it will begin scanning bags on the tarmac this year to improve accuracy. Other large U.S. airlines already use this technology. Southwest's contract with bag handlers did not include a provision for using scanners until 2016. The airline then spent time evaluating technology and testing scanners at a few airports. https://www.keranews.org/news/2019-02-20/southwest-airlines-grapples-with-new-labor-and-revenue-problems https://www.keranews.org/news/2019-02-20/southwest-airlines-...
    - ComputerGuru 4y ago
      That's obviously not a blanket case of "AMFA bag handlers categorically refuse to use bag scanner tech" but rather "SW, unlike other airlines, didn't negotiate (early enough) the use of bag scanners in its contract with AMFA bag handlers." SW decides what their priorities are when negotiating these contracts and deciding the terms. Other airlines obviously prioritized bag scanner tech and made the necessary tradeoffs to make it happen while SW didn't. Half the blame at the very least should be apportioned to SW.
      - eightysixfour 4y ago
        I didn’t say it was categorically anyone’s fault, I just wanted to illustrate that airlines have unusual constraints on them so assigning blame to any one party is hard. The limitation are unusual compared to what we experience in less regulated, less unionized industries.
bottlepalm 4y ago
Anyone else work at a large company with a massively complex system that basically controls everything, and a skeleton crew of devs that know how it all works?
- missedthecue 4y ago
  Wait, you have people that know how it works?
  - bottlepalm 4y ago
    Well not exactly, each dev knows their piece of the system. Impossible to understand it all.
  - pphysch 4y ago
    Working on it. What gives me a kick is the longtime sysadmin in my department who grumbles when I discuss introducing new software because "who will maintain it?" Buddy, no one even understands the current software (except me a little bit)! If I end up leaving a Ball of Mud behind, it is in total compliance with the organization's engineering practices.
- phpisthebest 4y ago
  I did, then all those people retired now we just have massively complicated system that controls everything....
- csours 4y ago
  https://www.youtube.com/watch?v=kh9j1slA7R4 https://www.youtube.com/watch?v=kh9j1slA7R4 (Ralph Wiggum "I'm in danger") Yup.
- luma 4y ago
  Nope, we did but PE bought the outfit and replaced that crew with offshore contractors before selling it off to the next guy. Now nobody knows how it works.
  - 4y ago
candiddevmike 4y ago
Their microservice rewrite will be ready any day now
- theCrowing 4y ago
  But, but since we started the rewrite 123123 better JavaScript frameworks were released... we should rewrite the rewrite.
  - thepasswordis 4y ago
    Okay cool, we're ready to go! Does anybody know why it's complaining about node versions, though?
    - partdavid 4y ago
      I think that means we need to use Rust to target WebAssembly...?
- Tao3300 4y ago
  Sorry, the rest of the microservice rewrite has to be delayed so my pet feature can be added. Maybe if you hurry you'll be able to pick between more of the rewrite and resolving crippling tech debt from the last time.
nostromo 4y ago
The actual answer is buried at the end of a long article. > Unlike many rival airlines, Southwest’s planes generally hop from one city to another, rather than orbiting a major hub. That approach lets Southwest maximize use of its planes and crew, but the daisy chain structure also makes its network more delicate—problems in one corner of the country can be difficult to contain
- masklinn 4y ago
  I mean... it's not entirely untrue but at the same time airlines have been winding down their "hub and spoke" model for a point to point one for a while. That's in part what doomed the A380, which was popular with airlines still going strong with hub-and-spoke (Emirates being by far the most prominent one) but is worthless in a point-to-point model.
  - ChrisMarshallNY 4y ago
    > the A380 My worst nightmare, was pulling into a gate, and seeing one of those puppies arriving on the same concourse. Immigration queues are bad enough, with 747s, but the A380 is much worse.
    - ikrenji 4y ago
      yeah well there is no reason why an entire A380 is being processed by two customs officers while 8 extra booths stand unused
  - nradov 4y ago
    There are still a number of major airports that are slot constrained, especially now that passenger demand is growing again. It seems like there could be a profitable market for an efficient twin-engine airliner larger than the Boeing 777X. Perhaps even a double-decker?
    - masklinn 4y ago
      > There are still a number of major airports that are slot constrained Major airports are major airports even in a point to point system. If everybody wants to go into or come out of NY, NY's airports are going to be slot constrained. Also when I say "winding down" I don't mean "killed entirely", most airlines still have home bases, and mix point to point with regional hubs (especially for international flights). But at the height of the hub-and-spokes model, unless you departed from or went to a hub you'd always need a layover.
burlesona 4y ago
It’s fascinating that the same hopscotch travel pattern that allows SWA to offer better service to more places is also what caused the network to suffer cascading failure. Once a critical mass of pieces (planes/crew) were out of position the whole network fell apart, and it’s large enough that it seems like neither the humans nor software can easily reason about how to resume operations. Hence the need for a “full system reboot” over many days. Anecdotally, I flew Southwest just before Christmas. The network was already buckling and we had major delays, but we were lucky and made it through. Despite the stress, the SWA crews were helpful, empathetic, and polite. They handled it better than I would have if I had been in their shoes.
- firstSpeaker 4y ago
  I imagine the system cannot account for where all the planes, staff, passengers are and where they want to go economically.
  - bumby 4y ago
    According the the article, the system makes some non-sensical tasking: "In one example during the storm, the system assigned a pilot to deadhead on a flight from Baltimore to Manchester, N.H., and then back to Baltimore the next day, without ever flying a plane"* * The article defines deadheading as sending a pilot as a passenger to get to another location. It would be interesting to look at what the system is trying to optimize for to make such choices.
    - brazzy 4y ago
      Sounds to me more like a bug in an edge case rather than trying to optimize for the wrong thing. Or (more likely, I think) the first deadhead was planned to have the pilot take over a flight which was then cancelled after the pilot was already underway.
      - smaudet 4y ago
        Also sounds like a case of not throwing enough adversarial data at the system - you can't just code coverage your code, you can't even establish KPIs, you have to establish its performance under system failure (does it freeze or gracefully shut down, does it persist to disk, what happens when the disk is yanked out of the system), etc. Very few software shops I am aware of that do anything like this.
LetsGetTechnicl 4y ago
It's good to see that the billions of dollars in federal bailouts they received during the pandemic was put to good use and not given to executives with salaries already in the millions of dollars and stakeholder dividends... oh wait...
- cratermoon 4y ago
  [flagged]
  - throwaway9191aa 4y ago
    That article says “We recently had the opportunity to incorporate Employees’ requests to enhance campus. We converted a basketball court into three pickleball courts – a growing request by Employees over the past few years."
    - cratermoon 4y ago
      Reading a news article planted by the PR department of a big company is a skill. I can understand how this reads like "we're doing this for our people". This is just "the execs wanted to play pickleball so they had courts put in" dolled up to sound like big, happy, community-oriented charity.
      - Tao3300 4y ago
        "which would you like? (A) Pickle ball courts (B) Nothing"
        phist_mcgee 4y ago
        What the hell is pickleball?
        moistly 4y ago
        Some sort of paddles-and-ball game that afaik is renown for its (a) endless, obnoxious, loud “THWOK! … THWOK! … THWOK!” sound and (b) obnoxious, self-entitled players. Seriously, I have never heard anything good about it from anyone who does not play, only controversy and upset from neighbours and other potential users of the court.
Spivak 4y ago
None of this actually explains what happened. Okay they have an off-the-shelf product called SkySolver that they use to manage their flights and it’s old (I guess the traveling salesman problem has really changed since 1930) and it couldn’t handle the sudden change in resource availability and flight constraints. But… why? What actually happened?
- doodlesdev 4y ago
  Some anecdotes here https://www.reddit.com/r/flying/comments/zw5lsl/comment/j1tne9z/ https://www.reddit.com/r/flying/comments/zw5lsl/comment/j1tn... Essentially the system never know if a flight was successful or not, it just assume it went well, in the case of a delay or a change some employee has to input data manually into the system. For a few flights that's ok, but for a exceptional event like this the problem snowballs and now the system doesn't know were planes, crew and passengers are. Quoting the top comment by 4Sammich here for those who refuse to touch Reddit: > I have friends in CS and the hotel assignment side too. There were 2 specific problems, the software for scheduling is woefully antiquated by at least 20 years. No app/internet options, all manual entry and it has settings that you DO NOT CHANGE for fear of crashing it. Those settings create the automated flow as a crewmember is moving about their day, it doesn’t know you flew the leg DAL-MCO it just assumes it and moves your piece forward. > In the event of a disruption you call scheduling and they manually adjust you. It does work, it just works for an airline 1/3 the size of SWA. > So the storm came and it impacted ground ops so bad that many many crews were now “unaccounted” for and the system in place couldn’t keep up. Then it happened for several more days. By Xmas evening the CS department had essentially reached the inability to do anything but simple, one off assignments. And to make matters worse, the phone system was updated not too long ago and it was not working well. > Last nite they did a web form and had planned to get the system up as much as possible with what communication they could muster, however it was too much to keep up on and ultimately the method for tracking crews failed again. > This 100% is at the feet of all management who refused to invest in technology updates because it is the southwest way to be stuck in 1993. Heck, they still do 35 min turns on a -700 and 45 on an -800 frequently with only 2 man gates. But the good news is HDQ has a pickle ball court now. > Edit: I just realized I never added the 2nd issue. Staffing. When the weather hit all those stations at once the ramp crews had to work in shifts to not become injured due to the cold. That slowed down the turns and backed up the planes. Many many ramp staff quit because of the management harassment (Denver) and just over it. So many rampers are new and making around 17/hr. Once they lost so much staff the crew scheduling software inputs couldn’t keep up because CS is also woefully understaffed and it became what we have today.
juujian 4y ago
Insane that the US has put all its eggs into one basket. One bad event and interregional passenger travel completely breaks down. Looks like it would be wise to invest in a more resilient mode of transportation.
- Someone1234 4y ago
  This is one airline out of several. Also, what other mode of transportation doesn't use scheduling software? Certainly not rail or inter-state buses, they have very similar headaches.
  - bombcar 4y ago
    And indeed multiple competing companies with planes is going to be more resilient - as each flight is dependent on the airports at each end and not much more, whereas a road or rail method has to have all the interconnecting working.
- kube-system 4y ago
  We're talking about the 4th largest airline in the US.
  - dehrmann 4y ago
    https://en.wikipedia.org/wiki/List_of_largest_airlines_in_North_America https://en.wikipedia.org/wiki/List_of_largest_airlines_in_No... Very coincidentally, their fleet size is 737.
- leguminous 4y ago
  What would be a more resilient mode of transportation? Few people would take a train from Chicago to Miami, for example. That's something like 1200 miles (2000km). Traveling from LA to NYC is like traveling from Lisbon to Moscow (2500mi/4000km by air). The US should definitely have more rail--and higher speed rail--but that is only practical in select areas with enough population density. Many of the people in those areas could drive to where rail would service.
crisdux 4y ago
I don't buy the narrative that inadequate technology is the main reason for the Southwest debacle. We must ask, why did this happen now and not before? Southwest has previously been able to better deal with disruptions like this. While the weather event did happen in the middle of their network, it wasn't unprecedented. I think a more obvious reasons is because of staffing issues brought on by covid, layoffs, and the vaccine mandates. They lost experienced employees who were able to wrangle the bad scheduling software. Throughout 2022, Southwest was having hiring issues because they were still mandating the vaccine through at least the summer for new employees. Their pilots association warned about this causing disruptions after a bunch of summer cancellations. Do people forget how flaky Southwest was during summer 2022? Southwest just recently reached staffing levels that matched their 2019 high. This "inadequate technology" narrative just seems like a convenient scapegoat.
- MaxHoppersGhost 4y ago
  This is probably a contributing factor but will get buried by the media. However, I doubt it was the sole or even primary cause. There is definitely a staffing component here in addition to their bad software.
- kube-system 4y ago
  They also had mass cancellations in Oct 2021
  - lotsofpulp 4y ago
    They had the same computer related problem and mass cancellations in 2016.
- Supermancho 4y ago
  > This "inadequate technology" narrative just seems like a convenient scapegoat. My brother has worked in the white and blue collar unions (he prefers his ramp job). It's not like there's some impermeable cover of secrecy. These are just regular people who you can talk to. It's a combination of computer problems and regulatory controls (sleep blocks) leaving insufficient staff (and mechanical dangers) due to weather. The ramp teams were sitting at almost quad pay with no planes to service out of Minneapolis for a significant part of the weekend. This same situation has occurred, to some degree, every year. Due to the inevitable Guld Stream collapse, this will be a routine problem until SWA triages it.
icambron 4y ago
I've told this story a few times, but maybe 10 years ago I had a cross-country JetBlue flight that was delayed perhaps 6 hours hours. It was a few days after a major storm. Like Southwest here, JetBlue didn't have much flex capacity and relied on the daisy chain to keep on chaining. Our plane had gotten stuck somewhere, so they had to find a different one at some far-away airport and fly it in, which took hours. But the kicker was that when the plane finally landed, the crew already onboard couldn't man the flight because that would exceed their duty limits. The airline didn't realize this ahead of time, so they had to gather a new crew (like literally call them in), which added a couple of hours to the delay. Naively, I'd assumed these kinds of things were handled in some sort of mission-control center with warnings from rule engines blinking on some big screen and a team of crack operators mapping out what needed done. But clearly that wasn't so: they were just making things up as they went along. Sounds like Southwest is in a similar spot, but this time on a much bigger scale.
- bombcar 4y ago
  I’ve seen it happen and it’s always strange that they seem to not realize the crew will be unusable until they arrive … I assume they had been trying to get another crew at the same time.
  - nradov 4y ago
    Sometimes the dispatchers expect that the crew will still be legal, but then the incoming flight gets delayed just enough to push them over the limit.
  - ipqk 4y ago
    There are several time limits, but I believe that one of them stops when pulling back from the gate. i.e. once the the plane is moving, they're granted several X more hours, but if they haven't left yet, then they've timed out. So the arriving crew may plausibly be usable upon arrival, but a few minutes late may be all it takes to time them out. I've been on a delayed plane where one of the pilots timed out while sitting at the gate, so we had another second delay finding another pilot.
  - 4y ago
christkv 4y ago
I thought this stuff also gets more likely to happen as you get towards the end of the month as you are close to the max number of hours the pilots can fly per month (100h in the us per calendar month), making pilot shortage cascades even more likely to happen.
- londonReed 4y ago
  Pilots are only allowed to fly 100h a month? That seems really low, considering a 40 hour work week over 4 weeks in a month is 160h. Although I guess I would rather have my pilot more energized than overworked.
  - christkv 4y ago
    https://www.flyingmag.com/guides/how-many-hours-do-pilots-work/ https://www.flyingmag.com/guides/how-many-hours-do-pilots-wo... Even less if commercial airplane pilot it seems. 100h is for transport
    - krisoft 4y ago
      There is some confusion here. Transport pilots are commercial pilots.
  - atdrummond 4y ago
    There’s work before and after the flight that isn’t included in that 100 hours.
thepasswordis 4y ago
I'm surprised they haven't tried to blame a cyberattack yet. That said, I feel like these sorts of catastrophic ultra-fragile McKinsey-consulted-to-death failures we keep seeing in various industries are basically a giant signal to any adversaries that say "Hi! Check out how easy it would be to grind this entire industry to a halt!" Resiliency is literally the opposite of efficiency. These systems need to have slack, aka inefficiency built into them. Unfortunately the business culture has moved towards ultra fragile, ultra efficient thinking.
- ghaff 4y ago
  In part because, in this case, customers will buy the ticket that is $10 cheaper.
- factsarelolz 4y ago
  That could potentially increase their cyber security insurance premiums if not cause the insurer to drop them immediately. Not to mention the broader impact on the market and industry as a whole.
ProAm 4y ago
Still my go to airline because the rest are so difficult, unfriendly or just greedy Id rather deal with Southwest every time to feel like I am a human being.
- noirbot 4y ago
  I mentioned this in another thread, but Southwest feels like they fuck up in ways that are understandable, if unfortunate. Delta, United, American and co. feel like they intentionally screw you out of perverse pleasure and a desire to cause you pain. Southwest's seating may be chaotic, but it's not United where you have to pay an extra $60 to not get a middle seat, likely in the back of the plane.
- themadturk 4y ago
  That's one thing I have seen over and over regarding the current situation, that SouthWest personnel are polite, empathetic and as helpful as they can be given the circumstances. Do their planes still have smiles painted on them?
coliveira 4y ago
Nobody seems to figure out the core problem for airlines: they're selling tickets that they cannot fulfill under current conditions. If they know they don't have enough pilots, enough attendants, and so on, why in the world are they selling all the tickets they've "planned"? Is this a realistic plan or it is a SCAM, i.e., taking people's money for a service that they know they won't be able to fulfill in the future? These companies need to be investigated!
- ec109685 4y ago
  They obviously weren’t counting on a massive multi day storm when originally selling the tickets. Their inability to get themselves out of it was clearly a failure that article discusses.
- jscipione 4y ago
  When I was stuck in Florida during Hurricane Ian earlier this year my flight got cancelled 3 times. Having a ticket means nothing in terms of reserving a seat on a plane, it’s a scam for sure.
- noirbot 4y ago
  They're taking reservations months ahead of time. It's not as if they can magically know what staffing they'll have in 3-9 months. I'm effected by this, but I bought my ticket in September. It doesn't seem reasonable to require any business to know for-sure that every employee needed to run a flight in December will be there in order to sell it in September. Anyone can quit, die, or become ill at any time. If they had to contract every employee into a binding contract where they must fly a given route on a given day at a given time, we'd be howling about how they're abusing their employees, and in fact that's not far off from what everyone's been complaining about the railroad industry doing. Predicting the future, even a few days at a time, is always best-effort.
- dehrmann 4y ago
  Overbooking is a thing, but this isn't a consequence of overbooking.
ComputerGuru 4y ago
Southwest is statistically the worst airline in terms of delays and cancellations but has deluded its customers into thinking its the best (according to surveys asking people to rate airlines on their reliability). https://www.insidehook.com/daily_brief/travel/airlines-fewest-delays-cancellations https://www.insidehook.com/daily_brief/travel/airlines-fewes...
- alanbernstein 4y ago
  (in 2022)
- kube-system 4y ago
  I believe this has changed in recent years due to similar hiccups, and their reliability in prior years was previously good.
  - ghaff 4y ago
    I suspect also that, like JetBlue in its early years, a lot of its flights were out of secondary airports that are generally less exposed to a lot of operational disruptions. (e.g. they flew in Hobby in Houston early on). They also had the advantage of being in a region that per its name that probably has fewer weather issues in general. The pattern I've seen over the years is that upstart airlines, as they grow, end up having to look--for various reasons--a lot more like legacy carriers over time. Whether that's flying into areas with seasonal bad weather, flying out of default airports, instituting various forms of passenger status, etc.
- tmpburning 4y ago
  You are lucky if your flights are on time 75% of the time on average, with any airline.
  - quickthrower2 4y ago
    Usually better I find as they overestimate the flight time as if headwinds are happening very time
bmitc 4y ago
Doesn’t it feel that, since COVID, we keep hearing more and more about these colossal, systemic failures more frequently? It makes me concerned because it seems most assume that at least someone knows what’s going on. Whereas the reality is that almost no one, if anyone, actually knows what’s going on, and that we have fragile software systems “running” everything. And COVID seems to have been a real shot to the brow for many of these systems. I just worry about what hell we’re creating. Software basically captures miscommunication and poor understanding and executes it.
- Karrot_Kream 4y ago
  Airlines and airports have been suffering from some bad labor shortages because of the pandemic. A lot of them were let gone or furloughed during the pandemic and staffing hasn't recovered yet. That's why it's happening so often right now.
  - ghaff 4y ago
    I was on a flight recently where apparently one of the flight attendants just didn't show up. It seems that they were eventually able to grab someone from the airline who was flight attendant-certified or whatever (he was just in street clothes) so we were able to go. At least the gate agent kept us informed.
- kube-system 4y ago
  There's a lot of people who know exactly what's going on. We built fragile systems that optimize for efficiency, except edge cases. Because edge cases are expensive to handle and usually don't happen. Except when edge cases happen, and then we have problems.
  - rsj_hn 4y ago
    This is endemic to the whole world now, which has supply chains stretched across the globe in a just in time inventory system. Some economist should figure out what percentage of our productivity gains weren't really productivity gains at all, but rather risk acceptance. E.g. if we are currently producing 1000 units per year, then a new plan in which we produce 1010 units 95% of the time, 1000 units 4.9% of the time, and then 1% of the time we'll produce only 400. And just as people do price comparisons of unreliable electricity generation to reliable generation without taking the reliability into account, the various executives that signed off on this assumed (correctly) that the public would see a gain and give them the appropriate bonuses. Wall Street did a similar thing during the Great Financial Crisis. This type of analysis has already been done by Dean Baker for productivity gains due to trade deficits[1] - e.g. instead of building an iPhone, we just design the iPhone, and then call up China and ask "make us 100 million of these", and because the designers can capture so much value of the finished phone, it appears like an explosion of labor productivity. But it's all contingent on having good terms of trade and strong IP enforcement. If these were to decline, we'd capture much less value and suddenly the productivity of those designers would plummet, even though they are still doing the same work per unit of time, because productivity is measured in terms of value creation. So it could well be that on an apples to apples comparison, total productivity growth in the US from 2000 to 2020 has been flat or negative, and we've merely hidden the losses elsewhere, for example in unexpected outages, labor force problems, or periods of rapidly rising prices (revaluations). [1] https://www.cepr.net/documents/publications/productivity_2007_06.pdf https://www.cepr.net/documents/publications/productivity_200...
alkonaut 4y ago
So if I understand this correctly, almost the only thing that would have been needed would be a system of self-reporting from crews where they and their plane are, in case they are delayed or diverted? The "manual entry" system is already there, so not much seems to be needed other than adding this seemingly simple front end. You don't muck around with old business critical software, you add to it whenever possible. And this seems to be exactly like that. The existing manual entry system feels like it should be able to accept self-reports from a mobile app or web site with minimal changes to the existing system.
calbear81 4y ago
I’ve been lucky to have caught a flight back to SF after cancellations and can wait at home while figuring out how to get to my original destination. What I don’t understand is how come SW couldn’t enlist help to get customers rebooked on other airlines - their phone lines were slammed (I waited 3 hours) just to get a refund since their app wouldn’t allow me to choose to rebook/cancel. If I was as customer focused as they say they are - I would’ve contacted AMEX global travel and gotten their entire network of booking agents to backfill and rebook customers on other flights.
- realityking 4y ago
  Southwest doesn’t have interline agreements with other airlines nor, AFAIK, the integration into reservation systems that allow rebooking onto another airline.
twobitshifter 4y ago
https://blog.geaerospace.com/technology/big-wins-in-flight-efficiency-analytics-2/ https://blog.geaerospace.com/technology/big-wins-in-flight-e... Skysolver is a GE Flight Services trademark - there’s a video here showing how it works and SW planes. Contrary to the reddit claim, it does appear to use a predictive algorithm. Highlight quote from the video: “It is humanly impossible when there’s a major disruption for somebody to figure out what the optimal approach is to get them back on schedule”
- cerved 4y ago
  I would expect it to be some kind of OR solver
dehrmann 4y ago
Anyone remember the A&E show Airline from 20 years ago? This would have made quite the episode.
tonymet 4y ago
none of these accounts review the human factors. Experienced administrators , pilots and crew were terminated over the past year and the replacement staff is not as experienced . Not everything is an algorithm error
crosen99 4y ago
It's easy to ask, "How could this happen?", but it's also a wonder this sort of thing doesn't happen more often with airlines and other businesses that rely on solutions to complex logistical challenges at their core. Overall, despite the horrors of war, perils of a pandemic, etc., sometime I pause and ponder how remarkably well the world works.
igetspam 4y ago
A friend of mine wrote on this topic today as well. https://www.seat31b.com/2022/12/the-great-southwest-meltdown-of-2022/ https://www.seat31b.com/2022/12/the-great-southwest-meltdown...
nlstitch 4y ago
I would be very interested in a post mortem of the software used called SkySolver. Its supposed to be a Java Application which is said to be developed by Accenture? Anyone have actual technical insights into why it failed?
- Tao3300 4y ago
  I'll bet it's just another excuse. The situation was probably so bad that the only solutions SkySolver could offer were beyond their capacity. There were probably partial solutions that were possible, but they rolled the dice on being able to resolve it by overworking people. They decided to double down instead of paying a whole lot of vouchers and refunds they'd rather not have to. Is it the software's fault the bet didn't pay out? As the problem snowballs and becomes more intractable, you get too far off the script and reach a point of no return. I know nothing about the SkySolver or leadership at Southwest, but that seems like a very likely scenario based on what I've seen of what corporate types expect from software and rank-and-file employees.
  - ainvb 4y ago
    SkySolver is a total shit show no doubt, but it’s not the entirety of the problem. Keep in mind when you have disruptions you actually have multiple products. SkySolver only manages the crew. They have another system that manages the aircraft and flight schedule. Believe it or not these systems are mostly decoupled - the schedule is modified first, then the crew, and there may even be a feedback loop back to the schedule if there is no crew solution. Multiple things can be true here.
- DLarsen 4y ago
  About a decade ago, I was part of a startup trying to disrupt crew scheduling. It's a non-trivial operations problem when you aim to honor crew preferences, union-negotiated affordances, FAA legalities, etc. We were only involved in the pre-planned schedules. At that time, the airlines we were courting had entirely different human-hravybsystem to resolve real time issues. There was some level of reserve redundancy baked in so the human planners had some wiggle room to work with... but redundancy is expensive to maintain. As a relatively new engineer at the time it was a pretty neat domain with big $$$ at stake. As it turns out, pretty much nobody wanted to take the risk on a new system even if it had provably better schedules for all parties. All it takes is one snafu for the whole thing to turn into a major regret.
jrochkind1 4y ago
From what we know, this to me sounds like a story about technical debt. "Sure, it's held together with rubber bands and is a mess, but it would cost hundreds of millions to fix, and it's working, isn't it? So the programmers complain a bit, that's their job." Which works until conditions change in some way and it catastrophically does not. I think a lot of our society is now run on unreliable fragile software. I expect to see a lot more of this. "Automation" is especially cost-savings when you don't min it being a fragile unreliable time-bomb.
pledess 4y ago
apparently one other factor is that long on-hold phone calls (by Southwest staff) count against FAA work time limits, and thus can prevent that person from serving on a flight - according to the former CCO of JetBlue (St. George) at https://twitter.com/martysg/status/1608161473083183106 https://twitter.com/martysg/status/1608161473083183106
paulpauper 4y ago
For a meltdown the stock is back to where is was in October, tracking other airlines and the overall market, which keeps falling. I think people have become so accustomed this sort of stuff that it does not affect business long term. After Covid, people are accustomed to major inconvenience when traveling.
snambi 4y ago
Too much centralization. Centralization is bound to fail eventually. It happened.
zx8080 4y ago
Aren't cases like this is where the automated solvers are expected to shine? If, on the other hand, it's not at all about software failures as many comments here suggest ("company management lost track of crews" notion), then does it have something to do with software at all?
- cube00 4y ago
  Automated solvers that have a super computer backing them sure, but I doubt an airline has that kind of hardware at their disposal. Especially if their solver on their lower end hardware works most of the time for most disruptions.
  - bob1029 4y ago
    Wouldn't this be one reasonable use case for a super computer? Perhaps not the fastest one on earth, but we use this stuff to predict the weather so I don't see why we couldn't use the same for air travel.
  - ainvb 4y ago
    Southwest has the legacy SkySolver version which - believe it or not - runs locally on Windows OS. There is a cloud version that Southwest was either too cheap to buy or too resistant to migrate.
marze 4y ago
I find it especially ironic that SWA system failed them, and this large failure was preceded by worse and worse "near failures", since SWA is in the aviation business. In the aviation arena, high reliability is maintained in part by careful analysis of "near failures": lessons are extracted and improvements are made to aircraft designs, procedures, etc. By contrast, the "near failures" of the SWA system as a whole don't appear to have been utilized to motivate system improvements.
- masklinn 4y ago
  Per comments on /r/southwestairlines for 20 years the C-suite were Wall Street boys and set that as management culture, as long as shares were up things were fine. Frontlines have been emitting concerns and warnings for years but management didn’t care until an ops CEO (Bob Jordan) got in recently (as in 2022), but now there’s 20 years of ops neglect to deal with, and less than a year is nowhere near enough to start enacting real changes.
  - Sakos 4y ago
    On that note, I think it's interesting how glowingly people talk about Herb Kelleher. Wouldn't he be responsible for allowing that tech debt to build until it finally caused the system to fail catastrophically?
    - masklinn 4y ago
      Keller stepped down in 2004, so not really. In 2004, replacing 10 years old systems starts becoming a consideration but it's not an active need. Kelly was CEO from 2004 to 2022. And is now chairman of the board. Bob Jordan is the new CEO, since February 2022.
      - Sakos 4y ago
        Apparently Keller was on the board until 2008 and remained an FTE until 2013. I suspect he still had plenty of influence over Southwest's direction.
mise_en_place 4y ago
You only get bitten in the ass by tech debt after it’s too late. I’m sure management justified not paying it down because, truthfully, the consequences are never really felt until it’s too late. It’s better to pay down tech debt incrementally, instead of grand projects promising full rewrites.
dpacmittal 4y ago
This is probably the biggest failure caused by software going into a bad state.
dougb 4y ago
A Friend of a friend is a technical recruiter for SWA. They told me that SWA pays their Software Engineers below market salaries. They complained that because of that, its hard to recruit people, and they end up with mediocre developers.
- bdavis__ 4y ago
  someone has to fill the right side of the bell curve. except in Wobegon, Minnesota. of course.
eternalban 4y ago
tldr: SkySolver O(n^2) meets a somewhat large n.
GeoffKnauth 4y ago
I wonder if there's an accurate record (time series) of what got canceled when, why, in what order, along with an accurate starting state for the whole SWA system. Then, for Monday morning quarterbacking, modelers and scientists could step through the meltdown in slow motion, and see, at each step, what they think could have been done differently.
TexasDawg 4y ago
As someone with over 20 years experience developing decision support models for domestic and international airlines, here's my take. Southwest Airlines is a domestic carrier flying a point to point schedule, using the same fleet type to eliminate the need to train pilots on multiple aircraft type and also reduce training costs for transitional training from a narrow to a wide-body jet. All pilots can (theoretically) fly all planes in their fleet, tremendous training and labor savings. When flight crews "bid" on their flights, having only one aircraft type also reduces the complexities of bid lines down to basic seniority. (Let's ignore over water to Hawaii, m'kay?) So, getting SWA back in the air should be over simplified compared to every carrier who got their planes and crews back flying the same or next day - get it? SWA has a single fleet and all of their crew is qualified to operate those jets or crew the cabin - so "wat da problem is"??? US Gov, inquiring minds want to know!!! At all airlines, crews and planes are scheduled, and optimized, using decision support models which take into account how many hours the crew (pilots and flight attendants) can fly by law and contract; how long each tail# can fly until it needs to be flown into a maintenance base for an A/B/C/D check for scheduled maintenance, and how to get the maximum airtime out of the asset per day - in perfect weather conditions. There are also decision support systems that monitor pricing of competitors every second of the day and why your airfares change from one browser refresh to another, yield management models which run overnight taking input from industry load data from the past year in the market SWA flies to help predict the passenger "load" which sets fares and also permits manual inputs for special events such as a World Series, or Super Bowl etc which would spike demand and drive airfares higher. And there are Air Operations systems which are similar to the named product which take into account weather and crew events and help an airline re-plan based on where crew is currently. These Ops systems should have interfaces built to the crew (pilot and flight attendant) systems to know where they are located as well as where the jets are. Those values, along with the number of hours the crew has worked, would be used to re-calculate the crew and fleet assignments with the associated fleet and crew scheduling decision support models. These DSS ran on either big iron multiple CPU Unix servers or multiple CPU Linux servers - point being the computational power was outstanding and yes, CPLEX was typically a library utilized by our PhD's. There was a lot of money spent on the hardware and the people who developed these models - it wasn't cheap but then none of the clients ever experienced this type of problem with scheduling and yes, the clients are named in the weather impact article alongside SWA, but they are up and running either same or next day... My world had a common database and data model where all data was integrated from various systems regardless of if it was our system or not because a decision support system without current and accurate data is like having corn cobs for toilet paper vs. Charmin... painful and not a lot of value. This is the one time I'm looking forward to the gov looking under the hood of private industry and revealing where the technical and management issues are. You can't blame the technical teams as they're only paid what SWA pays to hire both FTEs and contractors, and why I've never had a phone call that lasted beyond finding out what the position paid. We all know, you get what you pay for, but again, my opinion, and I'm greedy!!!
ChoGGi 4y ago
"Mr. Jordan said he believed if Southwest could power through a few tough days, things would improve—its usual playbook." I feel for the employees
w10-1 4y ago
So much chatter! I would expect any interview candidate to spot the issue within a minute. For hub systems, ready crews are either at the hub, or at a spoke, ready to come back. That gives the hub a queue of ready crews, and each spoke can return a crew-plane combination to the hub when available. So with natural queue's, there's no delay cascade: it's all a function of whether and crew/plane readiness. For point-to-point systems, crew-plane's are scattered, and the next flight opportunity might not be the next flight need. There is no buffer anywhere. Furthermore, any greedy/opportunistic strategy at one point can block a superior global solution. That's the point-to-point trade-off taken by SWA. In the common case of good weather, you avoid the extra miles from going via hubs. But in the rare case of global weather shutdowns, there is no good recovery. The only real question is whether SWA had any obligation to communicate this to investors and passengers. So far, Apple stock has gone down more than SouthWest's in this period, and passengers are remaining loyal, so no damage done.

283 points by wallflower 4y ago

https://archive.ph/J3pFF https://archive.ph/J3pFF

So they are blaming SkySolver software. The article says it is off-the-shelf software? But in other news reports, they make it sound like it was developed in-house.

phpisthebest 4y ago
Alot of enterprise software is both, it is more akin to a "super framework". Where you start with the base system that provides commons functions but then it extendable. Most ERP platforms are like this. over the decades it is not uncommon for little of the original platform to be used. My org.s ERP is like that, we use probably 10% of the commercial code, and 90% of functions are custom in house written code.
- ghaff 4y ago
  Even a SaaS poster-child like Salesforce is like that--between partners and in-house development of various sorts. I think it may be the largest tech conference these days and it's not because everyone buys a standard Salesforce subscription.
- partdavid 4y ago
  Often, even more complicated: the ERP software vendor is one party; then there's the consulting VAR who does the customer's "customizations", and then the in-house team which does the "administration" and some "configuration". The VAR may or may not have a continuing relationship and of course the three groups have ample opportunity for finger-pointing; and the "administration" team is often under-resourced and doesn't really have the right skills to continue or maintain the work. It's worse than a framework because most frameworks are designed for people to do some form of software engineering with. But with some "configurable" tool, you end up with something that you need to do software engineering but it doesn't have the tools, power or affordances to do so. I've done some Jira consulting and worked in MDM and I'm of the firm belief that "buy vs. build" isn't the real dichotomy: it's "build" vs. "buy and build", and the effort of building on top of the platform is usually underestimated and undervalued.
  - phpisthebest 4y ago
    We have a team of in house devs that do all of our customization but yea out-sourcing is still all the rage with many other companies VP's looking for that next bonus... Companies are allergic to headcount
kube-system 4y ago
If it's anything like an ERP, then that sounds about right. There's the out of box part that does the most generic things, and then heavy customization to integrate and customize it to fit the particular business.
dragonwriter 4y ago
> The article says it is off-the-shelf software? But in other news reports, they make it sound like it was developed in-house. Working in enterprise, there is a lot of “modifiable off-the-shelf” software; ideally, this provides basic functionality out of the box, but also the alignment to custom business needs of in-house software. (Often, it seems like it combines the up-front and ongoing external licensing cost of COTS software, with the internal development and maintenance costs of in-house software, and the combined problems of both.)
chris_wot 4y ago
It probably was - massive customizations would have had to have been done to make it fit for purpose. Good luck replacing it, on an operational airline that can’t shutdown. How you would do this is to me, just mind-boggling.
- hellcats 4y ago
  AA does shut down for about 5-10 mins to run their solver and upload new assignments after a major event.
Beltalowda 4y ago
> So they are blaming SkySolver software. The article says it is off-the-shelf software? The article actually says "SkySolver, an off-the-shelf piece of software that Southwest has customized and updated".
billsac 4y ago
So what year was it introduced? Is GE still in the software bix? Who knew.

colechristensen 4y ago

There really should be laws or regulations that fine airlines for this kind of behavior and compensate passengers. Reliability failures that have that much impact on people need more consequences than just seeking another airline next time.

toomuchtodo 4y ago
There are, but more importantly, as critical infrastructure you should have to demonstrate a BCDR (business continuity disaster recovery) plan exists and that you can perform it from a cold start and under extreme system stress.
luma 4y ago
One would hope that the market could work this out, and when people went looking for options, would keep this in mind. Instead what people do is find the lowest fare possible, all other considerations be damned. It’s not like this is the first time this has happened to SW.
- munificent 4y ago
  > One would hope that the market could work this out Southwest did lose 10% of its market value overnight, so it seems the market is punishing them. Given how much executive compensation is in stock, a lot of senior level folks at Southwest are likely feeling some pain (or, least, as much pain as any very rich person ever feels). > Instead what people do is find the lowest fare possible, all other considerations be damned. That's an over-simplification. Yes, passengers are very price sensitive. But whenever I talk to friends and family about travel, it's become increasingly clear that they factor predicted quality of service in to their choice. I don't fly United and try to avoid American. I'm certain that many many people will hesitate to fly on Southwest after this. People aren't stupid and no one wants to roll the dice on an entire vacation just to save twenty bucks.
  - Spivak 4y ago
    And people will buy the dip, the news cycle will blow over and in a year the market won’t remember this at all.

lowbloodsugar 4y ago

They rely on telephones???

phpisthebest 4y ago
They relied on hand written notes for baggage until 2017... 2017... This is going to be a prime example CIO's can use as what happens when a company fails to invest in IT, treating as a cost center to alway gut budgets from Somewhere there are Sysadmins, and Devs at southwest just face palming at the years and years they have beggged for resources to improve things
- eightysixfour 4y ago
  It isn’t just tech problems that drive this kind of stuff, SWA couldn’t get a deal worked out with AMFA to use bag scanners until 2016. > Southwest said it will begin scanning bags on the tarmac this year to improve accuracy. Other large U.S. airlines already use this technology. Southwest's contract with bag handlers did not include a provision for using scanners until 2016. The airline then spent time evaluating technology and testing scanners at a few airports. https://www.keranews.org/news/2019-02-20/southwest-airlines-grapples-with-new-labor-and-revenue-problems https://www.keranews.org/news/2019-02-20/southwest-airlines-...
  - ComputerGuru 4y ago
    That's obviously not a blanket case of "AMFA bag handlers categorically refuse to use bag scanner tech" but rather "SW, unlike other airlines, didn't negotiate (early enough) the use of bag scanners in its contract with AMFA bag handlers." SW decides what their priorities are when negotiating these contracts and deciding the terms. Other airlines obviously prioritized bag scanner tech and made the necessary tradeoffs to make it happen while SW didn't. Half the blame at the very least should be apportioned to SW.
    - eightysixfour 4y ago
      I didn’t say it was categorically anyone’s fault, I just wanted to illustrate that airlines have unusual constraints on them so assigning blame to any one party is hard. The limitation are unusual compared to what we experience in less regulated, less unionized industries.

bottlepalm 4y ago

Anyone else work at a large company with a massively complex system that basically controls everything, and a skeleton crew of devs that know how it all works?

missedthecue 4y ago
Wait, you have people that know how it works?
- bottlepalm 4y ago
  Well not exactly, each dev knows their piece of the system. Impossible to understand it all.
- pphysch 4y ago
  Working on it. What gives me a kick is the longtime sysadmin in my department who grumbles when I discuss introducing new software because "who will maintain it?" Buddy, no one even understands the current software (except me a little bit)! If I end up leaving a Ball of Mud behind, it is in total compliance with the organization's engineering practices.
phpisthebest 4y ago
I did, then all those people retired now we just have massively complicated system that controls everything....
csours 4y ago
https://www.youtube.com/watch?v=kh9j1slA7R4 https://www.youtube.com/watch?v=kh9j1slA7R4 (Ralph Wiggum "I'm in danger") Yup.
luma 4y ago
Nope, we did but PE bought the outfit and replaced that crew with offshore contractors before selling it off to the next guy. Now nobody knows how it works.
- 4y ago

candiddevmike 4y ago

Their microservice rewrite will be ready any day now

theCrowing 4y ago
But, but since we started the rewrite 123123 better JavaScript frameworks were released... we should rewrite the rewrite.
- thepasswordis 4y ago
  Okay cool, we're ready to go! Does anybody know why it's complaining about node versions, though?
  - partdavid 4y ago
    I think that means we need to use Rust to target WebAssembly...?
Tao3300 4y ago
Sorry, the rest of the microservice rewrite has to be delayed so my pet feature can be added. Maybe if you hurry you'll be able to pick between more of the rewrite and resolving crippling tech debt from the last time.

nostromo 4y ago

The actual answer is buried at the end of a long article. > Unlike many rival airlines, Southwest’s planes generally hop from one city to another, rather than orbiting a major hub. That approach lets Southwest maximize use of its planes and crew, but the daisy chain structure also makes its network more delicate—problems in one corner of the country can be difficult to contain

masklinn 4y ago
I mean... it's not entirely untrue but at the same time airlines have been winding down their "hub and spoke" model for a point to point one for a while. That's in part what doomed the A380, which was popular with airlines still going strong with hub-and-spoke (Emirates being by far the most prominent one) but is worthless in a point-to-point model.
- ChrisMarshallNY 4y ago
  > the A380 My worst nightmare, was pulling into a gate, and seeing one of those puppies arriving on the same concourse. Immigration queues are bad enough, with 747s, but the A380 is much worse.
  - ikrenji 4y ago
    yeah well there is no reason why an entire A380 is being processed by two customs officers while 8 extra booths stand unused
- nradov 4y ago
  There are still a number of major airports that are slot constrained, especially now that passenger demand is growing again. It seems like there could be a profitable market for an efficient twin-engine airliner larger than the Boeing 777X. Perhaps even a double-decker?
  - masklinn 4y ago
    > There are still a number of major airports that are slot constrained Major airports are major airports even in a point to point system. If everybody wants to go into or come out of NY, NY's airports are going to be slot constrained. Also when I say "winding down" I don't mean "killed entirely", most airlines still have home bases, and mix point to point with regional hubs (especially for international flights). But at the height of the hub-and-spokes model, unless you departed from or went to a hub you'd always need a layover.

burlesona 4y ago

It’s fascinating that the same hopscotch travel pattern that allows SWA to offer better service to more places is also what caused the network to suffer cascading failure. Once a critical mass of pieces (planes/crew) were out of position the whole network fell apart, and it’s large enough that it seems like neither the humans nor software can easily reason about how to resume operations. Hence the need for a “full system reboot” over many days. Anecdotally, I flew Southwest just before Christmas. The network was already buckling and we had major delays, but we were lucky and made it through. Despite the stress, the SWA crews were helpful, empathetic, and polite. They handled it better than I would have if I had been in their shoes.

firstSpeaker 4y ago
I imagine the system cannot account for where all the planes, staff, passengers are and where they want to go economically.
- bumby 4y ago
  According the the article, the system makes some non-sensical tasking: "In one example during the storm, the system assigned a pilot to deadhead on a flight from Baltimore to Manchester, N.H., and then back to Baltimore the next day, without ever flying a plane"* * The article defines deadheading as sending a pilot as a passenger to get to another location. It would be interesting to look at what the system is trying to optimize for to make such choices.
  - brazzy 4y ago
    Sounds to me more like a bug in an edge case rather than trying to optimize for the wrong thing. Or (more likely, I think) the first deadhead was planned to have the pilot take over a flight which was then cancelled after the pilot was already underway.
    - smaudet 4y ago
      Also sounds like a case of not throwing enough adversarial data at the system - you can't just code coverage your code, you can't even establish KPIs, you have to establish its performance under system failure (does it freeze or gracefully shut down, does it persist to disk, what happens when the disk is yanked out of the system), etc. Very few software shops I am aware of that do anything like this.

LetsGetTechnicl 4y ago

It's good to see that the billions of dollars in federal bailouts they received during the pandemic was put to good use and not given to executives with salaries already in the millions of dollars and stakeholder dividends... oh wait...

cratermoon 4y ago
[flagged]
- throwaway9191aa 4y ago
  That article says “We recently had the opportunity to incorporate Employees’ requests to enhance campus. We converted a basketball court into three pickleball courts – a growing request by Employees over the past few years."
  - cratermoon 4y ago
    Reading a news article planted by the PR department of a big company is a skill. I can understand how this reads like "we're doing this for our people". This is just "the execs wanted to play pickleball so they had courts put in" dolled up to sound like big, happy, community-oriented charity.
    - Tao3300 4y ago
      "which would you like? (A) Pickle ball courts (B) Nothing"
      - phist_mcgee 4y ago
        What the hell is pickleball?
        moistly 4y ago
        Some sort of paddles-and-ball game that afaik is renown for its (a) endless, obnoxious, loud “THWOK! … THWOK! … THWOK!” sound and (b) obnoxious, self-entitled players. Seriously, I have never heard anything good about it from anyone who does not play, only controversy and upset from neighbours and other potential users of the court.

Spivak 4y ago

None of this actually explains what happened. Okay they have an off-the-shelf product called SkySolver that they use to manage their flights and it’s old (I guess the traveling salesman problem has really changed since 1930) and it couldn’t handle the sudden change in resource availability and flight constraints. But… why? What actually happened?

doodlesdev 4y ago
Some anecdotes here https://www.reddit.com/r/flying/comments/zw5lsl/comment/j1tne9z/ https://www.reddit.com/r/flying/comments/zw5lsl/comment/j1tn... Essentially the system never know if a flight was successful or not, it just assume it went well, in the case of a delay or a change some employee has to input data manually into the system. For a few flights that's ok, but for a exceptional event like this the problem snowballs and now the system doesn't know were planes, crew and passengers are. Quoting the top comment by 4Sammich here for those who refuse to touch Reddit: > I have friends in CS and the hotel assignment side too. There were 2 specific problems, the software for scheduling is woefully antiquated by at least 20 years. No app/internet options, all manual entry and it has settings that you DO NOT CHANGE for fear of crashing it. Those settings create the automated flow as a crewmember is moving about their day, it doesn’t know you flew the leg DAL-MCO it just assumes it and moves your piece forward. > In the event of a disruption you call scheduling and they manually adjust you. It does work, it just works for an airline 1/3 the size of SWA. > So the storm came and it impacted ground ops so bad that many many crews were now “unaccounted” for and the system in place couldn’t keep up. Then it happened for several more days. By Xmas evening the CS department had essentially reached the inability to do anything but simple, one off assignments. And to make matters worse, the phone system was updated not too long ago and it was not working well. > Last nite they did a web form and had planned to get the system up as much as possible with what communication they could muster, however it was too much to keep up on and ultimately the method for tracking crews failed again. > This 100% is at the feet of all management who refused to invest in technology updates because it is the southwest way to be stuck in 1993. Heck, they still do 35 min turns on a -700 and 45 on an -800 frequently with only 2 man gates. But the good news is HDQ has a pickle ball court now. > Edit: I just realized I never added the 2nd issue. Staffing. When the weather hit all those stations at once the ramp crews had to work in shifts to not become injured due to the cold. That slowed down the turns and backed up the planes. Many many ramp staff quit because of the management harassment (Denver) and just over it. So many rampers are new and making around 17/hr. Once they lost so much staff the crew scheduling software inputs couldn’t keep up because CS is also woefully understaffed and it became what we have today.

juujian 4y ago

Insane that the US has put all its eggs into one basket. One bad event and interregional passenger travel completely breaks down. Looks like it would be wise to invest in a more resilient mode of transportation.

Someone1234 4y ago
This is one airline out of several. Also, what other mode of transportation doesn't use scheduling software? Certainly not rail or inter-state buses, they have very similar headaches.
- bombcar 4y ago
  And indeed multiple competing companies with planes is going to be more resilient - as each flight is dependent on the airports at each end and not much more, whereas a road or rail method has to have all the interconnecting working.
kube-system 4y ago
We're talking about the 4th largest airline in the US.
- dehrmann 4y ago
  https://en.wikipedia.org/wiki/List_of_largest_airlines_in_North_America https://en.wikipedia.org/wiki/List_of_largest_airlines_in_No... Very coincidentally, their fleet size is 737.
leguminous 4y ago
What would be a more resilient mode of transportation? Few people would take a train from Chicago to Miami, for example. That's something like 1200 miles (2000km). Traveling from LA to NYC is like traveling from Lisbon to Moscow (2500mi/4000km by air). The US should definitely have more rail--and higher speed rail--but that is only practical in select areas with enough population density. Many of the people in those areas could drive to where rail would service.

crisdux 4y ago

I don't buy the narrative that inadequate technology is the main reason for the Southwest debacle. We must ask, why did this happen now and not before? Southwest has previously been able to better deal with disruptions like this. While the weather event did happen in the middle of their network, it wasn't unprecedented. I think a more obvious reasons is because of staffing issues brought on by covid, layoffs, and the vaccine mandates. They lost experienced employees who were able to wrangle the bad scheduling software. Throughout 2022, Southwest was having hiring issues because they were still mandating the vaccine through at least the summer for new employees. Their pilots association warned about this causing disruptions after a bunch of summer cancellations. Do people forget how flaky Southwest was during summer 2022? Southwest just recently reached staffing levels that matched their 2019 high. This "inadequate technology" narrative just seems like a convenient scapegoat.

MaxHoppersGhost 4y ago
This is probably a contributing factor but will get buried by the media. However, I doubt it was the sole or even primary cause. There is definitely a staffing component here in addition to their bad software.
kube-system 4y ago
They also had mass cancellations in Oct 2021
- lotsofpulp 4y ago
  They had the same computer related problem and mass cancellations in 2016.
Supermancho 4y ago
> This "inadequate technology" narrative just seems like a convenient scapegoat. My brother has worked in the white and blue collar unions (he prefers his ramp job). It's not like there's some impermeable cover of secrecy. These are just regular people who you can talk to. It's a combination of computer problems and regulatory controls (sleep blocks) leaving insufficient staff (and mechanical dangers) due to weather. The ramp teams were sitting at almost quad pay with no planes to service out of Minneapolis for a significant part of the weekend. This same situation has occurred, to some degree, every year. Due to the inevitable Guld Stream collapse, this will be a routine problem until SWA triages it.

icambron 4y ago

I've told this story a few times, but maybe 10 years ago I had a cross-country JetBlue flight that was delayed perhaps 6 hours hours. It was a few days after a major storm. Like Southwest here, JetBlue didn't have much flex capacity and relied on the daisy chain to keep on chaining. Our plane had gotten stuck somewhere, so they had to find a different one at some far-away airport and fly it in, which took hours. But the kicker was that when the plane finally landed, the crew already onboard couldn't man the flight because that would exceed their duty limits. The airline didn't realize this ahead of time, so they had to gather a new crew (like literally call them in), which added a couple of hours to the delay. Naively, I'd assumed these kinds of things were handled in some sort of mission-control center with warnings from rule engines blinking on some big screen and a team of crack operators mapping out what needed done. But clearly that wasn't so: they were just making things up as they went along. Sounds like Southwest is in a similar spot, but this time on a much bigger scale.

bombcar 4y ago
I’ve seen it happen and it’s always strange that they seem to not realize the crew will be unusable until they arrive … I assume they had been trying to get another crew at the same time.
- nradov 4y ago
  Sometimes the dispatchers expect that the crew will still be legal, but then the incoming flight gets delayed just enough to push them over the limit.
- ipqk 4y ago
  There are several time limits, but I believe that one of them stops when pulling back from the gate. i.e. once the the plane is moving, they're granted several X more hours, but if they haven't left yet, then they've timed out. So the arriving crew may plausibly be usable upon arrival, but a few minutes late may be all it takes to time them out. I've been on a delayed plane where one of the pilots timed out while sitting at the gate, so we had another second delay finding another pilot.
- 4y ago

christkv 4y ago

I thought this stuff also gets more likely to happen as you get towards the end of the month as you are close to the max number of hours the pilots can fly per month (100h in the us per calendar month), making pilot shortage cascades even more likely to happen.

londonReed 4y ago
Pilots are only allowed to fly 100h a month? That seems really low, considering a 40 hour work week over 4 weeks in a month is 160h. Although I guess I would rather have my pilot more energized than overworked.
- christkv 4y ago
  https://www.flyingmag.com/guides/how-many-hours-do-pilots-work/ https://www.flyingmag.com/guides/how-many-hours-do-pilots-wo... Even less if commercial airplane pilot it seems. 100h is for transport
  - krisoft 4y ago
    There is some confusion here. Transport pilots are commercial pilots.
- atdrummond 4y ago
  There’s work before and after the flight that isn’t included in that 100 hours.

thepasswordis 4y ago

I'm surprised they haven't tried to blame a cyberattack yet. That said, I feel like these sorts of catastrophic ultra-fragile McKinsey-consulted-to-death failures we keep seeing in various industries are basically a giant signal to any adversaries that say "Hi! Check out how easy it would be to grind this entire industry to a halt!" Resiliency is literally the opposite of efficiency. These systems need to have slack, aka inefficiency built into them. Unfortunately the business culture has moved towards ultra fragile, ultra efficient thinking.

ghaff 4y ago
In part because, in this case, customers will buy the ticket that is $10 cheaper.
factsarelolz 4y ago
That could potentially increase their cyber security insurance premiums if not cause the insurer to drop them immediately. Not to mention the broader impact on the market and industry as a whole.

ProAm 4y ago

Still my go to airline because the rest are so difficult, unfriendly or just greedy Id rather deal with Southwest every time to feel like I am a human being.

noirbot 4y ago
I mentioned this in another thread, but Southwest feels like they fuck up in ways that are understandable, if unfortunate. Delta, United, American and co. feel like they intentionally screw you out of perverse pleasure and a desire to cause you pain. Southwest's seating may be chaotic, but it's not United where you have to pay an extra $60 to not get a middle seat, likely in the back of the plane.
themadturk 4y ago
That's one thing I have seen over and over regarding the current situation, that SouthWest personnel are polite, empathetic and as helpful as they can be given the circumstances. Do their planes still have smiles painted on them?

coliveira 4y ago

Nobody seems to figure out the core problem for airlines: they're selling tickets that they cannot fulfill under current conditions. If they know they don't have enough pilots, enough attendants, and so on, why in the world are they selling all the tickets they've "planned"? Is this a realistic plan or it is a SCAM, i.e., taking people's money for a service that they know they won't be able to fulfill in the future? These companies need to be investigated!

ec109685 4y ago
They obviously weren’t counting on a massive multi day storm when originally selling the tickets. Their inability to get themselves out of it was clearly a failure that article discusses.
jscipione 4y ago
When I was stuck in Florida during Hurricane Ian earlier this year my flight got cancelled 3 times. Having a ticket means nothing in terms of reserving a seat on a plane, it’s a scam for sure.
noirbot 4y ago
They're taking reservations months ahead of time. It's not as if they can magically know what staffing they'll have in 3-9 months. I'm effected by this, but I bought my ticket in September. It doesn't seem reasonable to require any business to know for-sure that every employee needed to run a flight in December will be there in order to sell it in September. Anyone can quit, die, or become ill at any time. If they had to contract every employee into a binding contract where they must fly a given route on a given day at a given time, we'd be howling about how they're abusing their employees, and in fact that's not far off from what everyone's been complaining about the railroad industry doing. Predicting the future, even a few days at a time, is always best-effort.
dehrmann 4y ago
Overbooking is a thing, but this isn't a consequence of overbooking.

ComputerGuru 4y ago

Southwest is statistically the worst airline in terms of delays and cancellations but has deluded its customers into thinking its the best (according to surveys asking people to rate airlines on their reliability). https://www.insidehook.com/daily_brief/travel/airlines-fewest-delays-cancellations https://www.insidehook.com/daily_brief/travel/airlines-fewes...

alanbernstein 4y ago
(in 2022)
kube-system 4y ago
I believe this has changed in recent years due to similar hiccups, and their reliability in prior years was previously good.
- ghaff 4y ago
  I suspect also that, like JetBlue in its early years, a lot of its flights were out of secondary airports that are generally less exposed to a lot of operational disruptions. (e.g. they flew in Hobby in Houston early on). They also had the advantage of being in a region that per its name that probably has fewer weather issues in general. The pattern I've seen over the years is that upstart airlines, as they grow, end up having to look--for various reasons--a lot more like legacy carriers over time. Whether that's flying into areas with seasonal bad weather, flying out of default airports, instituting various forms of passenger status, etc.
tmpburning 4y ago
You are lucky if your flights are on time 75% of the time on average, with any airline.
- quickthrower2 4y ago
  Usually better I find as they overestimate the flight time as if headwinds are happening very time

bmitc 4y ago

Doesn’t it feel that, since COVID, we keep hearing more and more about these colossal, systemic failures more frequently? It makes me concerned because it seems most assume that at least someone knows what’s going on. Whereas the reality is that almost no one, if anyone, actually knows what’s going on, and that we have fragile software systems “running” everything. And COVID seems to have been a real shot to the brow for many of these systems. I just worry about what hell we’re creating. Software basically captures miscommunication and poor understanding and executes it.

Karrot_Kream 4y ago
Airlines and airports have been suffering from some bad labor shortages because of the pandemic. A lot of them were let gone or furloughed during the pandemic and staffing hasn't recovered yet. That's why it's happening so often right now.
- ghaff 4y ago
  I was on a flight recently where apparently one of the flight attendants just didn't show up. It seems that they were eventually able to grab someone from the airline who was flight attendant-certified or whatever (he was just in street clothes) so we were able to go. At least the gate agent kept us informed.
kube-system 4y ago
There's a lot of people who know exactly what's going on. We built fragile systems that optimize for efficiency, except edge cases. Because edge cases are expensive to handle and usually don't happen. Except when edge cases happen, and then we have problems.
- rsj_hn 4y ago
  This is endemic to the whole world now, which has supply chains stretched across the globe in a just in time inventory system. Some economist should figure out what percentage of our productivity gains weren't really productivity gains at all, but rather risk acceptance. E.g. if we are currently producing 1000 units per year, then a new plan in which we produce 1010 units 95% of the time, 1000 units 4.9% of the time, and then 1% of the time we'll produce only 400. And just as people do price comparisons of unreliable electricity generation to reliable generation without taking the reliability into account, the various executives that signed off on this assumed (correctly) that the public would see a gain and give them the appropriate bonuses. Wall Street did a similar thing during the Great Financial Crisis. This type of analysis has already been done by Dean Baker for productivity gains due to trade deficits[1] - e.g. instead of building an iPhone, we just design the iPhone, and then call up China and ask "make us 100 million of these", and because the designers can capture so much value of the finished phone, it appears like an explosion of labor productivity. But it's all contingent on having good terms of trade and strong IP enforcement. If these were to decline, we'd capture much less value and suddenly the productivity of those designers would plummet, even though they are still doing the same work per unit of time, because productivity is measured in terms of value creation. So it could well be that on an apples to apples comparison, total productivity growth in the US from 2000 to 2020 has been flat or negative, and we've merely hidden the losses elsewhere, for example in unexpected outages, labor force problems, or periods of rapidly rising prices (revaluations). [1] https://www.cepr.net/documents/publications/productivity_2007_06.pdf https://www.cepr.net/documents/publications/productivity_200...

alkonaut 4y ago

So if I understand this correctly, almost the only thing that would have been needed would be a system of self-reporting from crews where they and their plane are, in case they are delayed or diverted? The "manual entry" system is already there, so not much seems to be needed other than adding this seemingly simple front end. You don't muck around with old business critical software, you add to it whenever possible. And this seems to be exactly like that. The existing manual entry system feels like it should be able to accept self-reports from a mobile app or web site with minimal changes to the existing system.

calbear81 4y ago

I’ve been lucky to have caught a flight back to SF after cancellations and can wait at home while figuring out how to get to my original destination. What I don’t understand is how come SW couldn’t enlist help to get customers rebooked on other airlines - their phone lines were slammed (I waited 3 hours) just to get a refund since their app wouldn’t allow me to choose to rebook/cancel. If I was as customer focused as they say they are - I would’ve contacted AMEX global travel and gotten their entire network of booking agents to backfill and rebook customers on other flights.

realityking 4y ago
Southwest doesn’t have interline agreements with other airlines nor, AFAIK, the integration into reservation systems that allow rebooking onto another airline.

twobitshifter 4y ago

https://blog.geaerospace.com/technology/big-wins-in-flight-efficiency-analytics-2/ https://blog.geaerospace.com/technology/big-wins-in-flight-e... Skysolver is a GE Flight Services trademark - there’s a video here showing how it works and SW planes. Contrary to the reddit claim, it does appear to use a predictive algorithm. Highlight quote from the video: “It is humanly impossible when there’s a major disruption for somebody to figure out what the optimal approach is to get them back on schedule”

cerved 4y ago
I would expect it to be some kind of OR solver

dehrmann 4y ago

Anyone remember the A&E show Airline from 20 years ago? This would have made quite the episode.

tonymet 4y ago

none of these accounts review the human factors. Experienced administrators , pilots and crew were terminated over the past year and the replacement staff is not as experienced . Not everything is an algorithm error

crosen99 4y ago

It's easy to ask, "How could this happen?", but it's also a wonder this sort of thing doesn't happen more often with airlines and other businesses that rely on solutions to complex logistical challenges at their core. Overall, despite the horrors of war, perils of a pandemic, etc., sometime I pause and ponder how remarkably well the world works.

igetspam 4y ago

A friend of mine wrote on this topic today as well. https://www.seat31b.com/2022/12/the-great-southwest-meltdown-of-2022/ https://www.seat31b.com/2022/12/the-great-southwest-meltdown...

nlstitch 4y ago

I would be very interested in a post mortem of the software used called SkySolver. Its supposed to be a Java Application which is said to be developed by Accenture? Anyone have actual technical insights into why it failed?

Tao3300 4y ago
I'll bet it's just another excuse. The situation was probably so bad that the only solutions SkySolver could offer were beyond their capacity. There were probably partial solutions that were possible, but they rolled the dice on being able to resolve it by overworking people. They decided to double down instead of paying a whole lot of vouchers and refunds they'd rather not have to. Is it the software's fault the bet didn't pay out? As the problem snowballs and becomes more intractable, you get too far off the script and reach a point of no return. I know nothing about the SkySolver or leadership at Southwest, but that seems like a very likely scenario based on what I've seen of what corporate types expect from software and rank-and-file employees.
- ainvb 4y ago
  SkySolver is a total shit show no doubt, but it’s not the entirety of the problem. Keep in mind when you have disruptions you actually have multiple products. SkySolver only manages the crew. They have another system that manages the aircraft and flight schedule. Believe it or not these systems are mostly decoupled - the schedule is modified first, then the crew, and there may even be a feedback loop back to the schedule if there is no crew solution. Multiple things can be true here.
DLarsen 4y ago
About a decade ago, I was part of a startup trying to disrupt crew scheduling. It's a non-trivial operations problem when you aim to honor crew preferences, union-negotiated affordances, FAA legalities, etc. We were only involved in the pre-planned schedules. At that time, the airlines we were courting had entirely different human-hravybsystem to resolve real time issues. There was some level of reserve redundancy baked in so the human planners had some wiggle room to work with... but redundancy is expensive to maintain. As a relatively new engineer at the time it was a pretty neat domain with big $$$ at stake. As it turns out, pretty much nobody wanted to take the risk on a new system even if it had provably better schedules for all parties. All it takes is one snafu for the whole thing to turn into a major regret.

jrochkind1 4y ago

From what we know, this to me sounds like a story about technical debt. "Sure, it's held together with rubber bands and is a mess, but it would cost hundreds of millions to fix, and it's working, isn't it? So the programmers complain a bit, that's their job." Which works until conditions change in some way and it catastrophically does not. I think a lot of our society is now run on unreliable fragile software. I expect to see a lot more of this. "Automation" is especially cost-savings when you don't min it being a fragile unreliable time-bomb.

pledess 4y ago

apparently one other factor is that long on-hold phone calls (by Southwest staff) count against FAA work time limits, and thus can prevent that person from serving on a flight - according to the former CCO of JetBlue (St. George) at https://twitter.com/martysg/status/1608161473083183106 https://twitter.com/martysg/status/1608161473083183106

paulpauper 4y ago

For a meltdown the stock is back to where is was in October, tracking other airlines and the overall market, which keeps falling. I think people have become so accustomed this sort of stuff that it does not affect business long term. After Covid, people are accustomed to major inconvenience when traveling.

snambi 4y ago

Too much centralization. Centralization is bound to fail eventually. It happened.

zx8080 4y ago

Aren't cases like this is where the automated solvers are expected to shine? If, on the other hand, it's not at all about software failures as many comments here suggest ("company management lost track of crews" notion), then does it have something to do with software at all?

cube00 4y ago
Automated solvers that have a super computer backing them sure, but I doubt an airline has that kind of hardware at their disposal. Especially if their solver on their lower end hardware works most of the time for most disruptions.
- bob1029 4y ago
  Wouldn't this be one reasonable use case for a super computer? Perhaps not the fastest one on earth, but we use this stuff to predict the weather so I don't see why we couldn't use the same for air travel.
- ainvb 4y ago
  Southwest has the legacy SkySolver version which - believe it or not - runs locally on Windows OS. There is a cloud version that Southwest was either too cheap to buy or too resistant to migrate.

marze 4y ago

I find it especially ironic that SWA system failed them, and this large failure was preceded by worse and worse "near failures", since SWA is in the aviation business. In the aviation arena, high reliability is maintained in part by careful analysis of "near failures": lessons are extracted and improvements are made to aircraft designs, procedures, etc. By contrast, the "near failures" of the SWA system as a whole don't appear to have been utilized to motivate system improvements.

masklinn 4y ago
Per comments on /r/southwestairlines for 20 years the C-suite were Wall Street boys and set that as management culture, as long as shares were up things were fine. Frontlines have been emitting concerns and warnings for years but management didn’t care until an ops CEO (Bob Jordan) got in recently (as in 2022), but now there’s 20 years of ops neglect to deal with, and less than a year is nowhere near enough to start enacting real changes.
- Sakos 4y ago
  On that note, I think it's interesting how glowingly people talk about Herb Kelleher. Wouldn't he be responsible for allowing that tech debt to build until it finally caused the system to fail catastrophically?
  - masklinn 4y ago
    Keller stepped down in 2004, so not really. In 2004, replacing 10 years old systems starts becoming a consideration but it's not an active need. Kelly was CEO from 2004 to 2022. And is now chairman of the board. Bob Jordan is the new CEO, since February 2022.
    - Sakos 4y ago
      Apparently Keller was on the board until 2008 and remained an FTE until 2013. I suspect he still had plenty of influence over Southwest's direction.

mise_en_place 4y ago

You only get bitten in the ass by tech debt after it’s too late. I’m sure management justified not paying it down because, truthfully, the consequences are never really felt until it’s too late. It’s better to pay down tech debt incrementally, instead of grand projects promising full rewrites.

dpacmittal 4y ago

This is probably the biggest failure caused by software going into a bad state.

dougb 4y ago

A Friend of a friend is a technical recruiter for SWA. They told me that SWA pays their Software Engineers below market salaries. They complained that because of that, its hard to recruit people, and they end up with mediocre developers.

bdavis__ 4y ago
someone has to fill the right side of the bell curve. except in Wobegon, Minnesota. of course.

eternalban 4y ago

tldr: SkySolver O(n^2) meets a somewhat large n.

GeoffKnauth 4y ago

I wonder if there's an accurate record (time series) of what got canceled when, why, in what order, along with an accurate starting state for the whole SWA system. Then, for Monday morning quarterbacking, modelers and scientists could step through the meltdown in slow motion, and see, at each step, what they think could have been done differently.

TexasDawg 4y ago

As someone with over 20 years experience developing decision support models for domestic and international airlines, here's my take. Southwest Airlines is a domestic carrier flying a point to point schedule, using the same fleet type to eliminate the need to train pilots on multiple aircraft type and also reduce training costs for transitional training from a narrow to a wide-body jet. All pilots can (theoretically) fly all planes in their fleet, tremendous training and labor savings. When flight crews "bid" on their flights, having only one aircraft type also reduces the complexities of bid lines down to basic seniority. (Let's ignore over water to Hawaii, m'kay?) So, getting SWA back in the air should be over simplified compared to every carrier who got their planes and crews back flying the same or next day - get it? SWA has a single fleet and all of their crew is qualified to operate those jets or crew the cabin - so "wat da problem is"??? US Gov, inquiring minds want to know!!! At all airlines, crews and planes are scheduled, and optimized, using decision support models which take into account how many hours the crew (pilots and flight attendants) can fly by law and contract; how long each tail# can fly until it needs to be flown into a maintenance base for an A/B/C/D check for scheduled maintenance, and how to get the maximum airtime out of the asset per day - in perfect weather conditions. There are also decision support systems that monitor pricing of competitors every second of the day and why your airfares change from one browser refresh to another, yield management models which run overnight taking input from industry load data from the past year in the market SWA flies to help predict the passenger "load" which sets fares and also permits manual inputs for special events such as a World Series, or Super Bowl etc which would spike demand and drive airfares higher. And there are Air Operations systems which are similar to the named product which take into account weather and crew events and help an airline re-plan based on where crew is currently. These Ops systems should have interfaces built to the crew (pilot and flight attendant) systems to know where they are located as well as where the jets are. Those values, along with the number of hours the crew has worked, would be used to re-calculate the crew and fleet assignments with the associated fleet and crew scheduling decision support models. These DSS ran on either big iron multiple CPU Unix servers or multiple CPU Linux servers - point being the computational power was outstanding and yes, CPLEX was typically a library utilized by our PhD's. There was a lot of money spent on the hardware and the people who developed these models - it wasn't cheap but then none of the clients ever experienced this type of problem with scheduling and yes, the clients are named in the weather impact article alongside SWA, but they are up and running either same or next day... My world had a common database and data model where all data was integrated from various systems regardless of if it was our system or not because a decision support system without current and accurate data is like having corn cobs for toilet paper vs. Charmin... painful and not a lot of value. This is the one time I'm looking forward to the gov looking under the hood of private industry and revealing where the technical and management issues are. You can't blame the technical teams as they're only paid what SWA pays to hire both FTEs and contractors, and why I've never had a phone call that lasted beyond finding out what the position paid. We all know, you get what you pay for, but again, my opinion, and I'm greedy!!!

ChoGGi 4y ago

"Mr. Jordan said he believed if Southwest could power through a few tough days, things would improve—its usual playbook." I feel for the employees

w10-1 4y ago

So much chatter! I would expect any interview candidate to spot the issue within a minute. For hub systems, ready crews are either at the hub, or at a spoke, ready to come back. That gives the hub a queue of ready crews, and each spoke can return a crew-plane combination to the hub when available. So with natural queue's, there's no delay cascade: it's all a function of whether and crew/plane readiness. For point-to-point systems, crew-plane's are scattered, and the next flight opportunity might not be the next flight need. There is no buffer anywhere. Furthermore, any greedy/opportunistic strategy at one point can block a superior global solution. That's the point-to-point trade-off taken by SWA. In the common case of good weather, you avoid the extra miles from going via hubs. But in the rare case of global weather shutdowns, there is no good recovery. The only real question is whether SWA had any obligation to communicate this to investors and passengers. So far, Apple stock has gone down more than SouthWest's in this period, and passengers are remaining loyal, so no damage done.