9 ms·
Ask HN: What's Happening at Southwest?
The media narrative is that there's a massive software failure going on in its scheduling system, and they need to stop flying for a day or two to "reset" it. Anyone have first-hand experience on their systems about what that really means?
- mtmail 4y agoVia reddit/r/bestof https://old.reddit.com/r/flying/comments/zw5lsl/southwest_pilots_hows_it_going/j1tne9z/ https://old.reddit.com/r/flying/comments/zw5lsl/southwest_pi... "So the storm came and it impacted ground ops so bad that many many crews were now “unaccounted” for and the system in place couldn’t keep up. Then it happened for several more days. By Xmas evening the CS department had essentially reached the inability to do anything but simple, one off assignments. And to make matters worse, the phone system was updated not too long ago and it was not working well." "I used to work for a large company trying to fill the void [huge gap in the market for good aviation scheduling software], and our software was damn good too. SW was one of the airlines interested, we would demo it exactly like the scenario today, but it was "too expensive" and they stayed on their homebuilt stuff."
- hrunt 4y ago> "I used to work for a large company trying to fill the void [huge gap in the market for good aviation scheduling software], and our software was damn good too. SW was one of the airlines interested, we would demo it exactly like the scenario today, but it was "too expensive" and they stayed on their homebuilt stuff." This is a common situation in any industry. A company has a solution it self-developed in its infancy to support itself, and as it grows, it comes to rely very heavily on this software. At a certain point, everyone involved with the software -- from the developers to the users to the customers -- knows that it is not on par with third-party software developed by a team dedicated to all of its potential use-cases and pitfalls, but it is very difficult for the business to replace it because the ongoing cost is a relatively low maintenance cost, and the replacement cost is a one-time, relatively high purchase fee (with, usually, a similar if not higher ongoing maintenance cost). So the immediate reaction is to stay with the low ongoing cost, even though everyone "knows" that the long-term benefits of the third-party solution far outweigh the financial savings of keeping the in-house solution, whether those benefits might be avoiding a national week-long service outage or something simpler, like the ability for staff to get more done in other areas of the company that need attention when getting larger. I have been there. It can be very difficult to make the financial case when the future benefits are somewhat speculative or intangible. Accountants tend to devalue those benefits, relative to the hard numbers they already have. It does not help that system replacement projects often go over budget and introduce other problems that cost more money. Replacing home-grown systems is hard because they are not just software replacements; they typically also require reworking business processes.
- mark-r 4y agoThe existing solution needn't even be home built. I worked for a company that replaced one ERP system with another. It was a medical manufacturer, so in many cases the new system had to warp to the established government mandated business practices. Over time and over budget, you bet - they weren't finished with the transition when I left.
- rektide 4y ago> and the replacement cost is a one-time, relatively high purchase fee (with, usually, a similar if not higher ongoing maintenance cost) i'd wager complex scheudling software is probably licensed, not sold outright. many many millions a year to license. but yeah, lot of good factors you cite.
- grantc 4y agoIt can be cost-driven, but it's just as likely to be complexity-gated (w/ cost frequently being implementation driven and a proxy for complexity). The cost comes from actually unwinding a gnarly legacy system (that may be less than well understood) and making the successor system work. Also, Gall's law has entered the chat: A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system.
- kjellsbells 4y agoYou just described the last 30 years of the US phone network.
- unwise-exe 4y ago[dead]
- cpeterso 4y agoFor folks interested in legacy software systems (and managing the teams managing these systems), I highly recommend Marianne Bellotti's "Kill It with Fire: Manage Aging Computer Systems (and Future Proof Modern Ones)". The author worked at the United States Digital Service. https://nostarch.com/kill-it-fire https://nostarch.com/kill-it-fire
- shagie 4y agoA key point is how they do routing - https://www.ajot.com/news/southwest-air-faces-gridlock-with-over-80-of-flights-scrapped-or-late https://www.ajot.com/news/southwest-air-faces-gridlock-with-... > Unlike competitors that use a so-called hub-and-spoke system to funnel passengers to large airports, Southwest is focused on point-to-point service, flying the same aircraft — Boeing Co. 737s — on trips that may hopscotch around the US. With a hub and spoke system, all the planes go from A to HUB and then from the HUB to somewhere. If the route A-HUB gets saturated, they can put more planes on that, and those planes can always be found at the HUB. This applies to crews too. You'll have something that looks like this: https://www.airlineroutemaps.com/maps/Delta_Air_Lines/North_America/Detroit https://www.airlineroutemaps.com/maps/Delta_Air_Lines/North_... This comes at the cost of having oversupply at some spots and its harder to offer the "ideal" routes as everyone needs to transfer to another plane with a layover somewhere... and your baggage is more likely to get lost. There's a bit not to like as a passenger on such an airline unless it's a nice one leg route - but then who wants to fly to Detroit? Southwest is different - they go from anywhere they want to anywhere they want with non-stop flights and picking the most lucrative routes they can. This lowers the effective cost per flight and are likely non-stop flights. Everything that a customer wants. Southwest routes from 2001 https://www.flickr.com/photos/erussell1984/15863298679 https://www.flickr.com/photos/erussell1984/15863298679 - you can see the lack of hubs there. This is done through some crafty constraint programming to try to make sure that all the capacity is where it needs to be when it needs to be there. However, when the capacity hits "holidays - everything is at max", along with "big storm prevents flights from going to where they need to be for the next leg" this system breaks down and planes and crews are out of position or need to sleep. Their software was able to handle this constraint system when it was a smaller company with fewer routes - there were fewer constraints. The "reset" is not "shut down the computers and start them back up" but rather "let all the crews get their required sleep and then go to the spot where they need to be in order to handle the load - not where they currently are (out of position)". Hypothetically, this is solvable if you have enough compute... but that's a lot of compute that needs to be recomputed each time something changes (weather, crew gets sick, passenger load changes) and that ends up being impractical and expensive. --- Related reading: NYT - What Caused the Chaos at Southwest : https://www.nytimes.com/2022/12/28/travel/southwest-airlines-flight-cancellations.html?unlocked_article_code=6VPiQilkeMys9acoeGC6vfMSS6_v012ZLqICDXYlXpz-XrxSRfoduoUKEGlC9I_Zd5iXUO4XuvVHOkMIkGQRb9_mtIoSx12rjtan4daGFRdI_ClhRDDy9Zuokp0yKBZ0vfl5eW9CQ02yQyqPkzWZzDk9SE5kIeNacg8UCitoZP_cL8Xfz8OM91TQ3udQiVpjObO2fPT9e2P1g7QOo8Nzz6krt4kjfmZ87lko5q7Vqx5dNPe8eNhJ82dV3qPPSU-xdCRb26f_XY05XMJH5SzQdMdEtnRkmYM__Btnkj2rYTePGsh9fD4o-aZZ-GwmEpZqldK-Eub8XqoHhnXw3mtjzyIZhdij_Q5mS3ZDGszHp-o&smid=share-url https://www.nytimes.com/2022/12/28/travel/southwest-airlines... WSJ - How Southwest Airlines Melted Down : https://www.wsj.com/articles/southwest-airlines-melting-down-flights-cancelled-11672257523?st=y31zwvcsw79hmd9&reflink=desktopwebshare_permalink https://www.wsj.com/articles/southwest-airlines-melting-down... --- https://news.ycombinator.com/item?id=34165791 https://news.ycombinator.com/item?id=34165791
- mikewarot 4y agoHow could a CEO let this persist for more than an hour? Is there nobody with the leaderships skills to just give everyone free flights and promise everything will be sorted while having people record who worked and who flew and what services were used as best they can? The software failed, just Jerry-rig something up to get people where they need to be, let the accountants and programmers clear up the mess after the fact. [Edit]Software helps optimize things, but as long as the crews and suppliers have sufficient faith that they will be treated fairly, you could run this all with paper and pen for a few days while the software gets straightened out. Decide which hubs are most important, and have someone work out what goes where by hand. Once those routes are up and running, you can work outward to the less trafficked airports. Just work on getting the most people to the places that help the most. Crews know how many hours they've worked, and can track that themselves for the moment, or perhaps the Captain could do that. Everyone has cell phones, and could route around this damage. The consensus here on HN seems to be to give in without trying, which I find disturbing.
- deleted 4y ago[deleted]
- _yb2s 4y agoWould that really work? How do you make sure too many people don’t show up to get on the same plane? How do you make sure every flight has a crew? How do you decide which flights to still run when only half the incoming planes actually showed up?
- burkaman 4y agoYou can't give people free flights if you don't know where your planes and crews are and don't know where to tell your angry customers to go. There are also laws to adhere to, there is a maximum amount of time a flight crew can work so if you tell them to "just fly if you can and we'll figure it out later" you're probably going to break a lot of laws and endanger a lot of passengers. I think they probably are trying to jerry-rig a system, but the airline industry is heavily regulated for safety reasons (a good idea that has been extremely successful), so it's very difficult to get a plane in the air if you don't know what the fuck you're doing.
- 4y ago
- mattbrewsbytes 4y agoBasically what I read was the former CEO was accountant educated and focused on stock price and financials and ignored system upgrades for like a decade. That coupled with their decentralized nature of operating (no hubs or star pattern) the recent storms caused their systems to not “know” where people or planes were even though the people knew where they were. That knowledge is the critical feature because hours of fly-time for pilots and airplanes is crucial for legally mandated rest periods, maintenance and logistics. Having personally experienced the delays and issues with SW earlier this year I can suggest not flying them other than direct flights. Them not having any hubs means if one plane is late or a crew times out because of delays you are stranded. We got stranded at one airport overnight and got routed thru 2 other airports, changing planes at one, before reaching our destination. They have zero capacity of crews/planes on standby at any airport, probably because of the former CEO leadership and pandemic complications.
- ww520 4y agoAh so bean counting treated IT as a cost center and cut CapEx to the bone. Running things with no reserved capacity, crew wise and system wise, is just asking for a catastrophic fall off the cliff style shutdown. Edit: a simple queue theory can explain why no reserved capacity leads to system breakdown.
- mattbrewsbytes 4y agoYup. For me they had brand prominence because of various customer friendly things like bags fly free, sit where you want, etc. It will take a mountain of good reports from other people over a significant period of time for me to want to fly with them.
- jen20 4y agoThis is the first time I’ve ever seen the plane-boarding hunger games of sit-where-you-like referred to as a customer-friendly thing.
- 4y ago
- sjm-lbm 4y agoI don't think the reset is around resetting the software logic exactly, but rather that things have gotten so far outside of what they should be they no longer know where planes are, where crews that can find those planes are, and how many people even are still planning on using Southwest to fly to their intended destination. They need a "reset" to spend a few days inputting as much of that data as possible into their computer system, at which point they can start operating with some level of efficiency and start functioning again.
- phil21 4y agoFrom what I know... Not affiliated at all with the airline industry, but I fly a lot and tend to nerd out on this stuff... Their crew (and flight) scheduling software functions in a manner where it more or less simulates a "perfect" day of operations. Airplanes take off on time, land, and continue on. If anything disrupts this simulation, crew members had to call in and talk to someone to update the computer system to tell it that both the airplane and crew members are not where the system thought they were. Once the call center got overwhelmed it was a cascading failure with Southwest quickly not understanding where most of their flight crew happened to be at any given moment. It appears they feel the only way to solve this is get everything (planes and crews) back into "starting position" to restart the simulation.
- Simon_O_Rourke 4y agoIn this age of more or less fully network connected devices, this kind of a setup seems archaic in the extreme, probably likely to be some hangover from the 70s or 80s.
- pavlov 4y agoAirline systems are some of the oldest computing architectures in use. The Sabre reservation system was created by IBM for American Airlines in the 1950s — when a computer filled an entire building floor and was built out of vacuum tubes — and remains actively used today by multiple airlines. The programming language has been switched several times over the past 60+ years, but essential compatibility remains.
- ethbr0 4y agoIndeed. Probably the only equally complex and old dynamic computing systems out there are railroad and federal economic ones.
- persona_reuse 4y agoI can't remember where I read this and I can't find a cite right away, but I bet someone can confirm-or-correct me: the 6-letter reservation code for your flight? That used to be an explicit pointer of memory in oldold Sabre.
- gitfan86 4y agoFrom what I've read here and other places there was a unique situation of: 1. Large holiday volume of flights AND a lot of cancelations due to weather happened at the same time. 2. Their software system was unable to recommend optimal next steps for flight crews because it had not been designed to take this specific situation into account. Other airlines also had problem #1. But since they mostly operate on a hub model it is easy to tell their flight crews where they should go once the weather gets better.
- _plg_ 4y agoLorin Hochstein's take on it: https://surfingcomplexity.blog/2022/12/28/southwest-airlines-a-case-study-in-brittleness/ https://surfingcomplexity.blog/2022/12/28/southwest-airlines...
- bena 4y agoFrom what I've gathered, Southwest effectively performs Just-In-Time resource management. They arrange flights and crews so that the right number of planes and people are in the right places at the right times. There's some tolerance in the system. So if the plane from New York to Cincinnati is late, it's ok-ish. The flight from Cincinnati to Dallas should be able to make it in time if things aren't too bad. Then the flight from Dallas to Phoenix should take off when it should. The Phoenix to Las Vegas flight will never know there was a problem. It also matters for crew. Pilots can only fly for so many hours. So if you have someone stuck in a holding pattern, that cuts into times. However, if that plane from New York to Cincinnati shits the bed, it'll fuck over Cincinnati to Dallas, Dallas to Phoenix, and Phoenix to Las Vegas. The failure just cascades. You lose planes, you lose crews, nothing is matching up and everything is fucked. Now imagine this happens a few hundred times. Thousands of flights are affected. Other airlines don't have this problem because they can just not do a flight. They fly people into a hub, then out of a hub. Delta will go from New York to Atlanta, and back again. Cincinnati to Atlanta, and back again. They work more like a busses. Miss a bus, catch the next one. So if you crap out a day's of flights, you can still put those people on planes and get them out. You know they're either at the hub or on their spoke. So if they're not in Atlanta, they're in their city.
- MontyCarloHall 4y agoA related question: can Southwest ever fully recover from this? Their reputation had taken such a hit that many loyal customers will hesitate to fly with them again, and even more fairweather customers will permanently cross Southwest off their list of carriers to consider flying with. Their stock price is only down ~10% since the meltdown, which seems extremely optimistic to me.
- mitchell_h 4y agoYes. I'm a very frequent flyer, and used to be far more frequent. No one, and I mean no one, flys southwest because it's their choice. "Southwest is the greyhound of the sky" is a well known phrase. You fly southwest because it's slightly better than being cramped on 3 legs of a 50-60 seat commuter jet. Your home airport dictates which airline you take. Unless another airline steps up and takes some of the routes & provides better planes, you're stuck with SWA.
- kgermino 4y agoAre you on one of the coasts? In the Midwest I know a ton of business travelers (>1 trip/month) who default to Southwest. The weekly travelers tend to prefer the legacy carriers if they're available (so I'm sure there's something to your point even here) but IME Southwest rules the monthly/short notice flyers around me.
- mitchell_h 4y agoHome airport is STL, I normally fly to the coasts.
- mastercheif 4y agoI can anecdotally back this statement up via experience with frequent flyers based in Nashville.
- jaxn 4y agoWeekly travelers prefer legacy carriers because they get upgraded to fist class, have special phone numbers for customer service and rebooking, board first, etc.
- n1b0m 4y agoUnlike other airlines that operate directly to and from hub airports, Southwest tends to operate a point-to-point service, meaning that when one flight is disrupted, there may not be spare aircraft and crews to pick up the route, leading to disruptions through the scheduling chain
- punkrex 4y agoWhat's being skipped over in a lot of reporting is that Southwest has serious issues maintaining ground staff staffing, and has been struggling with that the entire pandemic because they refuse to pay market rate. This means that planes can't be unloaded and turned around in any reasonable amount of time. I gave up completely on southwest after I was stuck on a plane for 4 hours this summer, because of lack of ground crew to unload the planes, and thus planes were stuck on the tarmac. Add on to that the latest technology woes, the storm was just the thing that tipped over a company that had cut operational capacity to the bone.
- iudqnolq 4y agoA few days ago they declared a "state of emergency" in a few airports, which they're pretty transparently using to squeeze staff. I don't see any non-bogus reason to, for example, stop accepting doctor's letters from telehealth visits only in cities where they're particularly short staffed.
- greggarious 4y agoI basically only used them for the PIT -> LAS direct flight for Defcon, and while I never got enough miles or whatever for any kind of formal loyalty program, I got the impression that a decade and a half of respecting sky law[1] got me some conversations most passengers don't get to have :-) One of their pilots once confided the reason he had to have them plant a drink cart in the aisle while he popped out to use the bathroom was because they're given such little time between connections they can't even stop at a urinal. (<15 minutes) They flew too close to the sun, and they had a cascading failure. Womp womp! [1] https://www.youtube.com/watch?v=vK1HINtjdHY https://www.youtube.com/watch?v=vK1HINtjdHY
- thomasjudge 4y agoFun fact I found in an NYT article and LinkedIn: The current CEO Bob Jordan started at Southwest in 1988 as a computer programmer. From 2006-2008 he was EVP of Strategy and Technology
- mastercheif 4y agoFWIW, a SouthWest pilot with 35 YOE at the company placed most of the blame on the previous CEO's lack of investment in operations in chase of shareholder profits. The pilot gave the current CEO Bob Jordan (who started in early 2022) a vote of confidence and stated that he has strongly signaled his intention to improve the state of SW's systems, but obviously was unable to do so before this meltdown. Source: https://www.reddit.com/r/SouthwestAirlines/comments/zxg6op/the_history_of_swa_destruction_from_within/ https://www.reddit.com/r/SouthwestAirlines/comments/zxg6op/t...
- mgkimsal 4y agoSomeone in charge of an org that large, less than 10 months in, getting the blame for over a decade of skipped maintenance.... pretty unfair.
- kjellsbells 4y agoIn HN terms: imagine your assets are encoded in an array or list. Movement is represented by shifting elements right or left. Most airlines are hub and spoke. Elements shift right (planes leave the hub) then shift left (return). Sometimes they shift two places right. If something disrupts this, like, a plane is stuck in [1] when it needs to be in [0], its generally recoverable when the event clears, or, you can use one of those 2 step moves in another flight to pick up stranded passengers on your way back to the starting position [0]. Southwest operate a point to point model. Assets start at [0] and have to traverse every point on the list through to [N] to succeed. N can be 3, 4 or higher. You can imagine what happens if a disruption happens in the middle of this list. Everybody upstream of the break is left planeless, everybody downstream who wants to go up cant make it further than the airport before the break, and everybody at the breakpoint is miserable. So Southwest then fallback to a new scheduling model for which their system was not designed. Its like having a graph traversal system and trying to get it to solve matrix equations. Yeah, there are some similarities, but its not really the same.
- Tempest1981 4y agoSo how do other airlines track where their planes are? Radio? ADS-B? How about their crews? In-house mobile-phone app? The Southwest telephone-based system sounds super low-tech. AirTags might work better.
- joezydeco 4y agoEveryone knows where the planes are. It's the pilots and crews that are dead heading around the country to get into position that got lost.
- SoftTalker 4y agoThis is going to add fuel to the fire of demand for autonomous/remotely piloted aircraft. Airlines would like nothing more than to be able to fly passengers in drones and fire all their flight crew.
- FesterCluck 4y agoEveryone here is at least a little misinformed regarding their software and it's simulation abilities. They have control systems with the data they need, and this data is fed into the flight scheduling & monitoring UI. Their tech works, it just gets bashed on because it's different than a lot of other airlines. Used properly it can account for these things. What causes things like this is overbooking. Overbooking happens in more than just butts in seats. If you're not going to have enough ground crew to handle your SLA on flights, flights should be preemptively cancelled to keep them under that SLA. Airlines keep that SLA as low as safety permits. The max is set by the FAA at 3 hours for domestic flights and 4 hours for international flights (both ends). Southwest's software is doing it's job, it has adjustable tolerances. It will even take into account weather conditions reducing ground crew therefore raising SLA. But, the effect of the weather conditions on the ground crew is also adjustable by humans. As a matter of fact, while one would think it would just reference historical data, that's not exactly true. It references predictive models that are adjusted by experienced individuals. Those individuals can be ordered to adjust the parameters outside their honest assessment to allow for steps at the beginning of the process to operate smoothly. Weather prediction has a horizon. Southwest allowed excess bookings beyond that horizon, or they allowed excess last minute bookings. This weather event was massive and one-sided, yes, but it was also completely predictable. A human made the decision to widen the guardrails. There aren't a ton of people allowed to do that, I can think of maybe 3. Disclosure: I wrote software for SWA many years ago. Nothing I've said here is privileged, in fact most airlines operate this exact way.
- belfalas 4y ago> Nothing I've said here is privileged, in fact most airlines operate this exact way. I consulted for a few large carriers and this was my conclusion also. In fact most very large American enterprises have all the same issues. With airlines the problems simply become really visible all at once really quickly when things go wrong. For the folks here on HN whose experience is mainly in Silicon Valley: it's hard to appreciate how little the execs at these companies care about software. They don't care, not even a little bit, and they definitely don't care what your opinions are. Their priority is growth, the stock price, and answering to the board (not necessarily in that order). The only carrier I heard about that cared for their IT was Continental and they were bought by United. Compounding the issue is how staffing works at these companies. Being a full-time employee at an airline is considered attractive because you get benefits including cheap airline tickets. My understanding is that the airline ticket perk used to be much more awesome than it is today. In any case, airlines bend over backwards not to have full-time employees. Depending on the company they have a vast army of contractors (on shore and off shore) who are on a revolving door policy lasting from 6 to 18 months. These folks come in, get trained up for a few months, turn around tickets in a grueling and dehumanizing environment, then they get to take a hike for a while until they can come back on another short-term stint. I had a colleague who had been full-time at SWA, he said his job literally only consisted of training people up on the systems, he rarely wrote much code himself, he was there to 'keep it all together.' But honestly, the current crisis is not a surprise. The IT systems at big American enterprises are truly horrific. It is decades of homegrown software "integrated" with decades of acquisitions where systems are smashed together on short timelines in service of quarterly goals. If you want to find the true culprit here, look up at the broader structure of the economic system. This mess is created by how we run the economy.
- temptemp888 4y agoTruth is even they won't really know until they have time to do a post mortem. Some observations, though. - Via the departure control, manifests, etc...they DID know where their crews went. Any employee flying non-rev, deadhead, or on duty is recorded by employee number. There was likely a problem with keeping that synced with the crew management platform. - Large parts of this have little to do with technology. Point to point versus hub and spoke means less slack. And recently, SWA has removed even more slack trying to drive revenue with aggressively optimistic schedules, overbooking, tight crew slack, etc. If they didn't pull pack the schedule in small, manageable pieces like the other airlines did, there's no system that would save them. Seems likely they didn't pull back enough before the storm. There's a brink you can't go past, and a rate of cancels you shouldn't exceed. You have to guess right earlier. - Crew unions have historically negotiated out anything that looks like big brother, location tracking for example...even on-demand "I'm here". So lots of things that could have made this better didn't exist , on purpose. - A fair amount of the reset is just practicalities of using planes empty of pax to get rested crews staged in the right cities and bags to the right places.
- simne 4y agoDisclaimer: I've read comments before write my own. It remembers me long history of failures of post-soviet passenger railroad booking system. They constantly have issues, it is just impossible reliable buy tickets when hot season. Situation becomes much better latest years (to be honest, we just don't see them), just because free market - railroads give up large share of load to other transport - to air, buses, private autos, and also large part of tourists go abroad.
- simne 4y agoReasons for so long constant decline are simple. First, old Soviet style of management, unfortunately saved in govt backed monopoly. Second, constantly underpaid computers/software depts.