5 ms·

I don’t see how a full system reboot should take days. If you don’t care about serving customers (which they don’t right now), then the problem could be simplif

• mgsouth 4y ago
  Well, let's see... A bit of internet searching shows SWA has about 60,000 total employees. Roughly 6,000 of those are pilots. So probably about 10,000 cabin crew (3 or 4 flight attendants, 2 pilots per flight). Most of those 15,000+ people need to be in exactly the expected place at the expected time, or things will get snarled up. If any one of those 5 or 6 people is a no-show then the plane can't legally fly passengers. There's limited fungibility; one of the pilots has to be a Captain. You don't want two Captains flying, if for no other reason than some other flight will then be short a Captain. One of the cabin crew has to be a Purser (supervisor). It's really preferable to have at least one of the pilots flying to the airports they usually frequent. Off-shift crew still need to be in the proper place at the proper time, or the plane has to stop mid-cycle. Then there's the ground and gate crews. And you can't just say "go to your usual starting airport". Flights are 24/7. Days of the week matter. Holidays matter. Even your local burger joint doesn't have cast-in-stone schedules. Much less a huge interconnected network that's trying to reboot. Even if 80% of the crew could go to a "usual airport", how would you know if you were in the 20%? Oh... everybody has to phone in, or the system has to send a huge number of notifications out. System crash. (Remember, normally the system depends on scheduling in advance, and only a few last-minute changes need to be handled.) And how do these what, 20k+ people get where they should be? If they aren't already at that airport, then normally by hopping on a SWA flight. Which aren't happening. So now what? Book flights on other airlines? Who's going to be doing the booking? How does it get paid? How many employees have a company credit card (any?) Use the employee's cards? How many are maxed out for Christmas? So let's suppose you're scheduled everyone and everybody knows where there's supposed to be and when. What happens when 10% call back in (without crashing the system!) and say they can't get there on time. (Remember, many SWA customers are stuck and are having a hard time arraning alternate travel. Crew without access to SWA flights would have the same trouble.) You're going to either apply massive changes to the existing schedule, or start all over and reschedule everyone. Which is exactly the problem they're currently having. They can't handle massive corrections. What's worse, a crew member may think they're good, and then their travel gets delayed somehow. How sensitive is the scheduling? If 10% of the crew are no-show or delayed then about 9% of the planes are affected; every 8 hours 90 planes will have the following 8 hours of flights cancelled or delayed (throwing further monkey wrenches in the schedule). About 40 of those planes will be missing a pilot, which means they can't even be deadheaded to where they're supposed to be next. So how to reboot? They've had to cancel two-thirds of their flights, so apparently they're able to keep 1/3 of them going. Keep those flying so they can shuttle crew around. You're initially only scheduling 1/3 of the crew, so the poor overloaded system can handle it. 2/3 of the flights are just outright cancelled, days in advance, so the customer support load is reduced. ("I'm sorry, your flight has been cancelled. We can't reschedule you until x days from now." vs. much back-and-forth trying to find something with an overloaded system.) Slowly add additional crew and flights so the number of phone-ins is kept manageable.
  • amluto 4y ago
    As you said, this is certainly a complicated situation. But this is HN, and, from an optimization / constraint satisfaction problem perspective / scale perspective, it’s really not a very large problem. The “going home” problem is a standard programming contest problem. This is more constrained, but a globally optimal solution isn’t needed. Southwest literally has a program called SkySolver. It should work. So you need to choose destinations for a few thousand planes? Pick the place you would normally have them at 4 AM two days hence. Need captains familiar with the airports? Surely the captains who would normally fly from a given airport at 4 AM two days hence are familiar with those airports. This gives a good starting solution for further optimization. You need to get a message to ~20k people? Great, Twilio will do that with minimal effort. Will a budget of a couple million dollars, this is not hard. There are good SMT solvers available for free. CPLEX is expensive but not on this scale. Managing test cases on this scale is straightforward. Even in an emergency, something good enough to get to the point where SkySolver starts working again should be doable. People who manage electric grids have protocols for “black starts”. These protocols are messy and complicated, but they are developed in advance, and they work. Airlines should have the equivalent capability.
    • mgsouth 4y ago
      You touched upon the core issues. Yes, SkySolver should be able to get a solution quickly. Yes, there should be systems in place to co-ordinate invidividual actions of 20k people with rapid adjustments. Yes, they should have procedures in place to do "black starts". But they don't. And you can't design, test, and roll out even a new communications system to 20k users literally overnight. If they existing scheduling system doesn't already email crew, then do they have email addresses for all them? If so, it's probably in HR somewhere; if that's outsourced then it could take days for SWA to get that pulled out even on an expedited basis. If they send email, what percentage of crew will receive it? In what timeframe? Email campaigns are notoriously porous. And how do crewmembers provide feedback? A standalone web page? Great--how do you authenticate? By interfacing the existing system? What will that take? Probably not, though--it appears the existing feedback mechanism is a manual phone call to a person. Do crewmembers even have login IDs? Finally, even with perfect communication there's the problem of getting people where they need to be when the transport system is very flaky. I think that's probably the issue when the CEO said 'they would get things manually set up, and then something would happen and we'd have to start all over again.' The pilot's union and others have been critizing SWA for not investing in infrastructure. The previous CEO, responding to the previous melt-down, said something about 'you can't test for these kind of scenarios.' That's wrong. You can't do it easily or cheaply, but it is possible. They haven't done it, they haven't developed robust systems, and now they can't instantly resolve the problem.
• nemothekid 4y ago
  >then the problem could be simplified to getting every plain and every crew home to where they would spend the night under a normal schedule. I'm confused, maybe my calibration of the scale involved here is wrong, but how do you not see this process taking days? Even if you had perfect information of where all the planes and crews were and where they should be, just coordinating with various airports to schedule the flights would take days; and that's assuming every plane is (1) flying empty, (2) fueled and (3) isn't carrying any passenger luggage.
• jasonwatkinspdx 4y ago
  It's far more complicated than that. For example, there's a lot of strict regulation around how many hours crew can work, how much rest they need between, etc. Likewise planes can't just randomly fly where-ever they want, whenever they want. There's mandatory maintenance, etc. You also have a limited number of hangers and jetways at each airport, so you have to coordinate how planes move around to not overload things. Oh we need to be sure there's enough fuel, fresh oxygen bottles, and so many other things. The problem is nothing like writing a script to scrape ADS-B data. This is the classic fallacy of a programmer thinking the most cartoonish imagination of the technology is the problem while being completely blind to the fundamental difficulty of organizing large groups of humans in some activity.
  • landemva 4y ago
    The other carriers made it through this high volume period. Do they have better programmers?
    • jasonwatkinspdx 4y ago
      No, as the article explains they use more of a hub and spoke dispatch pattern, which has less chained dependencies.
    • taway22323 4y ago
      The other carriers do also have a tipping point of no return, where their software can't handle the situation. But, they were more aggressive about canceling flights in careful waves before they got to that point. Some of the observations here are correct, but there's a big missing piece. Southwest went too far in with an overoptimistic schedule. They won't say that because then there's no easy scapegoat, it's just a pure management thing.
      • angry_octet 4y ago
        There is always a trade-off between robustness and efficiency. By leaving less slack the SWA schedule is more efficient, but also more tightly strung. When failures exceeded the slack capacity the whole schedule fell apart. The problem of recovery is also extensively studied. But it seems like SW did not put enough effort into having fault tolerant restart. This whole event is a management decision.
• irjustin 4y ago
  I'm downvoting. It's simple to say, "oh it would be this easy..." and not consider a single real life scenario. > problem could be simplified to getting every plain and every crew home to where they would spend the night under a normal schedule This alone would take days under your own recommendation.
  • aliswe 4y ago
    Weeks
  • Sakos 4y ago
    "If I ignore all real-world constraints and all the regulatory burdens, it's TRIVIAL. Why can't they figure this out when I could solve it in an internet comment?" He's ignorant and naive and disrespectful of the workers involved.
• smaudet 4y ago
  As others have said, you are ridiculously oversimplifying the process. If you are running a little video game that plots flight charts, sure perhaps you could write such a script. You cannot, however, script a bunch of logisitics support together across disparate, independent, multinational airports. Add the obligitary paperwork (we don't really want to be shipping people around in coffins, enabling human trafficking or illegal substance smuggling, trust me), and add your standard bit of managerial incompetence managed from an excel document and sharepoint, and it's actually downright impressive if it's only a couple days to completely reboot the system. Arguably, management doing the standard "oh that will never happen" is probably why it's not even better - you would think the airports would be able to automatically "fix" themselves with a self healing protocol, but that was probably deemed too expensive and left out of the feature set.
  • amluto 4y ago
    Of course I’m oversimplifying the process. But my point is the actual number of flights, passenger and luggage loads and unloads, etc can be a lot lower than a normal workload. On a normal day, an average Southwest plane flies quite a few hops, loads many passengers, unloads many passengers, etc. For a worst case reset, the normal routing doesn’t need to be followed. Of course this will introduce complexity. But it will also need decently less than a normal day’s number of operations. If I were designing a solver for where to move aircraft and crews (which is apparently what SkySolver does), I would test it on random and malicious inputs. If it’s initialized with every crew member and every plane at an independent random airport in the US (where Southwest has service) with a few broken for good measure and a few airports shut down, it should still find a solution. Waiting 12-24 hours to start so none of the crew is timed out could be part of a valid solution.
    • bfung 4y ago
      It’s still probably not that simple; the problem seems like it can be analogous to some sort of knapsack and/or traveling salesman problem, which are NP-hard! Even testing would take a long time.
      • 4y ago
• benced 4y ago
  Something worth pointing out here is that according to levels.fyi, a tech lead there who has worked for 20 years has a TC of 174K. It’s unlikely their technical talent would be up to what you propose which is related to the apparent decision at Southwest to underinvest - by their own reckoning - in technology.
  • scarface74 4y ago
    This again the HN tech bubble. That TC is around average for most senior software engineers in most corporations in most US cities in the US. And getting into a tech company and having high compensation doesn’t require “top talent”. Just the ability to memorize DS&A (junior to mid) and memorize system design and being able to regurgitate answers to behavioral questions showing “scope” and “impact”. It takes knowing two or three well known algorithms to implement the scheduling problem they are trying to solve and system design chops to know how to scale it. But, as far as having to be on hold for hours, that’s a staffing issue. Most of their issues are symptoms of poor management - not software engineers.
    • benced 4y ago
      To be clear, I don’t think you can blame these software engineers. Management has decided the average value of a tech lead to them is about 175K and they are getting that quality of engineer. I think their current tech issues suggest that the value management is placing on engineers is lower than it should be. Like I said earlier in the thread - there are many bad engineers that are overpaid and many good engineers that are underpaid. On average though, compensation and quality are correlated. In the absence of additional information, compensation is the best metric we have to asses the quality of the underlying engineering talent. It also passes the sniff test - try using the Southwest Airlines app!
      • scarface74 4y ago
        Again, you are just as deep in your bubble as everyone else. There are 2.7 million developers in the US. The vast majority of them are paid between $80K-$170k. They are working in banks, government, smaller startups, etc. And it doesn’t take a $400K senior developer that can reverse a binary tree on the whiteboard while juggling bowling balls while riding a unicycle on a tightrope to make a mobile app. How much do you think Delta developers make in Atlanta (hint: they aren’t making over 200K). Their app is excellent. It never ceases to amaze me how little most people on HN know about compensation throughout the industry.