8 ms·

It’s fascinating that the same hopscotch travel pattern that allows SWA to offer better service to more places is also what caused the network to suffer cascadi

It’s fascinating that the same hopscotch travel pattern that allows SWA to offer better service to more places is also what caused the network to suffer cascading failure. Once a critical mass of pieces (planes/crew) were out of position the whole network fell apart, and it’s large enough that it seems like neither the humans nor software can easily reason about how to resume operations. Hence the need for a “full system reboot” over many days. Anecdotally, I flew Southwest just before Christmas. The network was already buckling and we had major delays, but we were lucky and made it through. Despite the stress, the SWA crews were helpful, empathetic, and polite. They handled it better than I would have if I had been in their shoes.

firstSpeaker 4y ago
I imagine the system cannot account for where all the planes, staff, passengers are and where they want to go economically.
- bumby 4y ago
  According the the article, the system makes some non-sensical tasking: "In one example during the storm, the system assigned a pilot to deadhead on a flight from Baltimore to Manchester, N.H., and then back to Baltimore the next day, without ever flying a plane"* * The article defines deadheading as sending a pilot as a passenger to get to another location. It would be interesting to look at what the system is trying to optimize for to make such choices.
  - brazzy 4y ago
    Sounds to me more like a bug in an edge case rather than trying to optimize for the wrong thing. Or (more likely, I think) the first deadhead was planned to have the pilot take over a flight which was then cancelled after the pilot was already underway.
    - smaudet 4y ago
      Also sounds like a case of not throwing enough adversarial data at the system - you can't just code coverage your code, you can't even establish KPIs, you have to establish its performance under system failure (does it freeze or gracefully shut down, does it persist to disk, what happens when the disk is yanked out of the system), etc. Very few software shops I am aware of that do anything like this.
  - twobitshifter 4y ago
    If the solver is updating based on current events, it’s completely reasonable that something like that could happen. Manchester was projected to be short a pilot, but the flight made it on time, and Baltimore unexpectedly ended up short.
  - prewett 4y ago
    This sounds like what you might expect from a using a random process (genetic algorithms, simulated annealing) to solve the NP-complete problem. The randomness injects suboptimal routes like this, as well as more optimal ones, but the fitness/cost function has to distill a whole bunch of things into one scalar. In my experience what happens is that the different things you want to value sort of compete with each other. I'm guessing the current state of the system is quite suboptimal, and it might not be able to remove the randomness. It's quite feasible to get within a factor of 2 of the optimal solution (with much less processing power), which sounds great from a CS algorithm analysis standpoint, but a factor of 2 looks like an awful schedule from a human standpoint.
  - bell-cot 4y ago
    Compare the cost of keeping one surplus pilot forever deadheading around the system to the cost & disruption of canceling (say) one flight a month because "a pilot got sick, it was not legal to fly short-handed". Having a (relative) bunch of surplus critical-task workers, forever being shuffled to where they system guesses they're most likely suddenly be needed, makes perfect sense. (Yes, you'd have to be a bit more sophisticated, so your surplus pilots got enough flying time to stay certified. And didn't get pissed and quit. Assume that I know why RAID 5 is better than RAID 4:)
  - taway22323 4y ago
    "non-sensical tasking" That could also be the result of thrashing after they were really far into the mess. That is, the solver maybe did output something sensible, but that solution has to get into the system that's used at runtime. Someone may have run another solution, or did manual updates while it was too overwhelmed to get all the changes posted. The solvers also depend on other context that's changing underneath them.
TheCondor 4y ago
Is resumption difficult or is it resumption and then make whole the tens of thousands of customers that were supposed to be moved a week prior? No idea what SkySolver actually does in totality, I'm sure it's complicated but I would think a flight crew could indicate where they are right now and then it could maybe pickup the next possible course they could perform. Not sure why the phone lines "jam up" exactly, don't you have a hierarchical management structure for this sort of thing? Or do 1000 pilots all report to one person? They've got like 1000 planes and like 100-150 destinations, it's not the traveling sales man problem, an optimal plan isn't needed now so much as a functional one. Of course, it's easy to bitch about it not being hard when I've never seen the code. Maybe is also tracks hours and does payroll and a dozen other functions.
- lbotos 4y ago
  I suspect crews timing out also play a big factor into this planning challenge: https://flyingbynumbers.com/cabin-crew-duty-timeouts/ https://flyingbynumbers.com/cabin-crew-duty-timeouts/
  - deleted 4y ago
    [deleted]
  - TheCondor 4y ago
    Yes, but this has to be less of a problem when they've been canceling tons of flights, right?
    - toast0 4y ago
      Depends on the specifics of the duty time rules and how the flights were canceled. If the crew showed up and started getting the flight ready, but the flight was cancelled before takeoff, that may count as duty hours and could reduce their availability in the near term. If the duty time rules only count time after takeoff, or only time after doors are closed, those both offer a different set of times.
- masklinn 4y ago
cratermoon 4y ago
> Hence the need for a “full system reboot” over many days. My understanding is that the full system reboot wouldn't have taken all that long, it's just the the company was trying to do a major fix while keeping whatever was still sort-of-working running. As any sysop will tell you, patching a running system is all kinds of crazy risky.
amluto 4y ago
I don’t see how a full system reboot should take days. If you don’t care about serving customers (which they don’t right now), then the problem could be simplified to getting every plane and every crew home to where they would spend the night under a normal schedule. With few or no paying customers on each plane, there should be plenty of capacity to move misplaced crew members around. None of this needs to approximate normal routing, and fewer segments than normal are needed, since passengers can be ignored. That being said, the software sucks. Southwest may have lost track of where their employees are. The ground crews are quitting. I wouldn’t be utterly shocked if management doesn’t even have a good overview of their the planes are. (Obviously anyone halfway competent could hack up a script to find all the planes based on ADS-B data in a few hours. And it wouldn’t be terribly hard to text a link to all crew asking them to fill out a simple form with their location, nearest airport, and when they can get there. But this requires competence and agility.)
- mgsouth 4y ago
  Well, let's see... A bit of internet searching shows SWA has about 60,000 total employees. Roughly 6,000 of those are pilots. So probably about 10,000 cabin crew (3 or 4 flight attendants, 2 pilots per flight). Most of those 15,000+ people need to be in exactly the expected place at the expected time, or things will get snarled up. If any one of those 5 or 6 people is a no-show then the plane can't legally fly passengers. There's limited fungibility; one of the pilots has to be a Captain. You don't want two Captains flying, if for no other reason than some other flight will then be short a Captain. One of the cabin crew has to be a Purser (supervisor). It's really preferable to have at least one of the pilots flying to the airports they usually frequent. Off-shift crew still need to be in the proper place at the proper time, or the plane has to stop mid-cycle. Then there's the ground and gate crews. And you can't just say "go to your usual starting airport". Flights are 24/7. Days of the week matter. Holidays matter. Even your local burger joint doesn't have cast-in-stone schedules. Much less a huge interconnected network that's trying to reboot. Even if 80% of the crew could go to a "usual airport", how would you know if you were in the 20%? Oh... everybody has to phone in, or the system has to send a huge number of notifications out. System crash. (Remember, normally the system depends on scheduling in advance, and only a few last-minute changes need to be handled.) And how do these what, 20k+ people get where they should be? If they aren't already at that airport, then normally by hopping on a SWA flight. Which aren't happening. So now what? Book flights on other airlines? Who's going to be doing the booking? How does it get paid? How many employees have a company credit card (any?) Use the employee's cards? How many are maxed out for Christmas? So let's suppose you're scheduled everyone and everybody knows where there's supposed to be and when. What happens when 10% call back in (without crashing the system!) and say they can't get there on time. (Remember, many SWA customers are stuck and are having a hard time arraning alternate travel. Crew without access to SWA flights would have the same trouble.) You're going to either apply massive changes to the existing schedule, or start all over and reschedule everyone. Which is exactly the problem they're currently having. They can't handle massive corrections. What's worse, a crew member may think they're good, and then their travel gets delayed somehow. How sensitive is the scheduling? If 10% of the crew are no-show or delayed then about 9% of the planes are affected; every 8 hours 90 planes will have the following 8 hours of flights cancelled or delayed (throwing further monkey wrenches in the schedule). About 40 of those planes will be missing a pilot, which means they can't even be deadheaded to where they're supposed to be next. So how to reboot? They've had to cancel two-thirds of their flights, so apparently they're able to keep 1/3 of them going. Keep those flying so they can shuttle crew around. You're initially only scheduling 1/3 of the crew, so the poor overloaded system can handle it. 2/3 of the flights are just outright cancelled, days in advance, so the customer support load is reduced. ("I'm sorry, your flight has been cancelled. We can't reschedule you until x days from now." vs. much back-and-forth trying to find something with an overloaded system.) Slowly add additional crew and flights so the number of phone-ins is kept manageable.
MR4D 4y ago
I wonder if it’s that or simply a lack of slack in their system. It seems to me that just like pre-staged inventory helps in logistics management, that extra planes and crews in the rotation could improve operations under these circumstances.
- Tao3300 4y ago
  They probably had that kind of wiggle room once upon a time. Then some sort of pandemic turned everything to just enough planes and skeleton crews.
  - MR4D 4y ago
    It'll be interesting to see. I'm sure Congress with go through it pretty strongly (opinions, that is), assuming they ever have a successful vote for a Speaker.
- EMM_386 4y ago
  With a normal airline, you have pilots sitting on "reserve" at bases, who can be called in at any time to fill in any gaps that may occur. They are being paid but are not flying, it's quite a good gig if you can get on the reserve list. I don't know how this is handled at Southwest, who does not fly hub-and-spoke and thus doesn't have a bunch of pilots sitting reserve around a base at, say, Atlanta.
- mjevans 4y ago
  In the past I had a job where some contract required a trained body to be on site 24/7. The company hired EXACTLY enough workers to fill the position, with _zero_ slack for anything. That lack of slack is hell. It makes any disruption, even minor ones, require the other workers to work more time. Major disruptions mean soul-crushing crunch level hours to just get by. Slack _must_ be planned into a system, otherwise there won't be any safety / recovery margin, and you're seeing the results live with Southwest's implosion.
bogomipz 4y ago
>"Anecdotally, I flew Southwest just before Christmas. The network was already buckling and we had major delays, but we were lucky and made it through." Interesting. You don't say how far before Christmas you were traveling. Had this crazy weather system already started moving from West to East at that point? Or was the system buckling just from passenger volume at the point i.e similar to the Summer meltdown that Southwest had?
DrBazza 4y ago
> Once a critical mass of pieces (planes/crew) were out of position the whole network fell apart, and it’s large enough that it seems like neither the humans nor software can easily reason about how to resume operations. Hence the need for a “full system reboot” over many days. We had days like that in the UK but with the rail system. And it when it happens it’s also due to snow. We’ve seen it on global scale recently due to covid putting ships in the wrong places so that optimised shipping routes become a mess.

firstSpeaker 4y ago

I imagine the system cannot account for where all the planes, staff, passengers are and where they want to go economically.

bumby 4y ago
According the the article, the system makes some non-sensical tasking: "In one example during the storm, the system assigned a pilot to deadhead on a flight from Baltimore to Manchester, N.H., and then back to Baltimore the next day, without ever flying a plane"* * The article defines deadheading as sending a pilot as a passenger to get to another location. It would be interesting to look at what the system is trying to optimize for to make such choices.
- brazzy 4y ago
  Sounds to me more like a bug in an edge case rather than trying to optimize for the wrong thing. Or (more likely, I think) the first deadhead was planned to have the pilot take over a flight which was then cancelled after the pilot was already underway.
  - smaudet 4y ago
    Also sounds like a case of not throwing enough adversarial data at the system - you can't just code coverage your code, you can't even establish KPIs, you have to establish its performance under system failure (does it freeze or gracefully shut down, does it persist to disk, what happens when the disk is yanked out of the system), etc. Very few software shops I am aware of that do anything like this.
- twobitshifter 4y ago
  If the solver is updating based on current events, it’s completely reasonable that something like that could happen. Manchester was projected to be short a pilot, but the flight made it on time, and Baltimore unexpectedly ended up short.
- prewett 4y ago
  This sounds like what you might expect from a using a random process (genetic algorithms, simulated annealing) to solve the NP-complete problem. The randomness injects suboptimal routes like this, as well as more optimal ones, but the fitness/cost function has to distill a whole bunch of things into one scalar. In my experience what happens is that the different things you want to value sort of compete with each other. I'm guessing the current state of the system is quite suboptimal, and it might not be able to remove the randomness. It's quite feasible to get within a factor of 2 of the optimal solution (with much less processing power), which sounds great from a CS algorithm analysis standpoint, but a factor of 2 looks like an awful schedule from a human standpoint.
- bell-cot 4y ago
  Compare the cost of keeping one surplus pilot forever deadheading around the system to the cost & disruption of canceling (say) one flight a month because "a pilot got sick, it was not legal to fly short-handed". Having a (relative) bunch of surplus critical-task workers, forever being shuffled to where they system guesses they're most likely suddenly be needed, makes perfect sense. (Yes, you'd have to be a bit more sophisticated, so your surplus pilots got enough flying time to stay certified. And didn't get pissed and quit. Assume that I know why RAID 5 is better than RAID 4:)
- taway22323 4y ago
  "non-sensical tasking" That could also be the result of thrashing after they were really far into the mess. That is, the solver maybe did output something sensible, but that solution has to get into the system that's used at runtime. Someone may have run another solution, or did manual updates while it was too overwhelmed to get all the changes posted. The solvers also depend on other context that's changing underneath them.

TheCondor 4y ago

Is resumption difficult or is it resumption and then make whole the tens of thousands of customers that were supposed to be moved a week prior? No idea what SkySolver actually does in totality, I'm sure it's complicated but I would think a flight crew could indicate where they are right now and then it could maybe pickup the next possible course they could perform. Not sure why the phone lines "jam up" exactly, don't you have a hierarchical management structure for this sort of thing? Or do 1000 pilots all report to one person? They've got like 1000 planes and like 100-150 destinations, it's not the traveling sales man problem, an optimal plan isn't needed now so much as a functional one. Of course, it's easy to bitch about it not being hard when I've never seen the code. Maybe is also tracks hours and does payroll and a dozen other functions.

lbotos 4y ago
I suspect crews timing out also play a big factor into this planning challenge: https://flyingbynumbers.com/cabin-crew-duty-timeouts/ https://flyingbynumbers.com/cabin-crew-duty-timeouts/
- deleted 4y ago
  [deleted]
- TheCondor 4y ago
  Yes, but this has to be less of a problem when they've been canceling tons of flights, right?
  - toast0 4y ago
    Depends on the specifics of the duty time rules and how the flights were canceled. If the crew showed up and started getting the flight ready, but the flight was cancelled before takeoff, that may count as duty hours and could reduce their availability in the near term. If the duty time rules only count time after takeoff, or only time after doors are closed, those both offer a different set of times.
masklinn 4y ago

cratermoon 4y ago

> Hence the need for a “full system reboot” over many days. My understanding is that the full system reboot wouldn't have taken all that long, it's just the the company was trying to do a major fix while keeping whatever was still sort-of-working running. As any sysop will tell you, patching a running system is all kinds of crazy risky.

amluto 4y ago

I don’t see how a full system reboot should take days. If you don’t care about serving customers (which they don’t right now), then the problem could be simplified to getting every plane and every crew home to where they would spend the night under a normal schedule. With few or no paying customers on each plane, there should be plenty of capacity to move misplaced crew members around. None of this needs to approximate normal routing, and fewer segments than normal are needed, since passengers can be ignored. That being said, the software sucks. Southwest may have lost track of where their employees are. The ground crews are quitting. I wouldn’t be utterly shocked if management doesn’t even have a good overview of their the planes are. (Obviously anyone halfway competent could hack up a script to find all the planes based on ADS-B data in a few hours. And it wouldn’t be terribly hard to text a link to all crew asking them to fill out a simple form with their location, nearest airport, and when they can get there. But this requires competence and agility.)

mgsouth 4y ago
Well, let's see... A bit of internet searching shows SWA has about 60,000 total employees. Roughly 6,000 of those are pilots. So probably about 10,000 cabin crew (3 or 4 flight attendants, 2 pilots per flight). Most of those 15,000+ people need to be in exactly the expected place at the expected time, or things will get snarled up. If any one of those 5 or 6 people is a no-show then the plane can't legally fly passengers. There's limited fungibility; one of the pilots has to be a Captain. You don't want two Captains flying, if for no other reason than some other flight will then be short a Captain. One of the cabin crew has to be a Purser (supervisor). It's really preferable to have at least one of the pilots flying to the airports they usually frequent. Off-shift crew still need to be in the proper place at the proper time, or the plane has to stop mid-cycle. Then there's the ground and gate crews. And you can't just say "go to your usual starting airport". Flights are 24/7. Days of the week matter. Holidays matter. Even your local burger joint doesn't have cast-in-stone schedules. Much less a huge interconnected network that's trying to reboot. Even if 80% of the crew could go to a "usual airport", how would you know if you were in the 20%? Oh... everybody has to phone in, or the system has to send a huge number of notifications out. System crash. (Remember, normally the system depends on scheduling in advance, and only a few last-minute changes need to be handled.) And how do these what, 20k+ people get where they should be? If they aren't already at that airport, then normally by hopping on a SWA flight. Which aren't happening. So now what? Book flights on other airlines? Who's going to be doing the booking? How does it get paid? How many employees have a company credit card (any?) Use the employee's cards? How many are maxed out for Christmas? So let's suppose you're scheduled everyone and everybody knows where there's supposed to be and when. What happens when 10% call back in (without crashing the system!) and say they can't get there on time. (Remember, many SWA customers are stuck and are having a hard time arraning alternate travel. Crew without access to SWA flights would have the same trouble.) You're going to either apply massive changes to the existing schedule, or start all over and reschedule everyone. Which is exactly the problem they're currently having. They can't handle massive corrections. What's worse, a crew member may think they're good, and then their travel gets delayed somehow. How sensitive is the scheduling? If 10% of the crew are no-show or delayed then about 9% of the planes are affected; every 8 hours 90 planes will have the following 8 hours of flights cancelled or delayed (throwing further monkey wrenches in the schedule). About 40 of those planes will be missing a pilot, which means they can't even be deadheaded to where they're supposed to be next. So how to reboot? They've had to cancel two-thirds of their flights, so apparently they're able to keep 1/3 of them going. Keep those flying so they can shuttle crew around. You're initially only scheduling 1/3 of the crew, so the poor overloaded system can handle it. 2/3 of the flights are just outright cancelled, days in advance, so the customer support load is reduced. ("I'm sorry, your flight has been cancelled. We can't reschedule you until x days from now." vs. much back-and-forth trying to find something with an overloaded system.) Slowly add additional crew and flights so the number of phone-ins is kept manageable.

MR4D 4y ago

I wonder if it’s that or simply a lack of slack in their system. It seems to me that just like pre-staged inventory helps in logistics management, that extra planes and crews in the rotation could improve operations under these circumstances.

Tao3300 4y ago
They probably had that kind of wiggle room once upon a time. Then some sort of pandemic turned everything to just enough planes and skeleton crews.
- MR4D 4y ago
  It'll be interesting to see. I'm sure Congress with go through it pretty strongly (opinions, that is), assuming they ever have a successful vote for a Speaker.
EMM_386 4y ago
With a normal airline, you have pilots sitting on "reserve" at bases, who can be called in at any time to fill in any gaps that may occur. They are being paid but are not flying, it's quite a good gig if you can get on the reserve list. I don't know how this is handled at Southwest, who does not fly hub-and-spoke and thus doesn't have a bunch of pilots sitting reserve around a base at, say, Atlanta.
mjevans 4y ago
In the past I had a job where some contract required a trained body to be on site 24/7. The company hired EXACTLY enough workers to fill the position, with _zero_ slack for anything. That lack of slack is hell. It makes any disruption, even minor ones, require the other workers to work more time. Major disruptions mean soul-crushing crunch level hours to just get by. Slack _must_ be planned into a system, otherwise there won't be any safety / recovery margin, and you're seeing the results live with Southwest's implosion.

bogomipz 4y ago

>"Anecdotally, I flew Southwest just before Christmas. The network was already buckling and we had major delays, but we were lucky and made it through." Interesting. You don't say how far before Christmas you were traveling. Had this crazy weather system already started moving from West to East at that point? Or was the system buckling just from passenger volume at the point i.e similar to the Summer meltdown that Southwest had?

DrBazza 4y ago

> Once a critical mass of pieces (planes/crew) were out of position the whole network fell apart, and it’s large enough that it seems like neither the humans nor software can easily reason about how to resume operations. Hence the need for a “full system reboot” over many days. We had days like that in the UK but with the rail system. And it when it happens it’s also due to snow. We’ve seen it on global scale recently due to covid putting ships in the wrong places so that optimised shipping routes become a mess.