7 ms·

That is only part of the issue, not every airline is 100% hub system alot most are a pretty big mix. Further alot of major Hubs where imacted by the Storm, ye

That is only part of the issue, not every airline is 100% hub system alot most are a pretty big mix. Further alot of major Hubs where imacted by the Storm, yet those airlines where able to transition. Why? Well most airlines have mobile apps, and web portals and other techonology so their crews can be reassigned in almost real time, (just like I would get auto booked on a new flight before I even knew my flight was canceled via the mobile app) Instead Southwest has systems from the 80's they require crews and customers to call and talk to an actual live human...

masklinn 4y ago
> Instead Southwest has systems from the 80's they require crews and customers to call and talk to an actual live human... From what I understand (from /r/flying) it's the reporting / fixing of issues which is manual. This means when the reporting / fixing is overloaded and not given a respite it can't catch up, keeps getting more overloaded, and ultimately the scheduling system completely fell over (it lost track of crews entirely is what I understand). "Mobile apps, and web portals and other technology" is orthogonal to the issue at hand.
- AnimalMuppet 4y ago
  Not entirely orthogonal? If the fixing issues is manual (therefore slower), that means that once issues start happening, it's slower to recover, thereby making a total meltdown more likely. And when a total meltdown occurs, recovery is manual, therefore slower.
  - masklinn 4y ago
    And you can have mobile apps and a web portal in front of that and it won't help none.
    - AnimalMuppet 4y ago
      It will make it faster than manual phone calls. "Faster" actually will help here. If nothing else, it will help the recovery not take so long.
      - inferiorhuman 4y ago
        You can have a non-computerized system that is more efficient. Apps can and do fail, and I'll go out on an unpopular limb and suggest that what most folks working in tech would still fail frequently enough to be problematic in an airline of this size. The problem is that their disaster recovery plan (assuming that there is one) is highly centralized. Southwest has each and every body calling the scheduling department one at a time. The proper design (and what most other airlines actually do) is to decentralize a wee bit and have, for instance, crews report to station managers who in turn call the scheduling department.
    - mrandish 4y ago
      Mobile apps / web portals could have helped the part of the problem related to status updates of staff location/availability not ingested fast enough. Apparently communications ingest bottleneck was a significant part of why the overall problem tipped into an even worse runaway state with no viable recovery options short of "full reboot."
- phpisthebest 4y ago
  The interview I saw with one the exec seemed to indicate that even under normal conditions when a flight is cancelled each crew member needed to call into a call center to be reassigned.
  - Supermancho 4y ago
    All crew are also under a regulation that limits their time available for flight, similar to a trucker but with larger consequences. eg If you are on shift for a certain number of hours, waiting for a plane in a hotel or in the terminal, it_does_not_matter. You have to have a mandatory sleep block where you are now unavailable. Once thousands of staff started timing out, even after the weather clears, it_did_not_matter. They were now unavailable and their replacements hadn't come in due to the weather, leaving some locales understaffed and all schedules backed up (which continued to compound as weather shifted).
    - redtriumph 4y ago
      Similar to humans, is there a wait time [cooling time] for aircrafts also? I suspect before and after each takeoff and landing, there are bunch of tests run to verify proper function of aircraft parts and subsystems.
      - Supermancho 4y ago
        Transport aircraft can run continuously for multiple flights. They don't turn off the engines under certain conditions (short hops, type of plane, size of loads, etc). There are regulatory inspections that require it to hangar, but I'm not familiar enough to cite any.
      - TexasDawg 4y ago
        Aircraft have to be scheduled for various maintenance checks such as "A", "B", "C", and "D" (which are the heaviest maintenance checks and take them out of service for many days/weeks). They're of course checked by the Ramp Crew and also receive a pre-flight walk around by the pilots before take-off. The only wait time is if there is a necessary repair to deal with a warning/indicator light and related repair time, hopefully does not take the aircraft out of service. Better out of service than break down in-air.
- cratermoon 4y ago
  In a complex enough system, it's reasonable to assume that at any point in time something is broken and the system is not running at 100%. What this means is they probably had their reporting/fixing system running at the thin edge all the time, and once the failures exceeded a certain rate, it was game over. The fixing could never catch up.
- mlinsey 4y ago
  Mobile apps or a web portal absolutely could have helped, simply by massively increasing the throughput of crew members being able to report that they didn't make it to their assigned location. Which could have stopped the cascading effect much sooner. The manual "reporting/fixing" of issues, which was done over the phone, includes pilots and crew members reporting "hey, I didn't actually get on the flight I was scheduled to". If the system did not receive that message, it was built to assume that the crew members made it to their next location and would be available for their next scheduled flight. But once a certain threshold of flights got canceled because of the storm, the limiting factor became how quickly staff members could even report that they didn't make it to their next location - they'd stay on hold for hours, until their next flight in their next city was scheduled to leave, and the system would still assume they were on that flight because it hadn't yet heard otherwise. Lack of webapps certainly wasn't the only issue here - as the article notes, their whole point-to-point scheduling model is more vulnerable to cascading failure as well, vs. a hub-and-spoke model. But the phone-based system did certainly cause a feedback loop that kept the disruption happening for much longer than it should have. And moving away from point-to-point would have many more tradeoffs/downsides for consumers than simply modernizing their tech system.
- jrochkind1 4y ago
  I mean, a system for crew scheduling that requires manual notification and updating when flights get cancelled... seems to be missing very important functionality for a crew scheduling system?
Ekaros 4y ago
Major hub getting impacted is less of an issue. Either the planes and crews are stuck there, you cancel flights and send them home. Or they fly to nearest available location and wait until they can get back to the hub. And explaining that FAA or someone banned flying from this airport is much simpler.

masklinn 4y ago

> Instead Southwest has systems from the 80's they require crews and customers to call and talk to an actual live human... From what I understand (from /r/flying) it's the reporting / fixing of issues which is manual. This means when the reporting / fixing is overloaded and not given a respite it can't catch up, keeps getting more overloaded, and ultimately the scheduling system completely fell over (it lost track of crews entirely is what I understand). "Mobile apps, and web portals and other technology" is orthogonal to the issue at hand.

AnimalMuppet 4y ago
Not entirely orthogonal? If the fixing issues is manual (therefore slower), that means that once issues start happening, it's slower to recover, thereby making a total meltdown more likely. And when a total meltdown occurs, recovery is manual, therefore slower.
- masklinn 4y ago
  And you can have mobile apps and a web portal in front of that and it won't help none.
  - AnimalMuppet 4y ago
    It will make it faster than manual phone calls. "Faster" actually will help here. If nothing else, it will help the recovery not take so long.
    - inferiorhuman 4y ago
      You can have a non-computerized system that is more efficient. Apps can and do fail, and I'll go out on an unpopular limb and suggest that what most folks working in tech would still fail frequently enough to be problematic in an airline of this size. The problem is that their disaster recovery plan (assuming that there is one) is highly centralized. Southwest has each and every body calling the scheduling department one at a time. The proper design (and what most other airlines actually do) is to decentralize a wee bit and have, for instance, crews report to station managers who in turn call the scheduling department.
  - mrandish 4y ago
    Mobile apps / web portals could have helped the part of the problem related to status updates of staff location/availability not ingested fast enough. Apparently communications ingest bottleneck was a significant part of why the overall problem tipped into an even worse runaway state with no viable recovery options short of "full reboot."
phpisthebest 4y ago
The interview I saw with one the exec seemed to indicate that even under normal conditions when a flight is cancelled each crew member needed to call into a call center to be reassigned.
- Supermancho 4y ago
  All crew are also under a regulation that limits their time available for flight, similar to a trucker but with larger consequences. eg If you are on shift for a certain number of hours, waiting for a plane in a hotel or in the terminal, it_does_not_matter. You have to have a mandatory sleep block where you are now unavailable. Once thousands of staff started timing out, even after the weather clears, it_did_not_matter. They were now unavailable and their replacements hadn't come in due to the weather, leaving some locales understaffed and all schedules backed up (which continued to compound as weather shifted).
  - redtriumph 4y ago
    Similar to humans, is there a wait time [cooling time] for aircrafts also? I suspect before and after each takeoff and landing, there are bunch of tests run to verify proper function of aircraft parts and subsystems.
    - Supermancho 4y ago
      Transport aircraft can run continuously for multiple flights. They don't turn off the engines under certain conditions (short hops, type of plane, size of loads, etc). There are regulatory inspections that require it to hangar, but I'm not familiar enough to cite any.
    - TexasDawg 4y ago
      Aircraft have to be scheduled for various maintenance checks such as "A", "B", "C", and "D" (which are the heaviest maintenance checks and take them out of service for many days/weeks). They're of course checked by the Ramp Crew and also receive a pre-flight walk around by the pilots before take-off. The only wait time is if there is a necessary repair to deal with a warning/indicator light and related repair time, hopefully does not take the aircraft out of service. Better out of service than break down in-air.
cratermoon 4y ago
In a complex enough system, it's reasonable to assume that at any point in time something is broken and the system is not running at 100%. What this means is they probably had their reporting/fixing system running at the thin edge all the time, and once the failures exceeded a certain rate, it was game over. The fixing could never catch up.
mlinsey 4y ago
Mobile apps or a web portal absolutely could have helped, simply by massively increasing the throughput of crew members being able to report that they didn't make it to their assigned location. Which could have stopped the cascading effect much sooner. The manual "reporting/fixing" of issues, which was done over the phone, includes pilots and crew members reporting "hey, I didn't actually get on the flight I was scheduled to". If the system did not receive that message, it was built to assume that the crew members made it to their next location and would be available for their next scheduled flight. But once a certain threshold of flights got canceled because of the storm, the limiting factor became how quickly staff members could even report that they didn't make it to their next location - they'd stay on hold for hours, until their next flight in their next city was scheduled to leave, and the system would still assume they were on that flight because it hadn't yet heard otherwise. Lack of webapps certainly wasn't the only issue here - as the article notes, their whole point-to-point scheduling model is more vulnerable to cascading failure as well, vs. a hub-and-spoke model. But the phone-based system did certainly cause a feedback loop that kept the disruption happening for much longer than it should have. And moving away from point-to-point would have many more tradeoffs/downsides for consumers than simply modernizing their tech system.
jrochkind1 4y ago
I mean, a system for crew scheduling that requires manual notification and updating when flights get cancelled... seems to be missing very important functionality for a crew scheduling system?

Ekaros 4y ago

Major hub getting impacted is less of an issue. Either the planes and crews are stuck there, you cancel flights and send them home. Or they fly to nearest available location and wait until they can get back to the hub. And explaining that FAA or someone banned flying from this airport is much simpler.