6 ms·

> "What's unique is the partial failure, it's never happened," he said. "This isn't a drill you can run." As someone who writes some very thorough unit tests..

> "What's unique is the partial failure, it's never happened," he said. "This isn't a drill you can run." As someone who writes some very thorough unit tests... and also have had to have mandatory training... I find "this isn't a drill you can run" to be _very_ wrong.

SoftTalker 4y ago
Southwest is a "discount" airline. They do many things to economize, i.e. no assigned seats, they only fly 737s so they don't need to certify pilots or mechanics on any other types, you can only book with them and not with Expedia etc. It would not surprise me that their back-office operations are likewise economized and some things are just not done because "they can never happen."
- mason55 4y ago
  > It would not surprise me that their back-office operations are likewise economized and some things are just not done because "they can never happen." Meh, it doesn't even have to be "never". It just has to be cost multiplied by frequency is less than the cost to prepare. If they lose $100m every five years due to a system failure, and it would cost $30m/year to plan for those failures, they it's just cheaper to let it happen. And I don't mean this in a judgmental, Fight Club-car recall speech kind of way. It's just business reality. At some point every business has to decide that the cost of planning for something is higher than the cost of letting it happen.
  - e_y_ 4y ago
    What's the value of the reputation risk of a major, very high profile failure? Sometimes businesses end up on the wrong side of that bet. They see only the costs but not the benefits of preparedness (by the time it fails, there will probably be a different CEO in charge) and make a bad call.
    - mason55 4y ago
      Of course, no argument there. Ideally when you make that kind of decision you take reputational risk into account, as well as, like, is this an existential risk? The airline industry feels like one where each year it's a different carrier who has some catastrophic scheduling failure. Today, everyone says they're never flying Southwest again. But if you fly semi-regularly then it won't take very long before you don't have any airlines left to fly on. For people who weren't affected, I doubt very many are even going to remember this. Personally, I remember that this kind of thing has happened recently with other carriers but I couldn't even tell you who. And people who were affected can mostly be bought off if you need to. Some vouchers & hotel reimbursement and it's just the cost of doing business. Plus, the airline industry has proven over and over that people are willing to put up with a lot when you have the cheapest prices. It's different from an industry that's built on reputation and trust. Like, a password manager, the only real thing you're selling is your reputation. Losing trust is a real existential threat. Security costs need to be in the bucket of either "yes, we will do it" or "it's so expensive that if we do it then we don't have a business anyway, so we'll skip it and pray."
      - hattmall 4y ago
        Scheduling won't ruin an airlines reputation. Crashing the planes is what ruins an airline. Southwest has only ever had two passenger deaths and one of those was an attempted hijacker beaten to death by other passengers.
        khuey 4y ago
        Eh, to a first approximation the FAA won't let you crash the planes. It's been 13 years since there was a fatal plane crash on a US passenger airline.
        variant 4y ago
        FAA has an important role, but market forces won't tolerate an airline that experiences crashes, even rare ones. Airlines are highly incented to be safe, both by regulation and by the market itself.
- AlotOfReading 4y ago
  All airlines economize. An airline that doesn't is a bankrupt airline because typical industry margins on flight are razor thin. Southwest isn't a particularly budget airline compared to modern budget carriers like spirit and ryanair that haven't copied the open boarding policy. I suspect the opportunity to upsell seats / luggage and have distinct classes outweighs the turnaround time costs of assigned seating.
- LarryMullins 4y ago
  > Southwest is a "discount" airline. They're also the "friendly airline", they easily have the most personable and friendly staff. I don't know what they do different, but Southwest employees treat me human and all the rest generally treat me like human trash. It's got to be a company culture thing, maybe connected to Southwest not having a first-class section. Usually I fly with Southwest whenever possible without thinking twice about it, but this outage and the outage last year are forcing me to reconsider. Better to deal with rude people than to have my flight delayed..
  - WWLink 4y ago
    Southwest also has pretty good size seats, afaik, relative to other airlines' economy seat size. Plus the whole "you're all the same so we treat you all the same" thing is nice. I do wish there was an international equivalent.
  - dkarl 4y ago
    Yep, the other airlines are in the business of selling "class" and "status," and it's part of their product differentiation strategy to treat you according to how much you pay.
  - midoridensha 4y ago
    >I don't know what they do different, but Southwest employees treat me human and all the rest generally treat me like human trash. It's got to be a company culture thing Definitely. Normal American customer service is to treat you like human trash, so obviously Southwest has decided to do this differently and so probably does things like only hiring friendly people, training them to be friendly and positive even in difficult situations, and checking on them somehow to make sure they're doing this and not just faking it for the interview and training. Other airlines obviously don't, but it's not just airlines, it's everywhere in American customer-facing business these days. Rudeness is just a normal part of America culture now.
    - indymike 4y ago
      > Rudeness is just a normal part of America culture now. I have a backpack that has a phone holder on the shoulder strap. When you click your phone in it, the camera is visible. I've had many a situation where someone was about to go full New York on me, but noticed the camera staring at them and toned it down. I've never once been recording.
- thomasjudge 4y ago
  Just as a note: they are about to issue a $458M dividend. They plan to spend $4-4.5B in 2023 on planes. How much are they spending on system modernization?
  - namdnay 4y ago
    > How much are they spending on system modernization A fortune, they only just finished an 8-year migration to Amadeus
burnte 4y ago
"I find "this isn't a drill you can run" to be _very_ wrong" As a IT-VP/CIO, the statement of "there's no way to test it" is not acceptable.
- justinator 4y ago
  I’m very certain Chaos Engineering is known in the airline industry
- dstroot 4y ago
  Then you are senior enough to know what “ then-CEO now-chairman Gary Kelly” really meant was “I haven’t funded our technology team well enough to have resources to test a scenario like this”.
  - mason55 4y ago
    Or "we decided that the cost to plan for this is so high that it's not even worth testing. If it happens then we're fucked anyway and we'll just eat it."
- ninkendo 4y ago
  Testing reveals the presence of bugs, never their absence. With hindsight you can always feel smugly superior in saying “you should have tested for this”, but there’s an infinitude of things you might need to test, and if you haven’t encountered a failure you didn’t test for, you’re probably just lucky.
- hattmall 4y ago
  You can drill the initial failure, but not really the cascading events. In something as large as a global airline you are dependant on 1000s of third parties actions and the weather. No simulated drill is going to be sufficient or realistic. The only way to really mitigate or plan for something like this is multiple layers of segregation so that events in one area have less or no impact on others. Then you could drill total failure in various segments.
  - deleted 4y ago
ninkendo 4y ago
It’s funny that you use unit tests as an example of it being possible to run drills for this kind of thing. Unit tests are by their definition not the kind of thing that simulates this kind of failure. Perhaps you have a false sense of security about what you’ve really been testing?
- inetknght 4y ago
  > Unit tests are by their definition not the kind of thing that simulates this kind of failure. Thorough unit and integration tests include failure modes. Mine include things like "what happens if the OS reports that the storage was unmounted during read/write" (because that was a failure often seen in production with some flaky SAN devices) and "what happens if the server stops responding" (because networks are generally unreliable) and "what happens if invalid (random) data is given" because data corruption is a frequent occurrence for similar reasons. > Perhaps you have a false sense of security about what you’ve really been testing? Perhaps. On the other hand I've seen a _lot_ of other developers test only the happy path and call it a day then spend days/weeks/months debugging failures.

SoftTalker 4y ago

Southwest is a "discount" airline. They do many things to economize, i.e. no assigned seats, they only fly 737s so they don't need to certify pilots or mechanics on any other types, you can only book with them and not with Expedia etc. It would not surprise me that their back-office operations are likewise economized and some things are just not done because "they can never happen."

mason55 4y ago
> It would not surprise me that their back-office operations are likewise economized and some things are just not done because "they can never happen." Meh, it doesn't even have to be "never". It just has to be cost multiplied by frequency is less than the cost to prepare. If they lose $100m every five years due to a system failure, and it would cost $30m/year to plan for those failures, they it's just cheaper to let it happen. And I don't mean this in a judgmental, Fight Club-car recall speech kind of way. It's just business reality. At some point every business has to decide that the cost of planning for something is higher than the cost of letting it happen.
- e_y_ 4y ago
  What's the value of the reputation risk of a major, very high profile failure? Sometimes businesses end up on the wrong side of that bet. They see only the costs but not the benefits of preparedness (by the time it fails, there will probably be a different CEO in charge) and make a bad call.
  - mason55 4y ago
    Of course, no argument there. Ideally when you make that kind of decision you take reputational risk into account, as well as, like, is this an existential risk? The airline industry feels like one where each year it's a different carrier who has some catastrophic scheduling failure. Today, everyone says they're never flying Southwest again. But if you fly semi-regularly then it won't take very long before you don't have any airlines left to fly on. For people who weren't affected, I doubt very many are even going to remember this. Personally, I remember that this kind of thing has happened recently with other carriers but I couldn't even tell you who. And people who were affected can mostly be bought off if you need to. Some vouchers & hotel reimbursement and it's just the cost of doing business. Plus, the airline industry has proven over and over that people are willing to put up with a lot when you have the cheapest prices. It's different from an industry that's built on reputation and trust. Like, a password manager, the only real thing you're selling is your reputation. Losing trust is a real existential threat. Security costs need to be in the bucket of either "yes, we will do it" or "it's so expensive that if we do it then we don't have a business anyway, so we'll skip it and pray."
    - hattmall 4y ago
      Scheduling won't ruin an airlines reputation. Crashing the planes is what ruins an airline. Southwest has only ever had two passenger deaths and one of those was an attempted hijacker beaten to death by other passengers.
      - khuey 4y ago
        Eh, to a first approximation the FAA won't let you crash the planes. It's been 13 years since there was a fatal plane crash on a US passenger airline.
        variant 4y ago
        FAA has an important role, but market forces won't tolerate an airline that experiences crashes, even rare ones. Airlines are highly incented to be safe, both by regulation and by the market itself.
AlotOfReading 4y ago
All airlines economize. An airline that doesn't is a bankrupt airline because typical industry margins on flight are razor thin. Southwest isn't a particularly budget airline compared to modern budget carriers like spirit and ryanair that haven't copied the open boarding policy. I suspect the opportunity to upsell seats / luggage and have distinct classes outweighs the turnaround time costs of assigned seating.
LarryMullins 4y ago
> Southwest is a "discount" airline. They're also the "friendly airline", they easily have the most personable and friendly staff. I don't know what they do different, but Southwest employees treat me human and all the rest generally treat me like human trash. It's got to be a company culture thing, maybe connected to Southwest not having a first-class section. Usually I fly with Southwest whenever possible without thinking twice about it, but this outage and the outage last year are forcing me to reconsider. Better to deal with rude people than to have my flight delayed..
- WWLink 4y ago
  Southwest also has pretty good size seats, afaik, relative to other airlines' economy seat size. Plus the whole "you're all the same so we treat you all the same" thing is nice. I do wish there was an international equivalent.
- dkarl 4y ago
  Yep, the other airlines are in the business of selling "class" and "status," and it's part of their product differentiation strategy to treat you according to how much you pay.
- midoridensha 4y ago
  >I don't know what they do different, but Southwest employees treat me human and all the rest generally treat me like human trash. It's got to be a company culture thing Definitely. Normal American customer service is to treat you like human trash, so obviously Southwest has decided to do this differently and so probably does things like only hiring friendly people, training them to be friendly and positive even in difficult situations, and checking on them somehow to make sure they're doing this and not just faking it for the interview and training. Other airlines obviously don't, but it's not just airlines, it's everywhere in American customer-facing business these days. Rudeness is just a normal part of America culture now.
  - indymike 4y ago
    > Rudeness is just a normal part of America culture now. I have a backpack that has a phone holder on the shoulder strap. When you click your phone in it, the camera is visible. I've had many a situation where someone was about to go full New York on me, but noticed the camera staring at them and toned it down. I've never once been recording.
thomasjudge 4y ago
Just as a note: they are about to issue a $458M dividend. They plan to spend $4-4.5B in 2023 on planes. How much are they spending on system modernization?
- namdnay 4y ago
  > How much are they spending on system modernization A fortune, they only just finished an 8-year migration to Amadeus

burnte 4y ago

"I find "this isn't a drill you can run" to be _very_ wrong" As a IT-VP/CIO, the statement of "there's no way to test it" is not acceptable.

justinator 4y ago
I’m very certain Chaos Engineering is known in the airline industry
dstroot 4y ago
Then you are senior enough to know what “ then-CEO now-chairman Gary Kelly” really meant was “I haven’t funded our technology team well enough to have resources to test a scenario like this”.
- mason55 4y ago
  Or "we decided that the cost to plan for this is so high that it's not even worth testing. If it happens then we're fucked anyway and we'll just eat it."
ninkendo 4y ago
Testing reveals the presence of bugs, never their absence. With hindsight you can always feel smugly superior in saying “you should have tested for this”, but there’s an infinitude of things you might need to test, and if you haven’t encountered a failure you didn’t test for, you’re probably just lucky.
hattmall 4y ago
You can drill the initial failure, but not really the cascading events. In something as large as a global airline you are dependant on 1000s of third parties actions and the weather. No simulated drill is going to be sufficient or realistic. The only way to really mitigate or plan for something like this is multiple layers of segregation so that events in one area have less or no impact on others. Then you could drill total failure in various segments.
- deleted 4y ago

ninkendo 4y ago

It’s funny that you use unit tests as an example of it being possible to run drills for this kind of thing. Unit tests are by their definition not the kind of thing that simulates this kind of failure. Perhaps you have a false sense of security about what you’ve really been testing?

inetknght 4y ago
> Unit tests are by their definition not the kind of thing that simulates this kind of failure. Thorough unit and integration tests include failure modes. Mine include things like "what happens if the OS reports that the storage was unmounted during read/write" (because that was a failure often seen in production with some flaky SAN devices) and "what happens if the server stops responding" (because networks are generally unreliable) and "what happens if invalid (random) data is given" because data corruption is a frequent occurrence for similar reasons. > Perhaps you have a false sense of security about what you’ve really been testing? Perhaps. On the other hand I've seen a _lot_ of other developers test only the happy path and call it a day then spend days/weeks/months debugging failures.