12 ms·

Same thing happened on July 20, 2016: https://www.dallasnews.com/business/local-companies/2016/08/10/cost-of-southwest-s-tech-outage-climbs-to-at-least-54-mill

• vrc 4y ago
  Google runs Disaster Recovery Training annually (DiRT) where security teams are tasked with simulating these “black swan” events. Seems like this practice needs to expand to more industries.
  • bobkazamakis 4y ago
    >Seems like this practice needs to expand to more industries. I think you've mistaken this for something immediately increases quarterly gains with no regard to long-term strategy.
    • Analemma_ 4y ago
      In that case the government should smash Southwest with a billion dollar fine so the cost of not doing this drilling exceeds the cost of doing it.
      • hallway_monitor 4y ago
        Asking the government to step in for additional regulation is rarely helpful. For this type of failure, the free market will determine whether processes and tools improve, or whether the status quo is good enough.
        • heavenlyblue 4y ago
          Free market?
        • ceejayoz 4y ago
          Regulation sometimes helps remind the free market that fuckups like this can come with real human costs.
          • variant 4y ago
            Regulation imposes those costs and more on real humans and rarely heads off fuckups. Market forces can correct here. Vote with your wallet.
        • astrange 4y ago
          What's an airline got to do with the free market? They're a extremely highly regulated business.
    • 23B1 4y ago
      What's most annoying is that there's plenty of employees on the front line who not only care about testing for this sort of thing, but it actually interests them, they're motivated by it, and they understand the dire reality of what happens – to them, primarily – if the company isn't prepared to handle it. And you can guess what their managers' response typically is: "We need to focus on OKRs and QBRs and KPIs right now... maybe next quarter" I'm fully convinced that achieving 'manager status' is directly correlated to cowardice. Companies need top-down decision-making, but those decision-makers need to spend more time on the front line.
      • giantrobot 4y ago
        > Companies need top-down decision-making, but those decision-makers need to spend more time on the front line. This is not rewarded so it doesn't happen. Managers are rewarded for line goes up so they only focus on line goes up. If line ever doesn't go up it costs them money (advancement, compensation) even if there's little they could have done to make line go up.
    • dzhiurgis 4y ago
      Until regulation steps in.
    • kevinmchugh 4y ago
      My wife and I spent a few hours this morning dealing with the cancellation of our return flight. Southwest has long been preferable for me in many cases, including my most flown route. Between the headache of this outage and the apparently dismal state of their operations there's a strong chance I never fly with them again.
      • jdeibele 4y ago
        Our family was in San Jose last week when our Southwest flight was cancelled. It used to be that my first priority would be to go into the terminal and try to talk to somebody. I figured they were the experts. From what I've read, the staff use an antiquated system that takes you from one airport to another, then they can try to get you from that city to where you want to go. That's why there's so much tapping of keys and why it takes so long. It's better to present them with a route that you've found on Google Flights or similar. The Southwest first flight out was supposed to be yesterday evening, the day after Christmas. In our case, the only thing we could find before Christmas was getting us from SJC to Seattle via Phoenix on Alaska. We ended up renting a car and driving home to Portland. Things got bad around Eugene - I stopped counting after 40 wrecked cars and semis - and got worse as you got closer to Portland.
    • astrange 4y ago
      Airlines don't have quarterly gains. They regularly go bankrupt and get picked up again, because the country needs airlines and because they have large union contracts. See any gains here? https://www.google.com/finance/quote/LUV:NYSE?window=5Y https://www.google.com/finance/quote/LUV:NYSE?window=5Y
  • rumdonut 4y ago
    I figure most companies are too small for that to be budgeted. Though, it’s possibly a good selling point for cloud if it’s capable of it.
    • JustLurking2022 4y ago
      Cloud doesn't solve badly designed processes or poorly written software, which seem to be at play with Southwest. Yes, it can help provide more stable infrastructure and there are some (but by no means all) black swan events that can be mitigated simply by throwing more kit at the problem during a surge but it's no silver bullet.
  • splistud 4y ago
    [dead]
  • tomjakubowski 4y ago
    Training for these scenarios may help with responding to true black swan events, like Rick Rescorla's WTC evacuation drills ahead of 9/11. But, nitpicking, if you've predicted something will happen, by way of simulating it, it's not a black swan when it does.
    • adrr 4y ago
      There are cost considerations. Business continuity costs money. Finance firms have significant capital and income to have empty but built out building around around airports for business continuity. Which doesn’t even make sense since they can work from home as proven with covid. Airlines can’t work from home.
  • paganel 4y ago
    Google is also a trillion dollar company, as other have pointed out Soutwest is a low-cost carrier which most probably doesn't have the luxury of hiring FAANG-level engineers on 500k yearly comp in order to best simulate "black swan" events.
    • deleted 4y ago
      [deleted]
    • makestuff 4y ago
      This is probably the best argument for AWS/GCP/Azure even though it is becoming more and more obvious you don't really save that much money. If you have a black swan event like this and you listened to your solutions architect you will have a disaster recovery plan or even better a multi region setup. Worst case you have highly paid support engineers at the cloud providers who will do everything they can to get you back online.
      • 0cf8612b2e1e 4y ago
        This does not seem like a hardware failure scenario where the cloud has anything to offer. More like their intricate software/database systems became out of sync with reality and disentangling the mess is a highly manual process.
        • makestuff 4y ago
          In this case no, but I was more referring to the 2016 delta ground stop that was due to their datacenter burning to the ground.
          • therealjumbo 4y ago
            I remember reading about the Delta incident a ways back, here they claim it cost them ~$150 million. https://www.datacenterknowledge.com/archives/2016/09/08/delta-data-center-outage-cost-us-150m https://www.datacenterknowledge.com/archives/2016/09/08/delt... That's not the article I hoped to find however. I seem to remember there was another article where they hired a investigator/consultant to figure out the price to migrate to the cloud and ensure "this never happens again." My recollection of that was: their scheduling/ops team is also in the same city (Atlanta GA) as this datacenter, and that teams work was brought to a halt by the datacenter outage. The investigator concluded that Delta would need redundant copies of the ops team or the whole effort of moving the software to the cloud would just be at risk to something happening to the human team all in the same city. That would obviously cost to much money, so Delta decided to skip it.
  • thriftwy 4y ago
    Yandex used to run datacenter loss training every week, where they will nullroute one DC and see what breaks all while handling live taffic.
  • adamsb6 4y ago
    At Facebook we would simulate an entire datacenter disappearing. When we first started doing it the datacenter would be chosen months in advance so that teams would have plenty of time to ensure their services can run without that specific datacenter. When I left this year, the datacenter would be randomly chosen on the same day it would be cut off.
    • chasd00 4y ago
      That's pretty cool and ideal practice for a software firm but in one of the reddit threads they're talking about mass quits/refusal to work of ground crew at Denver because of the weather. I wonder how you could ever prepare for that? Keep a backup, airport scoped, ground crew in the waiting room?? You can't really do hot spares for people without time to gear/train up and the weather event is so widespread I doubt there's enough spare SWA human capacity across the whole nation even if you had C130s on standby everywhere ready to take workers where they're needed most. From a national security perspective, situations like this is why the Marines exist right? Ensure a rapid response while the rest of the machine gets moving. I feel bad for everyone involved, those affected and those trying to figure out a solution.
      • landemva 4y ago
        > I wonder how you could ever prepare for that? Management could consider how pay and performance programs can help ensure business continuity. HR and MBA xls wizards don't understand how to manage for business longevity.
        • dpkirchner 4y ago
          Pay might not really be enough. Maybe management could try to find folks to babysit kids/take care of parents trapped at home, freeing up workers to come fly. They'll certainly fail, but at least they'll understand the plight of their workers.
          • bsg75 4y ago
  • Scarblac 4y ago
    Google is made of money, and the reason they are is not because of DiRT. Other industries can't afford the same things that Google can while continuing to be a business.
    • fragmede 4y ago
      After canceling 5,400 flights, I can't see how Southwest can afford not to test. Even if they only made $1000 off each flight, that's still $5 million they just lost.
      • Scarblac 4y ago
        They probably would not have made it this far if they tested for every possible scenario, their margins are razor thin. Anyway, doesn't even the best testing only catch 40% of bugs or thereabouts? It's not a silver bullet.
        • fragmede 4y ago
          It's not any bullet, but spending a week each year to do downtime testing is going to expose more issues than not spending any time on it at all.
  • landemva 4y ago
    This does not need Google deep pockets. It needs the motivation and some funding. SWA does not care. Pull the backup tapes, hand those to DR team, provide bare metal, and start the stopwatch. I participated in this in 1990s across the Mississippi.
  • barrenko 4y ago
    Personally one of the basic tenets of my adulthood is realizing how many companies are a hair away from a similar scenario (differing in magnitude from an airline ofc).
    • bookofjoe 4y ago
      EMP FTW
  • aspectmin 4y ago
    That... and Chaos Engineering in general, and also just generally a forward looking team that identifies threats, vulnerabilities, and risks in the future and works backwards to identify potential mitigating controls. (Look forward, reason backward).
• bombcar 4y ago
  1/1/2000 sounds like a default value when it lost the data or never had it. Even more obvious would be if it threw you to 1/1/70.
  • Kon-Peki 4y ago
    On our database systems, we have some date fields for which the default value should never, ever be used and if it is, there is a big problem. All of those dates are set to the dates of well-known natural disasters that happened in the 1800s or earlier. The thought was that it needs to be something that isn’t believable to a non-technical user seeing it on their computer screen. It turns out that this is not necessarily useful. I listened to a guy talking about some issues with a record; he says “1871? What’s up with that?” And then just moved on as if “well it came out of the computer, must be right” or something. I think that databases need to have the concept of NaN for dates and time stamps, except that this should be configurable to something like a poop emoji or something like ⁉🆘. It has to be something where your grandma would look at it and confidently say “your computer is broken”
    • CydeWeys 4y ago
      > I think that databases need to have the concept of NaN for dates and time stamps, except that this should be configurable to something like a poop emoji or something like ⁉ How about just NULL?
      • Gwypaas 4y ago
        "The server returned an unexpected error." Now GLHF!
      • Kon-Peki 4y ago
        Making database columns nullable isn’t a free ride. In some situations, you are trading one known point of failure for a million unknown ones. Among other problems :)
        • CydeWeys 4y ago
• inetknght 4y ago
  > "What's unique is the partial failure, it's never happened," he said. "This isn't a drill you can run." As someone who writes some very thorough unit tests... and also have had to have mandatory training... I find "this isn't a drill you can run" to be _very_ wrong.
  • SoftTalker 4y ago
    Southwest is a "discount" airline. They do many things to economize, i.e. no assigned seats, they only fly 737s so they don't need to certify pilots or mechanics on any other types, you can only book with them and not with Expedia etc. It would not surprise me that their back-office operations are likewise economized and some things are just not done because "they can never happen."
    • mason55 4y ago
      > It would not surprise me that their back-office operations are likewise economized and some things are just not done because "they can never happen." Meh, it doesn't even have to be "never". It just has to be cost multiplied by frequency is less than the cost to prepare. If they lose $100m every five years due to a system failure, and it would cost $30m/year to plan for those failures, they it's just cheaper to let it happen. And I don't mean this in a judgmental, Fight Club-car recall speech kind of way. It's just business reality. At some point every business has to decide that the cost of planning for something is higher than the cost of letting it happen.
      • e_y_ 4y ago
        What's the value of the reputation risk of a major, very high profile failure? Sometimes businesses end up on the wrong side of that bet. They see only the costs but not the benefits of preparedness (by the time it fails, there will probably be a different CEO in charge) and make a bad call.
        • mason55 4y ago
          Of course, no argument there. Ideally when you make that kind of decision you take reputational risk into account, as well as, like, is this an existential risk? The airline industry feels like one where each year it's a different carrier who has some catastrophic scheduling failure. Today, everyone says they're never flying Southwest again. But if you fly semi-regularly then it won't take very long before you don't have any airlines left to fly on. For people who weren't affected, I doubt very many are even going to remember this. Personally, I remember that this kind of thing has happened recently with other carriers but I couldn't even tell you who. And people who were affected can mostly be bought off if you need to. Some vouchers & hotel reimbursement and it's just the cost of doing business. Plus, the airline industry has proven over and over that people are willing to put up with a lot when you have the cheapest prices. It's different from an industry that's built on reputation and trust. Like, a password manager, the only real thing you're selling is your reputation. Losing trust is a real existential threat. Security costs need to be in the bucket of either "yes, we will do it" or "it's so expensive that if we do it then we don't have a business anyway, so we'll skip it and pray."
• NovemberWhiskey 4y ago
  >"What's unique is the partial failure, it's never happened," he said. "This isn't a drill you can run." The unspoken part you have to hear there is "... within the economic model of the airline business". Business continuity gets exponentially more expensive as you chase the blackest of swans: the sheer volume of plan development and maintenance, developing exercises, table-top vs. walkthrough vs. simulation, assumptions about how many different uncorrelated failures you're prepared for deal with at once etc. I've no doubt you could run an airline to be as resilient as (say) USAF Air Mobility Command, but no-one could afford the tickets.
  • Frost1x 4y ago
    What's ironic here is that groups like USAF are constantly pressured to adopt private industry models to be more "economically efficient" and completely ignoring that resiliency is a requirement baked into the high cost. I understand why both take the approaches they do but it seems everyone holds private industry barely running with no resiliency optimizations above all else, which don't make sense in all contexts. Corner cutting is fine in many contexts, especially when you know the side effects of their failures which may be quite insignificant.
• Chris2048 4y ago
  > Delta had a similar outage due to a datacenter fire They only have one geo-located data-centre?
• splonk 4y ago
  > many airlines still need to modernize. Most of the travel industry runs on old software that would horrify a lot of people here, especially those who've never worked for a large, 30+ year old company. When I used to interview a lot of people I made it a point to mention some of the more "interesting" aspects so they'd know what they were getting into. One example: ever tried to book a flight a year in advance? On a lot (almost all?) of systems you can't, because the underlying date format is "DEC27". Edit to address a couple comments: logistics are hard and there are plenty of reasons why airlines wouldn't want to support booking that far out. However, the reason you can book a flight 330 days from now but not 360 days from now is almost certainly due to the date format. (I believe the windows used are less than 365 days because it's helpful to be able to have dates in the recent past. I remember seeing documentation for 360, but AA and United seem to be in the 330-340 range on their websites).
  • Frost1x 4y ago
    While this is humorous in that there are limiting assumptions like this baked into the system, I also have to wonder, who needs or even wants to book a flight a year in advance? I dread planning out a flight 4 months in advance and dealing with the almost inevitable cascade of conflicts this introduces of juggling and rescheduling things to make things align correctly. One year makes me cringe.
    • steveklabnik 4y ago
      I could see it for major holidays. I spent too much money to fly home this year because I am bad at scheduling. I would consider booking next year's flight during this year's trip just so I know it's knocked out and I don't have to worry about it.
    • dendrite9 4y ago
      There are events that I can see purchasing flights well in advance for. I used to a go to a conference that was held every other year at the same time, it would have been easy to buy tickets more than a year in advance for that without much concern. Eclipses, certain sporting events, or reservations for activities with a wait list of more than a year could qualify as well. Despite that I am like you and rarely have tickets far in advance of a trip.
• babyshake 4y ago
  > "This isn't a drill you can run." When characterized as something that can't be done instead of something they don't know how to do, you know exactly where they are on the Dunning–Kruger curve.
• photochemsyn 4y ago
  Modernization of equipment, hiring more pilots and other employees, investing in updating the code base - how can that be done? It's far more important to keep the stock price high by whatever means necessary, such as using government bailouts to buy back shares. Investment capitalism is really a garbage system when it comes to building and maintaining basic infrasctructure like transportation, electricity grids, roads and so on. China has demonstrated that convincingly over the past two decades, hasn't it?
• pksebben 4y ago
  "this isn't a drill you can run". And yet, Netflix has chaos Kong do it with regularity. The difference between what's true, what some people will buy, and what you can get away with saying is gross, y'all.
  • Supermancho 4y ago
    > "this isn't a drill you can run". And yet, Netflix has chaos Kong do it with regularity. Netflix and airlines are so different as to make this comparison laughable. The cost of setup and consequence of problems actually being found (ie Federal Regulations) that are not addressable (it's not like SWA didn't know about some of the eventualities), easily outclasses the need for testing every combination of situations. Kong doesn't run anything that has to do with weather turning jet fuel into sludge or 12x pre-staffing in case of massive computer failures along with assessing the possible legal consequences from each locale. The hubris of pretending that physical services on a national scale, is as deterministic as a complex automated system, is unsurprising from a certain crowd, I guess.
  • anarticle 4y ago
    Airplanes aren't 21st century move fast and break things software, please keep them that way!