8 ms·

Google runs Disaster Recovery Training annually (DiRT) where security teams are tasked with simulating these “black swan” events. Seems like this practice needs

by vrc 4y ago

Google runs Disaster Recovery Training annually (DiRT) where security teams are tasked with simulating these “black swan” events. Seems like this practice needs to expand to more industries.

bobkazamakis 4y ago
>Seems like this practice needs to expand to more industries. I think you've mistaken this for something immediately increases quarterly gains with no regard to long-term strategy.
- Analemma_ 4y ago
  In that case the government should smash Southwest with a billion dollar fine so the cost of not doing this drilling exceeds the cost of doing it.
  - hallway_monitor 4y ago
    Asking the government to step in for additional regulation is rarely helpful. For this type of failure, the free market will determine whether processes and tools improve, or whether the status quo is good enough.
    - heavenlyblue 4y ago
      Free market?
    - ceejayoz 4y ago
      Regulation sometimes helps remind the free market that fuckups like this can come with real human costs.
      - variant 4y ago
        Regulation imposes those costs and more on real humans and rarely heads off fuckups. Market forces can correct here. Vote with your wallet.
    - astrange 4y ago
      What's an airline got to do with the free market? They're a extremely highly regulated business.
- 23B1 4y ago
  What's most annoying is that there's plenty of employees on the front line who not only care about testing for this sort of thing, but it actually interests them, they're motivated by it, and they understand the dire reality of what happens – to them, primarily – if the company isn't prepared to handle it. And you can guess what their managers' response typically is: "We need to focus on OKRs and QBRs and KPIs right now... maybe next quarter" I'm fully convinced that achieving 'manager status' is directly correlated to cowardice. Companies need top-down decision-making, but those decision-makers need to spend more time on the front line.
  - giantrobot 4y ago
    > Companies need top-down decision-making, but those decision-makers need to spend more time on the front line. This is not rewarded so it doesn't happen. Managers are rewarded for line goes up so they only focus on line goes up. If line ever doesn't go up it costs them money (advancement, compensation) even if there's little they could have done to make line go up.
- dzhiurgis 4y ago
  Until regulation steps in.
- kevinmchugh 4y ago
  My wife and I spent a few hours this morning dealing with the cancellation of our return flight. Southwest has long been preferable for me in many cases, including my most flown route. Between the headache of this outage and the apparently dismal state of their operations there's a strong chance I never fly with them again.
  - jdeibele 4y ago
    Our family was in San Jose last week when our Southwest flight was cancelled. It used to be that my first priority would be to go into the terminal and try to talk to somebody. I figured they were the experts. From what I've read, the staff use an antiquated system that takes you from one airport to another, then they can try to get you from that city to where you want to go. That's why there's so much tapping of keys and why it takes so long. It's better to present them with a route that you've found on Google Flights or similar. The Southwest first flight out was supposed to be yesterday evening, the day after Christmas. In our case, the only thing we could find before Christmas was getting us from SJC to Seattle via Phoenix on Alaska. We ended up renting a car and driving home to Portland. Things got bad around Eugene - I stopped counting after 40 wrecked cars and semis - and got worse as you got closer to Portland.
- astrange 4y ago
  Airlines don't have quarterly gains. They regularly go bankrupt and get picked up again, because the country needs airlines and because they have large union contracts. See any gains here? https://www.google.com/finance/quote/LUV:NYSE?window=5Y https://www.google.com/finance/quote/LUV:NYSE?window=5Y
rumdonut 4y ago
I figure most companies are too small for that to be budgeted. Though, it’s possibly a good selling point for cloud if it’s capable of it.
- JustLurking2022 4y ago
  Cloud doesn't solve badly designed processes or poorly written software, which seem to be at play with Southwest. Yes, it can help provide more stable infrastructure and there are some (but by no means all) black swan events that can be mitigated simply by throwing more kit at the problem during a surge but it's no silver bullet.
splistud 4y ago
[dead]
tomjakubowski 4y ago
Training for these scenarios may help with responding to true black swan events, like Rick Rescorla's WTC evacuation drills ahead of 9/11. But, nitpicking, if you've predicted something will happen, by way of simulating it, it's not a black swan when it does.
- adrr 4y ago
  There are cost considerations. Business continuity costs money. Finance firms have significant capital and income to have empty but built out building around around airports for business continuity. Which doesn’t even make sense since they can work from home as proven with covid. Airlines can’t work from home.
paganel 4y ago
Google is also a trillion dollar company, as other have pointed out Soutwest is a low-cost carrier which most probably doesn't have the luxury of hiring FAANG-level engineers on 500k yearly comp in order to best simulate "black swan" events.
- deleted 4y ago
  [deleted]
- makestuff 4y ago
  This is probably the best argument for AWS/GCP/Azure even though it is becoming more and more obvious you don't really save that much money. If you have a black swan event like this and you listened to your solutions architect you will have a disaster recovery plan or even better a multi region setup. Worst case you have highly paid support engineers at the cloud providers who will do everything they can to get you back online.
  - 0cf8612b2e1e 4y ago
    This does not seem like a hardware failure scenario where the cloud has anything to offer. More like their intricate software/database systems became out of sync with reality and disentangling the mess is a highly manual process.
    - makestuff 4y ago
      In this case no, but I was more referring to the 2016 delta ground stop that was due to their datacenter burning to the ground.
      - therealjumbo 4y ago
        I remember reading about the Delta incident a ways back, here they claim it cost them ~$150 million. https://www.datacenterknowledge.com/archives/2016/09/08/delta-data-center-outage-cost-us-150m https://www.datacenterknowledge.com/archives/2016/09/08/delt... That's not the article I hoped to find however. I seem to remember there was another article where they hired a investigator/consultant to figure out the price to migrate to the cloud and ensure "this never happens again." My recollection of that was: their scheduling/ops team is also in the same city (Atlanta GA) as this datacenter, and that teams work was brought to a halt by the datacenter outage. The investigator concluded that Delta would need redundant copies of the ops team or the whole effort of moving the software to the cloud would just be at risk to something happening to the human team all in the same city. That would obviously cost to much money, so Delta decided to skip it.
thriftwy 4y ago
Yandex used to run datacenter loss training every week, where they will nullroute one DC and see what breaks all while handling live taffic.
adamsb6 4y ago
At Facebook we would simulate an entire datacenter disappearing. When we first started doing it the datacenter would be chosen months in advance so that teams would have plenty of time to ensure their services can run without that specific datacenter. When I left this year, the datacenter would be randomly chosen on the same day it would be cut off.
- chasd00 4y ago
  That's pretty cool and ideal practice for a software firm but in one of the reddit threads they're talking about mass quits/refusal to work of ground crew at Denver because of the weather. I wonder how you could ever prepare for that? Keep a backup, airport scoped, ground crew in the waiting room?? You can't really do hot spares for people without time to gear/train up and the weather event is so widespread I doubt there's enough spare SWA human capacity across the whole nation even if you had C130s on standby everywhere ready to take workers where they're needed most. From a national security perspective, situations like this is why the Marines exist right? Ensure a rapid response while the rest of the machine gets moving. I feel bad for everyone involved, those affected and those trying to figure out a solution.
  - landemva 4y ago
    > I wonder how you could ever prepare for that? Management could consider how pay and performance programs can help ensure business continuity. HR and MBA xls wizards don't understand how to manage for business longevity.
    - dpkirchner 4y ago
      Pay might not really be enough. Maybe management could try to find folks to babysit kids/take care of parents trapped at home, freeing up workers to come fly. They'll certainly fail, but at least they'll understand the plight of their workers.
      - bsg75 4y ago
Scarblac 4y ago
Google is made of money, and the reason they are is not because of DiRT. Other industries can't afford the same things that Google can while continuing to be a business.
- fragmede 4y ago
  After canceling 5,400 flights, I can't see how Southwest can afford not to test. Even if they only made $1000 off each flight, that's still $5 million they just lost.
  - Scarblac 4y ago
    They probably would not have made it this far if they tested for every possible scenario, their margins are razor thin. Anyway, doesn't even the best testing only catch 40% of bugs or thereabouts? It's not a silver bullet.
    - fragmede 4y ago
      It's not any bullet, but spending a week each year to do downtime testing is going to expose more issues than not spending any time on it at all.
landemva 4y ago
This does not need Google deep pockets. It needs the motivation and some funding. SWA does not care. Pull the backup tapes, hand those to DR team, provide bare metal, and start the stopwatch. I participated in this in 1990s across the Mississippi.
barrenko 4y ago
Personally one of the basic tenets of my adulthood is realizing how many companies are a hair away from a similar scenario (differing in magnitude from an airline ofc).
- bookofjoe 4y ago
  EMP FTW
aspectmin 4y ago
That... and Chaos Engineering in general, and also just generally a forward looking team that identifies threats, vulnerabilities, and risks in the future and works backwards to identify potential mitigating controls. (Look forward, reason backward).

by vrc 4y ago

Google runs Disaster Recovery Training annually (DiRT) where security teams are tasked with simulating these “black swan” events. Seems like this practice needs to expand to more industries.

bobkazamakis 4y ago

>Seems like this practice needs to expand to more industries. I think you've mistaken this for something immediately increases quarterly gains with no regard to long-term strategy.

Analemma_ 4y ago
In that case the government should smash Southwest with a billion dollar fine so the cost of not doing this drilling exceeds the cost of doing it.
- hallway_monitor 4y ago
  Asking the government to step in for additional regulation is rarely helpful. For this type of failure, the free market will determine whether processes and tools improve, or whether the status quo is good enough.
  - heavenlyblue 4y ago
    Free market?
  - ceejayoz 4y ago
    Regulation sometimes helps remind the free market that fuckups like this can come with real human costs.
    - variant 4y ago
      Regulation imposes those costs and more on real humans and rarely heads off fuckups. Market forces can correct here. Vote with your wallet.
  - astrange 4y ago
    What's an airline got to do with the free market? They're a extremely highly regulated business.
23B1 4y ago
What's most annoying is that there's plenty of employees on the front line who not only care about testing for this sort of thing, but it actually interests them, they're motivated by it, and they understand the dire reality of what happens – to them, primarily – if the company isn't prepared to handle it. And you can guess what their managers' response typically is: "We need to focus on OKRs and QBRs and KPIs right now... maybe next quarter" I'm fully convinced that achieving 'manager status' is directly correlated to cowardice. Companies need top-down decision-making, but those decision-makers need to spend more time on the front line.
- giantrobot 4y ago
  > Companies need top-down decision-making, but those decision-makers need to spend more time on the front line. This is not rewarded so it doesn't happen. Managers are rewarded for line goes up so they only focus on line goes up. If line ever doesn't go up it costs them money (advancement, compensation) even if there's little they could have done to make line go up.
dzhiurgis 4y ago
Until regulation steps in.
kevinmchugh 4y ago
My wife and I spent a few hours this morning dealing with the cancellation of our return flight. Southwest has long been preferable for me in many cases, including my most flown route. Between the headache of this outage and the apparently dismal state of their operations there's a strong chance I never fly with them again.
- jdeibele 4y ago
  Our family was in San Jose last week when our Southwest flight was cancelled. It used to be that my first priority would be to go into the terminal and try to talk to somebody. I figured they were the experts. From what I've read, the staff use an antiquated system that takes you from one airport to another, then they can try to get you from that city to where you want to go. That's why there's so much tapping of keys and why it takes so long. It's better to present them with a route that you've found on Google Flights or similar. The Southwest first flight out was supposed to be yesterday evening, the day after Christmas. In our case, the only thing we could find before Christmas was getting us from SJC to Seattle via Phoenix on Alaska. We ended up renting a car and driving home to Portland. Things got bad around Eugene - I stopped counting after 40 wrecked cars and semis - and got worse as you got closer to Portland.
astrange 4y ago
Airlines don't have quarterly gains. They regularly go bankrupt and get picked up again, because the country needs airlines and because they have large union contracts. See any gains here? https://www.google.com/finance/quote/LUV:NYSE?window=5Y https://www.google.com/finance/quote/LUV:NYSE?window=5Y

rumdonut 4y ago

I figure most companies are too small for that to be budgeted. Though, it’s possibly a good selling point for cloud if it’s capable of it.

JustLurking2022 4y ago
Cloud doesn't solve badly designed processes or poorly written software, which seem to be at play with Southwest. Yes, it can help provide more stable infrastructure and there are some (but by no means all) black swan events that can be mitigated simply by throwing more kit at the problem during a surge but it's no silver bullet.

splistud 4y ago

[dead]

tomjakubowski 4y ago

Training for these scenarios may help with responding to true black swan events, like Rick Rescorla's WTC evacuation drills ahead of 9/11. But, nitpicking, if you've predicted something will happen, by way of simulating it, it's not a black swan when it does.

adrr 4y ago
There are cost considerations. Business continuity costs money. Finance firms have significant capital and income to have empty but built out building around around airports for business continuity. Which doesn’t even make sense since they can work from home as proven with covid. Airlines can’t work from home.

paganel 4y ago

Google is also a trillion dollar company, as other have pointed out Soutwest is a low-cost carrier which most probably doesn't have the luxury of hiring FAANG-level engineers on 500k yearly comp in order to best simulate "black swan" events.

deleted 4y ago
[deleted]
makestuff 4y ago
This is probably the best argument for AWS/GCP/Azure even though it is becoming more and more obvious you don't really save that much money. If you have a black swan event like this and you listened to your solutions architect you will have a disaster recovery plan or even better a multi region setup. Worst case you have highly paid support engineers at the cloud providers who will do everything they can to get you back online.
- 0cf8612b2e1e 4y ago
  This does not seem like a hardware failure scenario where the cloud has anything to offer. More like their intricate software/database systems became out of sync with reality and disentangling the mess is a highly manual process.
  - makestuff 4y ago
    In this case no, but I was more referring to the 2016 delta ground stop that was due to their datacenter burning to the ground.
    - therealjumbo 4y ago
      I remember reading about the Delta incident a ways back, here they claim it cost them ~$150 million. https://www.datacenterknowledge.com/archives/2016/09/08/delta-data-center-outage-cost-us-150m https://www.datacenterknowledge.com/archives/2016/09/08/delt... That's not the article I hoped to find however. I seem to remember there was another article where they hired a investigator/consultant to figure out the price to migrate to the cloud and ensure "this never happens again." My recollection of that was: their scheduling/ops team is also in the same city (Atlanta GA) as this datacenter, and that teams work was brought to a halt by the datacenter outage. The investigator concluded that Delta would need redundant copies of the ops team or the whole effort of moving the software to the cloud would just be at risk to something happening to the human team all in the same city. That would obviously cost to much money, so Delta decided to skip it.

thriftwy 4y ago

Yandex used to run datacenter loss training every week, where they will nullroute one DC and see what breaks all while handling live taffic.

adamsb6 4y ago

At Facebook we would simulate an entire datacenter disappearing. When we first started doing it the datacenter would be chosen months in advance so that teams would have plenty of time to ensure their services can run without that specific datacenter. When I left this year, the datacenter would be randomly chosen on the same day it would be cut off.

chasd00 4y ago
That's pretty cool and ideal practice for a software firm but in one of the reddit threads they're talking about mass quits/refusal to work of ground crew at Denver because of the weather. I wonder how you could ever prepare for that? Keep a backup, airport scoped, ground crew in the waiting room?? You can't really do hot spares for people without time to gear/train up and the weather event is so widespread I doubt there's enough spare SWA human capacity across the whole nation even if you had C130s on standby everywhere ready to take workers where they're needed most. From a national security perspective, situations like this is why the Marines exist right? Ensure a rapid response while the rest of the machine gets moving. I feel bad for everyone involved, those affected and those trying to figure out a solution.
- landemva 4y ago
  > I wonder how you could ever prepare for that? Management could consider how pay and performance programs can help ensure business continuity. HR and MBA xls wizards don't understand how to manage for business longevity.
  - dpkirchner 4y ago
    Pay might not really be enough. Maybe management could try to find folks to babysit kids/take care of parents trapped at home, freeing up workers to come fly. They'll certainly fail, but at least they'll understand the plight of their workers.
    - bsg75 4y ago

Scarblac 4y ago

Google is made of money, and the reason they are is not because of DiRT. Other industries can't afford the same things that Google can while continuing to be a business.

fragmede 4y ago
After canceling 5,400 flights, I can't see how Southwest can afford not to test. Even if they only made $1000 off each flight, that's still $5 million they just lost.
- Scarblac 4y ago
  They probably would not have made it this far if they tested for every possible scenario, their margins are razor thin. Anyway, doesn't even the best testing only catch 40% of bugs or thereabouts? It's not a silver bullet.
  - fragmede 4y ago
    It's not any bullet, but spending a week each year to do downtime testing is going to expose more issues than not spending any time on it at all.

landemva 4y ago

This does not need Google deep pockets. It needs the motivation and some funding. SWA does not care. Pull the backup tapes, hand those to DR team, provide bare metal, and start the stopwatch. I participated in this in 1990s across the Mississippi.

barrenko 4y ago

Personally one of the basic tenets of my adulthood is realizing how many companies are a hair away from a similar scenario (differing in magnitude from an airline ofc).

bookofjoe 4y ago
EMP FTW

aspectmin 4y ago

That... and Chaos Engineering in general, and also just generally a forward looking team that identifies threats, vulnerabilities, and risks in the future and works backwards to identify potential mitigating controls. (Look forward, reason backward).