4 ms·

How is it even possible for all aspects of such a massive enterprise to all share a single point of failure like that? And why can't they just cut their losses

How is it even possible for all aspects of such a massive enterprise to all share a single point of failure like that? And why can't they just cut their losses on the past N days of business, restore all these servers from snapshots and get back to business?

SteveNuts 3y ago
Often the attackers wait until the retention policy for backups has been hit before unleashing the payload of ransomware. This way, even if you have a snapshot from 7 days ago, it's also infected. Or even worse they have physical access to the backup server/storage and just delete backups infect them as well.
jachee 3y ago
“The DR planning meeting has been once again postponed until next quarter…” Some middle-manager somewhere, probably.
specialp 3y ago
There are multiple things that are done here. Suppose you had great, immutable backups. They still have many things that can ruin your business 1. Restoring networks, servers, third party services with knowledge that anything you restore could be compromised as well. Keys 2. The attackers will then threaten to dump all of your private information. It is more than just restoring data, it is restoring and resetting your entire infrastructure. And most places have backups, but they don't practice entire restores
- baby_souffle 3y ago
  > And most places have backups, but they don't practice entire restores Or worse, they only practice part of it. Only once in my career have I seen a "restore.txt" that didn't start with something along the lines of "connect to $server". Ok, that assumes a LOT is already in place. Where is the "restore.txt" that goes over how to get $network up so that I can resolve the IP(s) for the server I need to restore? I can't prove it, but I suspect that most businesses know deep down that they _cant_ do a "black start" and they know that even a practice run is likely to find some pretty basic and embarrassing issues that will just be too costly to address.
  - catchnear4321 3y ago
    to go one step further, then there are all the companies that have done a proper recovery plan, and even tested it. several years ago. hopefully there were no externally managed dependencies, as those can change. hopefully the documentation was entirely written down, not just 80%, with that last 20% having retired, been laid off, or died since then. how many companies are left at that point?
not_real_acct 3y ago
Maybe they shouldn't have their corporate login page on the Internet. https://mgmresorts.okta.com/app/mgmresortsprod_neocaseemployeeportal_1/exk1dtuzxkwZGJ5301d8/sso/saml https://mgmresorts.okta.com/app/mgmresortsprod_neocaseemploy...

SteveNuts 3y ago

Often the attackers wait until the retention policy for backups has been hit before unleashing the payload of ransomware. This way, even if you have a snapshot from 7 days ago, it's also infected. Or even worse they have physical access to the backup server/storage and just delete backups infect them as well.

jachee 3y ago

“The DR planning meeting has been once again postponed until next quarter…” Some middle-manager somewhere, probably.

specialp 3y ago

There are multiple things that are done here. Suppose you had great, immutable backups. They still have many things that can ruin your business 1. Restoring networks, servers, third party services with knowledge that anything you restore could be compromised as well. Keys 2. The attackers will then threaten to dump all of your private information. It is more than just restoring data, it is restoring and resetting your entire infrastructure. And most places have backups, but they don't practice entire restores

baby_souffle 3y ago
> And most places have backups, but they don't practice entire restores Or worse, they only practice part of it. Only once in my career have I seen a "restore.txt" that didn't start with something along the lines of "connect to $server". Ok, that assumes a LOT is already in place. Where is the "restore.txt" that goes over how to get $network up so that I can resolve the IP(s) for the server I need to restore? I can't prove it, but I suspect that most businesses know deep down that they _cant_ do a "black start" and they know that even a practice run is likely to find some pretty basic and embarrassing issues that will just be too costly to address.
- catchnear4321 3y ago
  to go one step further, then there are all the companies that have done a proper recovery plan, and even tested it. several years ago. hopefully there were no externally managed dependencies, as those can change. hopefully the documentation was entirely written down, not just 80%, with that last 20% having retired, been laid off, or died since then. how many companies are left at that point?

not_real_acct 3y ago

Maybe they shouldn't have their corporate login page on the Internet. https://mgmresorts.okta.com/app/mgmresortsprod_neocaseemployeeportal_1/exk1dtuzxkwZGJ5301d8/sso/saml https://mgmresorts.okta.com/app/mgmresortsprod_neocaseemploy...