8 ms·

What's the simplest way to prevent this from happening these days? Daily Backups off site?

jeffmcjunkin 3y ago
That helps with "we've encrypted your data; pay us for the key" but doesn't help you with "we've made copies of your patient records, leadership's emails; pay us or we publish it all". The phrase to describe this is double extortion. As for your question, https://www.cisa.gov/stopransomware https://www.cisa.gov/stopransomware is a decent start, but it's a complicated issue. In short, if a pentester can get inside your environment and gain privileges, so can an attacker. You want to slow down attackers enough to buy time for detection and response capabilities.
- endisneigh 3y ago
  Hm - is the expectation that this stuff isn't encrypted at rest?
  - adolph 3y ago
    Since the user's eyeballs don't have builtin decryption there is a window of opportunity to steal information after encrypted at rest and encrypted transport. Hopefully vendors will be able to fix this defect by using Neuralink.
    - Aeolun 3y ago
      If it’s at rest encrypted you generally do not manage to get 4TB of data before anyone notices.
      - blincoln 3y ago
        I think you'd be surprised. How many organizations have their "encrypted at rest" data in a cloud provider account that's set up to give all developers (or at least all production support engineers) access to decrypt the data, maybe even transparently? How many have the "encrypted at rest" data on servers that are set up to give all administrators transparent access to the data? How many only allow application service accounts access to decrypt the data directly, but the credentials for those service accounts are stored as Kubernetes secrets that anyone in IT can read? Etc.
        nradov 3y ago
        I can guarantee you that UnitedHealth Group (Change Healthcare) doesn't give regular developers the credentials to decrypt production data, or access production environments at all.
        adolph 3y ago
        probably not "developers," probably "data scientists" Executives at UnitedHealth Group told workers to mine old medical records for more illnesses, to identify diagnoses of serious diseases that might have never existed, inflating bills paid by the federal government's Medicare Advantage program. https://en.wikipedia.org/wiki/UnitedHealth_Group https://en.wikipedia.org/wiki/UnitedHealth_Group
      - reaperducer 3y ago
        If it’s at rest encrypted you generally do not manage to get 4TB of data before anyone notices. If you're talking about your own personal hard drive in your home, yes. But in this case, we're talking about a huge company conducting literally billions of database queries for tens of thousands of clients an hour. You only have to have a listening post in one small part of the system that can see things in plaintext for a short time in order to accumulate 4TB in a matter of days.
      - adolph 3y ago
        Looks like the issue has to do with claims processing between pharmacies and UHG, effectively data in flight not at rest. "We estimate more than 90% of the nation’s 70,000+ pharmacies have modified electronic claim processing to mitigate impacts from the Change Healthcare cyber security issue; the remainder have offline processing workarounds," Mason said. https://www.bleepingcomputer.com/news/security/unitedhealth-subsidiary-optum-hack-linked-to-blackcat-ransomware/ https://www.bleepingcomputer.com/news/security/unitedhealth-... The pharmacy network, which connects pharmacies and PBMs, is in final end-to-end testing with our partners. We anticipate that our Change Healthcare Pharmacy network will be back online for the vast majority of submitters as soon as Thursday. https://www.unitedhealthgroup.com/ns/changehealthcare.html https://www.unitedhealthgroup.com/ns/changehealthcare.html
    - MichaelZuo 3y ago
      You can also solve this problem by air-gapping any part of the system with physical human interaction.
  - fbdab103 3y ago
    Presumably they did not break into the data center and lift a bunch of hard drives. Instead they compromised a server which had credentials to read the data in a clear format.
smiley1437 3y ago
In this particular case, it isn't an issue of 'data loss' which could be recovered from good backups. It's a problem of 'data leakage' where the bad guys have a copy of corporate data - supposedly 4TB of personal medical information - and threaten to release it to the public which can cause all sorts of reputational or other damage. Backups don't help much in this situation, you need to convince the attackers to delete the data they copied from your network, usually via lots of money but even then there's no guarantee they'll actually delete it and may extort you for more money in the future with the same data. Should one trust a criminal to keep their word?
- endisneigh 3y ago
  Right but isn't that why stuff is encrypted? Is there even a way to guarantee this doesn't happen?
  - bombcar 3y ago
    No real way to guarantee though you can go to great lengths to reduce the chance it happens. Because it gets decrypted at some point.
  - QuiDortDine 3y ago
    You can't encrypt all your data at all times. It has to be decrypted somewhere along the line to be used, and hackers can gain that access as well unless the security is truly extremely tight. Security is not anywhere close to "good" in most corporate environments however, and many things are still stored in plaintext that should not (e.g. passwords), let alone data that is merely "private".
    - boringuser2 3y ago
      This isn't quite right. You don't need to decrypt the entire payload at any time. You decrypt parts of the data.
    - dt3ft 3y ago
      Encrypting data while at rest (in storage) as well as in-transit is the way to go. The servers on your infrastructure should be considered hostile at all times. Put on a hat and pretend you’re a bad actor. Give yourself access to the server where your most important data is stored. Now look around. Is there anything you can do to extort money? You could encrypt/destroy the data. (A backup solution saves you here). You could exfiltrate the data (download or upload to a remote server). What can you do with this data if it was encrypted at rest? Not much. What else could you do on thisnserver while you have access? This is where things get interesting. Can you force the application to decrypt the data or dump the data somehow? Unlikely, if the cert management is done properly. The thing is, majority of organisations do not encrypt data at rest. Databases are not encrypted, hard data is not encrypted. If this was not the case, we wouldn’t be hearing about these data leaks.
technion 3y ago
The whole problem has historically been that any "simple" solution lacks nuance. I've been across a number of these sorts of environments. Just look at your answer: Daily backups off site. Sure, so I've seen this. There's a Veeam server with a backup repo in the main datacenter. And it takes backups to a server "offsite". Let's forget for a moment I've got an environment that take six days to do a full backup. Then one day the attacker gains a vendor's Teamviewer account and finds themselves on a server console that's been left logged on as a Domain Admin. And they open windows file explorer, and browse to \\offsiteserver\backups, and then press "delete". Noone cares if it's offsite, it's gone. OK, so when you said "offsite backups" I'm sure you meant something not contactable from an average network machine right? Maybe ACLs only allow access from the backup server itself. Well fear not, the attacker can still just RDP to that Veeam server and repeat. OK OK so what you really wanted was a properly isolated network for all the backup content and they can't make a connection from the general network to it right? Once again fear not, there's a Group Policy deploying this ransomware, which means it's going on every domain joined machine including the backup server. Look, now we're getting somewhere, there should be an entirely separate administration domain for backups and infrastructure. Well firstly someone from Microsoft will yell at you because "ESAE" is deprecated, and some overpriced consultant is about to explain to management that you're incompetent because you separated the networks (from personal experience). Fortunately that doesn't matter to the one guy with a popped Domain Admin account on the general domain used the same password on the administrative domain against policy, and the attacker spreads anyway. Yes you've got options here. For example someone might mention "Veeam Immutable Storage", which is pretty effective. But now you'll find the iLO for that server still presents a forgotten entrypoint to wiping it. There's absolutely ways to do this properly but it's never simple, and the further you go down the hole the more likely you are to hit pricing or political stumbling blocks.
- namaria 3y ago
  Opsec is expensive and requires discipline. The average for profit organization can at best perform box checking security theater. Any real hindrance to their business goal is gonna get diluted or sidelined into irrelevance. Security is not a technical problem, it's a sociopolitical problem. That's my main gripe with the business of computer security (even the name cybersecurity rubs me the wrong way... cybernetics is about robotics control systems). All these hard selling seemingly highly advanced stuff, all these script kiddies showing companies how much vulnerability is peppered all over their systems. And in the end you can call up people and they will cheerily give away credentials against vague verbal assurances.
YouWhy 3y ago
The scenario at hand is data leakage; accordingly the prevention strategy is by having a more secure storage system. One way of addressing that is by using a secure vault for sensitive data storage. This is beyond the skillset of most backend engineers, but new products such as Piiano may bridge the gap (proper disclosure: I'm a friend of one of the people behind Piiano)
interestica 3y ago
The city of Hamilton, Ontario is going through this right now in real-time. I think they only admitted today that it was a ransomware attack. I'm very curious to hear if they do any kind of public postmortem as other mid level cities in the country as just as vulnerable.
mey 3y ago
Layers and looking at what the business needs to achieve for business continuity and recovery. How quickly do you need to restore services after a major incident (natural disaster to internal user with God access)? What is an acceptable amount of data loss at any point in time? What are your legal obligations? Then finally, compare the financial/social/legal impact of those events against what the company will allow against what it can afford. This same conversation should tie directly into your Service Level agreements/uptime goals. Simple? Nah, doing this properly is an entire department/role (CISO) plus other groups. Some quick practical basics. - The fewer things, the smaller the potential attack surface. - NIST publishes some good starting points https://www.nist.gov/cyberframework - Shared credentials are evil. All admins should be operating off their own accounts - MFA and SSO - A backup isn't a backup if you haven't validated you can restore it - Replication isn't backup. You need something that is immutable and has history. To start I would suggest a nightly backup, with each backup retained for 45 days at minimum. GDPR complicates this with right to be forgotten but this is a starting point.
m-p-3 3y ago
It's a small part of the whole Disaster Recovery Plan. Having a backup isn't worth a lot if you don't practice the recovery process and can't bring it back in a timely manner. For the data exfiltration issue, there's not much you can do other than paying and hoping for the best, and it depends on their reputation of keeping their word. You won't get paid much and last long in the ransomware world if you have a reputation of leaking the data when paid.

That helps with "we've encrypted your data; pay us for the key" but doesn't help you with "we've made copies of your patient records, leadership's emails; pay us or we publish it all". The phrase to describe this is double extortion. As for your question, https://www.cisa.gov/stopransomware https://www.cisa.gov/stopransomware is a decent start, but it's a complicated issue. In short, if a pentester can get inside your environment and gain privileges, so can an attacker. You want to slow down attackers enough to buy time for detection and response capabilities.

endisneigh 3y ago
Hm - is the expectation that this stuff isn't encrypted at rest?
- adolph 3y ago
  Since the user's eyeballs don't have builtin decryption there is a window of opportunity to steal information after encrypted at rest and encrypted transport. Hopefully vendors will be able to fix this defect by using Neuralink.
  - Aeolun 3y ago
    If it’s at rest encrypted you generally do not manage to get 4TB of data before anyone notices.
    - blincoln 3y ago
      I think you'd be surprised. How many organizations have their "encrypted at rest" data in a cloud provider account that's set up to give all developers (or at least all production support engineers) access to decrypt the data, maybe even transparently? How many have the "encrypted at rest" data on servers that are set up to give all administrators transparent access to the data? How many only allow application service accounts access to decrypt the data directly, but the credentials for those service accounts are stored as Kubernetes secrets that anyone in IT can read? Etc.
      - nradov 3y ago
        I can guarantee you that UnitedHealth Group (Change Healthcare) doesn't give regular developers the credentials to decrypt production data, or access production environments at all.
        adolph 3y ago
        probably not "developers," probably "data scientists" Executives at UnitedHealth Group told workers to mine old medical records for more illnesses, to identify diagnoses of serious diseases that might have never existed, inflating bills paid by the federal government's Medicare Advantage program. https://en.wikipedia.org/wiki/UnitedHealth_Group https://en.wikipedia.org/wiki/UnitedHealth_Group
    - reaperducer 3y ago
      If it’s at rest encrypted you generally do not manage to get 4TB of data before anyone notices. If you're talking about your own personal hard drive in your home, yes. But in this case, we're talking about a huge company conducting literally billions of database queries for tens of thousands of clients an hour. You only have to have a listening post in one small part of the system that can see things in plaintext for a short time in order to accumulate 4TB in a matter of days.
    - adolph 3y ago
      Looks like the issue has to do with claims processing between pharmacies and UHG, effectively data in flight not at rest. "We estimate more than 90% of the nation’s 70,000+ pharmacies have modified electronic claim processing to mitigate impacts from the Change Healthcare cyber security issue; the remainder have offline processing workarounds," Mason said. https://www.bleepingcomputer.com/news/security/unitedhealth-subsidiary-optum-hack-linked-to-blackcat-ransomware/ https://www.bleepingcomputer.com/news/security/unitedhealth-... The pharmacy network, which connects pharmacies and PBMs, is in final end-to-end testing with our partners. We anticipate that our Change Healthcare Pharmacy network will be back online for the vast majority of submitters as soon as Thursday. https://www.unitedhealthgroup.com/ns/changehealthcare.html https://www.unitedhealthgroup.com/ns/changehealthcare.html
  - MichaelZuo 3y ago
    You can also solve this problem by air-gapping any part of the system with physical human interaction.
- fbdab103 3y ago
  Presumably they did not break into the data center and lift a bunch of hard drives. Instead they compromised a server which had credentials to read the data in a clear format.

smiley1437 3y ago

In this particular case, it isn't an issue of 'data loss' which could be recovered from good backups. It's a problem of 'data leakage' where the bad guys have a copy of corporate data - supposedly 4TB of personal medical information - and threaten to release it to the public which can cause all sorts of reputational or other damage. Backups don't help much in this situation, you need to convince the attackers to delete the data they copied from your network, usually via lots of money but even then there's no guarantee they'll actually delete it and may extort you for more money in the future with the same data. Should one trust a criminal to keep their word?

endisneigh 3y ago
Right but isn't that why stuff is encrypted? Is there even a way to guarantee this doesn't happen?
- bombcar 3y ago
  No real way to guarantee though you can go to great lengths to reduce the chance it happens. Because it gets decrypted at some point.
- QuiDortDine 3y ago
  You can't encrypt all your data at all times. It has to be decrypted somewhere along the line to be used, and hackers can gain that access as well unless the security is truly extremely tight. Security is not anywhere close to "good" in most corporate environments however, and many things are still stored in plaintext that should not (e.g. passwords), let alone data that is merely "private".
  - boringuser2 3y ago
    This isn't quite right. You don't need to decrypt the entire payload at any time. You decrypt parts of the data.
  - dt3ft 3y ago
    Encrypting data while at rest (in storage) as well as in-transit is the way to go. The servers on your infrastructure should be considered hostile at all times. Put on a hat and pretend you’re a bad actor. Give yourself access to the server where your most important data is stored. Now look around. Is there anything you can do to extort money? You could encrypt/destroy the data. (A backup solution saves you here). You could exfiltrate the data (download or upload to a remote server). What can you do with this data if it was encrypted at rest? Not much. What else could you do on thisnserver while you have access? This is where things get interesting. Can you force the application to decrypt the data or dump the data somehow? Unlikely, if the cert management is done properly. The thing is, majority of organisations do not encrypt data at rest. Databases are not encrypted, hard data is not encrypted. If this was not the case, we wouldn’t be hearing about these data leaks.

technion 3y ago

The whole problem has historically been that any "simple" solution lacks nuance. I've been across a number of these sorts of environments. Just look at your answer: Daily backups off site. Sure, so I've seen this. There's a Veeam server with a backup repo in the main datacenter. And it takes backups to a server "offsite". Let's forget for a moment I've got an environment that take six days to do a full backup. Then one day the attacker gains a vendor's Teamviewer account and finds themselves on a server console that's been left logged on as a Domain Admin. And they open windows file explorer, and browse to \\offsiteserver\backups, and then press "delete". Noone cares if it's offsite, it's gone. OK, so when you said "offsite backups" I'm sure you meant something not contactable from an average network machine right? Maybe ACLs only allow access from the backup server itself. Well fear not, the attacker can still just RDP to that Veeam server and repeat. OK OK so what you really wanted was a properly isolated network for all the backup content and they can't make a connection from the general network to it right? Once again fear not, there's a Group Policy deploying this ransomware, which means it's going on every domain joined machine including the backup server. Look, now we're getting somewhere, there should be an entirely separate administration domain for backups and infrastructure. Well firstly someone from Microsoft will yell at you because "ESAE" is deprecated, and some overpriced consultant is about to explain to management that you're incompetent because you separated the networks (from personal experience). Fortunately that doesn't matter to the one guy with a popped Domain Admin account on the general domain used the same password on the administrative domain against policy, and the attacker spreads anyway. Yes you've got options here. For example someone might mention "Veeam Immutable Storage", which is pretty effective. But now you'll find the iLO for that server still presents a forgotten entrypoint to wiping it. There's absolutely ways to do this properly but it's never simple, and the further you go down the hole the more likely you are to hit pricing or political stumbling blocks.

namaria 3y ago
Opsec is expensive and requires discipline. The average for profit organization can at best perform box checking security theater. Any real hindrance to their business goal is gonna get diluted or sidelined into irrelevance. Security is not a technical problem, it's a sociopolitical problem. That's my main gripe with the business of computer security (even the name cybersecurity rubs me the wrong way... cybernetics is about robotics control systems). All these hard selling seemingly highly advanced stuff, all these script kiddies showing companies how much vulnerability is peppered all over their systems. And in the end you can call up people and they will cheerily give away credentials against vague verbal assurances.

YouWhy 3y ago

The scenario at hand is data leakage; accordingly the prevention strategy is by having a more secure storage system. One way of addressing that is by using a secure vault for sensitive data storage. This is beyond the skillset of most backend engineers, but new products such as Piiano may bridge the gap (proper disclosure: I'm a friend of one of the people behind Piiano)

interestica 3y ago

The city of Hamilton, Ontario is going through this right now in real-time. I think they only admitted today that it was a ransomware attack. I'm very curious to hear if they do any kind of public postmortem as other mid level cities in the country as just as vulnerable.

mey 3y ago

Layers and looking at what the business needs to achieve for business continuity and recovery. How quickly do you need to restore services after a major incident (natural disaster to internal user with God access)? What is an acceptable amount of data loss at any point in time? What are your legal obligations? Then finally, compare the financial/social/legal impact of those events against what the company will allow against what it can afford. This same conversation should tie directly into your Service Level agreements/uptime goals. Simple? Nah, doing this properly is an entire department/role (CISO) plus other groups. Some quick practical basics. - The fewer things, the smaller the potential attack surface. - NIST publishes some good starting points https://www.nist.gov/cyberframework - Shared credentials are evil. All admins should be operating off their own accounts - MFA and SSO - A backup isn't a backup if you haven't validated you can restore it - Replication isn't backup. You need something that is immutable and has history. To start I would suggest a nightly backup, with each backup retained for 45 days at minimum. GDPR complicates this with right to be forgotten but this is a starting point.

m-p-3 3y ago

It's a small part of the whole Disaster Recovery Plan. Having a backup isn't worth a lot if you don't practice the recovery process and can't bring it back in a timely manner. For the data exfiltration issue, there's not much you can do other than paying and hoping for the best, and it depends on their reputation of keeping their word. You won't get paid much and last long in the ransomware world if you have a reputation of leaking the data when paid.