6 ms·

CISO: https://cio.ucop.edu/spotlight-patrick-phelan-once-a-ucla-bruin-now-ciso-at-ucsf/ https://cio.ucop.edu/spotlight-patrick-phelan-once-a-ucla-br... I can't

CISO: https://cio.ucop.edu/spotlight-patrick-phelan-once-a-ucla-bruin-now-ciso-at-ucsf/ https://cio.ucop.edu/spotlight-patrick-phelan-once-a-ucla-br... I can't think of any reason not to use a cloud hosted service for backup today. OneDrive, Dropbox, and Google Drive all sign BAAs and give you versioning amongst a million other security features. AWS even has offerings that let you take periodic snapshots of on-premise volumes. Point in time recoveries for the entire account would be nice add too but not having to fork over a million dollars in exchange for a few clicks sounds like a bargain. Hell we were setting up write-only S3 buckets for critical data stores 5+ years ago. Gross incompetence, I'd fire everyone. A district-wide outage for a week, fine. A 1.4 million dollar check to get data that should have been archived somewhere GTFO.

AnthonyMouse 6y ago
There are reasons not to use cloud services for backups, not trusting them with your data being a major one. But then you should still be using some kind of internal backup system. There is no excuse for not having backups.
- jtokoph 6y ago
  If you don’t trust them with your data, you can encrypt it with your own keys.
  - briffle 6y ago
    Really, someone just did that for them, for the bargain price of $1.4M. I'm guessing they saved at least that much by outsourcing their entire IT department a few years ago.
    - znpy 6y ago
      This is really the best possible answer. you just did the root cause analysis for them too (you should send them an invoice). Nothing new, in a way: cut the spending on IT, lower quality, get incidents. Meh.
  - AnthonyMouse 6y ago
    But then you lose the main advantages of using them to begin with. It becomes harder to use, it may not be able to do efficient differential backups or snapshots anymore or require you to do some complicated thing to make it work because their interface isn't meant to be used that way etc. And if it's actually important for the data to remain private, you then have to get the cryptography right, not think that your backups are "encrypted" because you used TLS, or use some snake oil encryption software which is using export ciphers or bad random number generation or anything like that. In-house backups aren't that hard. The hardest thing is to make sure everything is getting backed up that ought to be, and actually test that it is, which is no different with cloud backups.
    - throw0101a 6y ago
      > But then you lose the main advantages of using them to begin with. Poppycock. There's plenty of software that can do encrypted incrementals / differentials: Commvault, NetBackup, Veeam, tarsnap, Duplicity, ZFS snapshot send-recv,
      - AnthonyMouse 6y ago
        You're asserting that these things are as easy to use as keeping all your junk on a shared Google Drive or whatever it is many companies are currently doing? They all require you to have a level of technical competence equivalent to what would be needed to implement the backups yourself.
        throw0101a 6y ago
        If you're talking about something that is worth paying US$ 1M in ransom, then paying someone a few (tens of) thousands in cost centre chargebacks upfront, and probably an ongoing annual fee, is cheap insurance if you want to go with the 'enterprise software'. And if you want 'consumer software' that offers cloud / offsite encrypted back, then Backblaze offers it in a very clicky-clicky fashion that most folks can handle with a bit of hand-holding: * https://help.backblaze.com/hc/en-us/articles/217664688-Can-you-tell-me-more-about-the-encryption-Backblaze-uses- https://help.backblaze.com/hc/en-us/articles/217664688-Can-y... As does Arq, with a documented file format: * https://www.arqbackup.com/arq_data_format.txt https://www.arqbackup.com/arq_data_format.txt Every cloud-capable backup system (for consumers and enterprises) does encryption nowadays, so talking about 'data access' in the cloud is a straw man.
- nindalf 6y ago
  “No, we can’t use leading cloud providers to store this data. What if someone unauthorised looks at it?” This attitude leaves the victim paying millions in ransom. Look, Microsoft and Amazon probably have a better handle on security than your IT dept which is two people who also have to fix any issues like the WiFi not working or software not updating.
  - bpfrh 6y ago
    There are stories of MSPs who got their offline backup from the same provider who they bought their cloud based RMM tool. One day an account get's exploited and suddenly the backups are deleted and the systems are all encrypted remotely. Cloud backups are not offline backups, if you want security use tapes or remote systems which turn on manually.
    - nindalf 6y ago
      A backup that can be overwritten is not a backup. It is possible to use a cloud provider and also write an immutable backup.
  - AnthonyMouse 6y ago
    > This attitude leaves the victim paying millions in ransom. Not if they still have backups that aren't cloud backups. > Look, Microsoft and Amazon probably have a better handle on security than your IT dept which is two people who also have to fix any issues like the WiFi not working or software not updating. And that's part of the problem, right? Your IT department isn't too bright, so they encrypt their cloud backups with a tool that does the encryption wrong, and the smarter engineers at the cloud provider know that it's vulnerable so now they can read your data when you thought they couldn't. Or you use a weak password for the cloud backup account and then some third party breaks in and gets them. Not really a concern with a backup tape stored in a fire safe.
  - vinay_ys 6y ago
    There is no reason to believe clouds don't have security issues or software bugs that cause inadvertent data loss. Just because the cloud provider is a big company doesn't mean the security is better. At the end of the day, they are a team of software engineers with all the usual people and org problems and can have usual human mistakes. But cloud solutions are better in practice due to totally non-technical reasons. Here's how - If UCSF can afford to pay millions in ransom, they can definitely afford to hire the right experts to do their IT systems properly. It is likely that they hire the right people, but then the most commonly recommended security policies were considered but not implemented due to resistence from powerful non-security team members inside the organization. (Usually, the highly paid CISO is playing politics with peers to keep the job by ignoring those security experts). If it also possible that the IT+security team is incompetent and is only good for deploying expensive but useless vendor solutions that do a lot of security theater but do very little to improve actual security. Also possible that the vendor solutions are susceptible to be misconfigured easily to become insecure. Public cloud solutions can help overcome these above shortcomings of in-house security+IT teams in a org politics friendly manner.
- throw0101a 6y ago
  > […] not trusting them with your data being a major one. Which is why you encrypt the data before sending it over the wire. Just like how you'd use encryption on LTO tapes before sending them offsite.
throw0101a 6y ago
> I can't think of any reason not to use a cloud hosted service for backup today. Or just use a non-Windows file server that supports snapshots: NetApp, Isilon, FreeNAS, etc. If an end-user devices gets infected, and then encrypts mapped drives under the credentials of the user in question, that won't do anything to the read-only data. Snapshots are not backups, but for a lot of the most common situations, they offer a convenient, quick win so people can move on with their day (often in a self-service fashion by just going into a .snapshot/ directory).
JeremyNT 6y ago
These sound like servers used by researchers. I've worked with higher education research computing and you might be surprised at what you would find. Researchers may be generating or churning through countless TB of intermediary data, scratch files, etc. Often, the people who actually run the it infrastructure for researchers are... grad students. Sometimes they have grants for hardware and tight budgets, and paying anything for backups isn't part of it. Sometimes, if you're lucky, the it department will be aware of the work and allowed to help. Now maybe that's not the case here, and there really is one department responsible, and that department decided against spending money on backups. Well maybe they were told by the provost or the dean of whoever that they couldn't afford to back up everything, so they should just stick to file servers. Maybe the boxes compromised here are compute only, and all code and valuable artifacts are expected to be stored safely somewhere else. And maybe the researchers heard this and understood it when they agreed to use the system. But maybe the new grad student didn't get the memo and developed his model in vim on the compute node. The point is, academic computing is kind of the wild west. Weird fiefdoms and weird restrictions, budgetary and otherwise. It's tough to guess which of these scenarios played out from the outside and we really can't know whether the CISO or CTO, or even anybody working for them, dropped the ball.
- beh9540 6y ago
  I'm so glad I'm not the only one who has experienced this. Only thing I'd add is the lack of understanding about how hard backups are at some of the scales of data in research. Trying to explain to researchers who run the departments that doing a full backup on 35TB+ of data on all budget drives is going to take weeks was something I could never get across, until they lost all their data. And no, snapshots weren't an option on a legacy clustered filesystem that they wouldn't migrate from.
  - briffle 6y ago
    But that is only like 3x 12TB USB drives.. </sarcasm>
- justinclift 6y ago
  > But maybe the new grad student didn't get the memo and developed his model in vim on the compute node. Was trying to think through scenario's like that as well, but they don't really make sense. If various staff members created a few weeks (or even months) worth of work on storage that isn't backed up, then the response from mgmt would generally be "Bad luck, you'll need to redo it". But mgmt decided that work was worth paying US$1.1M+ for, and copping the reputation damage. So, it's a weird mix of potential scenarios to actually get that happening. ;) Maybe someone internal was actually responsible for that malware and is going for some kind of weird payday?
ss2003 6y ago
"I'd fire everyone." Too late. They did that 3 years ago. Fired the IT staff and outsourced to India. A great decision that saved them tons of money. /s https://sanfrancisco.cbslocal.com/2017/02/28/ucsf-tech-workers-have-last-day-on-job-after-outsourcing/ https://sanfrancisco.cbslocal.com/2017/02/28/ucsf-tech-worke... Fire the leadership from the top down. Every one approved outsourcing the IT staff.
- carapace 6y ago
  From that link: > This move will save UCSF $30 million over the next 5 years. So they're ahead? I'm being slightly snarky here, but also earnest. If they save 30 but lose 1.14 isn't it worth it?
  - to11mtm 6y ago
    Depends what the total cost was for this... The 1.14M is just the ransom, doesn't account for how much money was wasted on other aspects of recovery and total downtime...
    - justinclift 6y ago
      Substantial reputational damage will have to figure in there somewhere as well. ;)
- techie128 6y ago
  > ...Fired the IT staff and outsourced to India... Are you asserting that the breach would not have happened if the IT operations were in-house? Could you also clarify why did you feel it necessary to qualify the location of outsourcing? If the project was outsourced to any other country besides India, this would not have happened?
  - wizzwizz4 6y ago
    > If the project was outsourced to any other country besides India, this would not have happened? I think the emphasis is on the fact it was outsourced to another country; you can't generally check the credentials of someone on the other side of the world very easily. The University of Delhi probably has good IT staff, but “Delhi International Computer Help”? Probably not.
hashouseincali 6y ago
Cloud solutions are great unless youre storing huge amounts of data thats being served, modified, and/or backed up. Go with JBOD's, ZFS+snapshots+rsync (or gluster even), and lots of firewalls with MFA and youll be fine (im averaging 60 to 120 TB's in my boxes raid5-7). Google drive is fine for docs. Data should stay home and be backed up elsewhere in the 'house' (still home ... just a separate basement data center or something that wont also burn down). Lets not forget this could have been an inside job that will get around a lot of security other than another humans/montiors vigilance. UCSF outsourcing all their talent is exactly what enabled this.

AnthonyMouse 6y ago

There are reasons not to use cloud services for backups, not trusting them with your data being a major one. But then you should still be using some kind of internal backup system. There is no excuse for not having backups.

jtokoph 6y ago
If you don’t trust them with your data, you can encrypt it with your own keys.
- briffle 6y ago
  Really, someone just did that for them, for the bargain price of $1.4M. I'm guessing they saved at least that much by outsourcing their entire IT department a few years ago.
  - znpy 6y ago
    This is really the best possible answer. you just did the root cause analysis for them too (you should send them an invoice). Nothing new, in a way: cut the spending on IT, lower quality, get incidents. Meh.
- AnthonyMouse 6y ago
  But then you lose the main advantages of using them to begin with. It becomes harder to use, it may not be able to do efficient differential backups or snapshots anymore or require you to do some complicated thing to make it work because their interface isn't meant to be used that way etc. And if it's actually important for the data to remain private, you then have to get the cryptography right, not think that your backups are "encrypted" because you used TLS, or use some snake oil encryption software which is using export ciphers or bad random number generation or anything like that. In-house backups aren't that hard. The hardest thing is to make sure everything is getting backed up that ought to be, and actually test that it is, which is no different with cloud backups.
  - throw0101a 6y ago
    > But then you lose the main advantages of using them to begin with. Poppycock. There's plenty of software that can do encrypted incrementals / differentials: Commvault, NetBackup, Veeam, tarsnap, Duplicity, ZFS snapshot send-recv,
    - AnthonyMouse 6y ago
      You're asserting that these things are as easy to use as keeping all your junk on a shared Google Drive or whatever it is many companies are currently doing? They all require you to have a level of technical competence equivalent to what would be needed to implement the backups yourself.
      - throw0101a 6y ago
        If you're talking about something that is worth paying US$ 1M in ransom, then paying someone a few (tens of) thousands in cost centre chargebacks upfront, and probably an ongoing annual fee, is cheap insurance if you want to go with the 'enterprise software'. And if you want 'consumer software' that offers cloud / offsite encrypted back, then Backblaze offers it in a very clicky-clicky fashion that most folks can handle with a bit of hand-holding: * https://help.backblaze.com/hc/en-us/articles/217664688-Can-you-tell-me-more-about-the-encryption-Backblaze-uses- https://help.backblaze.com/hc/en-us/articles/217664688-Can-y... As does Arq, with a documented file format: * https://www.arqbackup.com/arq_data_format.txt https://www.arqbackup.com/arq_data_format.txt Every cloud-capable backup system (for consumers and enterprises) does encryption nowadays, so talking about 'data access' in the cloud is a straw man.
nindalf 6y ago
“No, we can’t use leading cloud providers to store this data. What if someone unauthorised looks at it?” This attitude leaves the victim paying millions in ransom. Look, Microsoft and Amazon probably have a better handle on security than your IT dept which is two people who also have to fix any issues like the WiFi not working or software not updating.
- bpfrh 6y ago
  There are stories of MSPs who got their offline backup from the same provider who they bought their cloud based RMM tool. One day an account get's exploited and suddenly the backups are deleted and the systems are all encrypted remotely. Cloud backups are not offline backups, if you want security use tapes or remote systems which turn on manually.
  - nindalf 6y ago
    A backup that can be overwritten is not a backup. It is possible to use a cloud provider and also write an immutable backup.
- AnthonyMouse 6y ago
  > This attitude leaves the victim paying millions in ransom. Not if they still have backups that aren't cloud backups. > Look, Microsoft and Amazon probably have a better handle on security than your IT dept which is two people who also have to fix any issues like the WiFi not working or software not updating. And that's part of the problem, right? Your IT department isn't too bright, so they encrypt their cloud backups with a tool that does the encryption wrong, and the smarter engineers at the cloud provider know that it's vulnerable so now they can read your data when you thought they couldn't. Or you use a weak password for the cloud backup account and then some third party breaks in and gets them. Not really a concern with a backup tape stored in a fire safe.
- vinay_ys 6y ago
  There is no reason to believe clouds don't have security issues or software bugs that cause inadvertent data loss. Just because the cloud provider is a big company doesn't mean the security is better. At the end of the day, they are a team of software engineers with all the usual people and org problems and can have usual human mistakes. But cloud solutions are better in practice due to totally non-technical reasons. Here's how - If UCSF can afford to pay millions in ransom, they can definitely afford to hire the right experts to do their IT systems properly. It is likely that they hire the right people, but then the most commonly recommended security policies were considered but not implemented due to resistence from powerful non-security team members inside the organization. (Usually, the highly paid CISO is playing politics with peers to keep the job by ignoring those security experts). If it also possible that the IT+security team is incompetent and is only good for deploying expensive but useless vendor solutions that do a lot of security theater but do very little to improve actual security. Also possible that the vendor solutions are susceptible to be misconfigured easily to become insecure. Public cloud solutions can help overcome these above shortcomings of in-house security+IT teams in a org politics friendly manner.
throw0101a 6y ago
> […] not trusting them with your data being a major one. Which is why you encrypt the data before sending it over the wire. Just like how you'd use encryption on LTO tapes before sending them offsite.

throw0101a 6y ago

> I can't think of any reason not to use a cloud hosted service for backup today. Or just use a non-Windows file server that supports snapshots: NetApp, Isilon, FreeNAS, etc. If an end-user devices gets infected, and then encrypts mapped drives under the credentials of the user in question, that won't do anything to the read-only data. Snapshots are not backups, but for a lot of the most common situations, they offer a convenient, quick win so people can move on with their day (often in a self-service fashion by just going into a .snapshot/ directory).

JeremyNT 6y ago

These sound like servers used by researchers. I've worked with higher education research computing and you might be surprised at what you would find. Researchers may be generating or churning through countless TB of intermediary data, scratch files, etc. Often, the people who actually run the it infrastructure for researchers are... grad students. Sometimes they have grants for hardware and tight budgets, and paying anything for backups isn't part of it. Sometimes, if you're lucky, the it department will be aware of the work and allowed to help. Now maybe that's not the case here, and there really is one department responsible, and that department decided against spending money on backups. Well maybe they were told by the provost or the dean of whoever that they couldn't afford to back up everything, so they should just stick to file servers. Maybe the boxes compromised here are compute only, and all code and valuable artifacts are expected to be stored safely somewhere else. And maybe the researchers heard this and understood it when they agreed to use the system. But maybe the new grad student didn't get the memo and developed his model in vim on the compute node. The point is, academic computing is kind of the wild west. Weird fiefdoms and weird restrictions, budgetary and otherwise. It's tough to guess which of these scenarios played out from the outside and we really can't know whether the CISO or CTO, or even anybody working for them, dropped the ball.

beh9540 6y ago
I'm so glad I'm not the only one who has experienced this. Only thing I'd add is the lack of understanding about how hard backups are at some of the scales of data in research. Trying to explain to researchers who run the departments that doing a full backup on 35TB+ of data on all budget drives is going to take weeks was something I could never get across, until they lost all their data. And no, snapshots weren't an option on a legacy clustered filesystem that they wouldn't migrate from.
- briffle 6y ago
  But that is only like 3x 12TB USB drives.. </sarcasm>
justinclift 6y ago
> But maybe the new grad student didn't get the memo and developed his model in vim on the compute node. Was trying to think through scenario's like that as well, but they don't really make sense. If various staff members created a few weeks (or even months) worth of work on storage that isn't backed up, then the response from mgmt would generally be "Bad luck, you'll need to redo it". But mgmt decided that work was worth paying US$1.1M+ for, and copping the reputation damage. So, it's a weird mix of potential scenarios to actually get that happening. ;) Maybe someone internal was actually responsible for that malware and is going for some kind of weird payday?

ss2003 6y ago

"I'd fire everyone." Too late. They did that 3 years ago. Fired the IT staff and outsourced to India. A great decision that saved them tons of money. /s https://sanfrancisco.cbslocal.com/2017/02/28/ucsf-tech-workers-have-last-day-on-job-after-outsourcing/ https://sanfrancisco.cbslocal.com/2017/02/28/ucsf-tech-worke... Fire the leadership from the top down. Every one approved outsourcing the IT staff.

carapace 6y ago
From that link: > This move will save UCSF $30 million over the next 5 years. So they're ahead? I'm being slightly snarky here, but also earnest. If they save 30 but lose 1.14 isn't it worth it?
- to11mtm 6y ago
  Depends what the total cost was for this... The 1.14M is just the ransom, doesn't account for how much money was wasted on other aspects of recovery and total downtime...
  - justinclift 6y ago
    Substantial reputational damage will have to figure in there somewhere as well. ;)
techie128 6y ago
> ...Fired the IT staff and outsourced to India... Are you asserting that the breach would not have happened if the IT operations were in-house? Could you also clarify why did you feel it necessary to qualify the location of outsourcing? If the project was outsourced to any other country besides India, this would not have happened?
- wizzwizz4 6y ago
  > If the project was outsourced to any other country besides India, this would not have happened? I think the emphasis is on the fact it was outsourced to another country; you can't generally check the credentials of someone on the other side of the world very easily. The University of Delhi probably has good IT staff, but “Delhi International Computer Help”? Probably not.

hashouseincali 6y ago

Cloud solutions are great unless youre storing huge amounts of data thats being served, modified, and/or backed up. Go with JBOD's, ZFS+snapshots+rsync (or gluster even), and lots of firewalls with MFA and youll be fine (im averaging 60 to 120 TB's in my boxes raid5-7). Google drive is fine for docs. Data should stay home and be backed up elsewhere in the 'house' (still home ... just a separate basement data center or something that wont also burn down). Lets not forget this could have been an inside job that will get around a lot of security other than another humans/montiors vigilance. UCSF outsourcing all their talent is exactly what enabled this.