10 ms·

Update on IT Security Incident at UCSF

• ed25519FUUU 6y ago
  > The data that was encrypted is important to some of the academic work we pursue as a university serving the public good. We therefore made the difficult decision to pay some portion of the ransom, approximately $1.14 million Once you’ve paid the Danegeld, you’ll never get rid of the Dane.
• deleted 6y ago
  [deleted]
• chmaynard 6y ago
  Ouch. No backup?
  • craftyguy 6y ago
    It's pretty obvious that the answer is 'no backup'...
    • scottwb 6y ago
      It’s not that obvious. It could be assurance that the data would be deleted and not sold/shared.
      • craftyguy 6y ago
        > We therefore made the difficult decision to pay some portion of the ransom, approximately $1.14 million, to the individuals behind the malware attack in exchange for a tool to unlock the encrypted data and the return of the data they obtained. I assume you read that though before you replied, right?
        • rovr138 6y ago
          > and the return of the data they obtained
          • hanche 6y ago
            … as if you can’t “return” the data and keep a copy at the same time …
            • rovr138 6y ago
              Might require ‘best effort’ to keep data private.
• hoomank3 6y ago
  The paid ransom, will unfortunately embolden the criminals to strike again in search of the next big payday. If it worked once, it could work again.
  • draw_down 6y ago
    They ain’t doing it for the love of the game!
  • verandaguy 6y ago
    Optimistic counterpoint: a high-profile, (relatively) high-value ransom payout like USC's may incentivise other orgs vulnerable to this kind of attack to take steps to prevent this kind of issue. Anything from restricting program capabilities/permissions for external executables, to keeping "colder" backups of business-critical data, to monitoring and responding to software that looks like it's traversing the whole filesystem, could reduce the harm ransomware causes.
    • garmaine 6y ago
      You can't really protect against this sort of thing. A lot of our IT security runs on trust. The only way to really prevent this is to make sure that ransom attacks don't pay out. EDIT: I should mention that I've managed IT services for a major private university earlier in my career, and I am now a software security consultant. When I say it is not possible, I mean that pragmatically. A FAANG company can control their IT well enough to make sure this doesn't happen to them, but a hospital or university relies on computer systems running software way outside of their control. That MRI machine? Its controller is probably running some ancient version of Windows Server 2003 with proprietary drivers. That university registration app? Custom coded by generations of CS student interns running on a shared system whose operating constraints are set by the Novell GroupWise instance that is co-hosted on it. As a practical matter, one of these organizations simply cannot reduce their risk to zero or near zero. There's too many attack vectors they don't have control over. The IT departments can't mandate proper security because they don't have the budget to enforce.
      • 6y ago
• adrianN 6y ago
  How did they not have backups of important data?
  • trhway 6y ago
    there was mentioned a threat of publishing of some secret data, and that i think may be a reason why UCSF was willing to pay https://www.trialsitenews.com/hacking-group-launches-successful-ransomware-attacks-against-ucsf-experts-think-covid-19-connection/ https://www.trialsitenews.com/hacking-group-launches-success...
    • eigenvalue 6y ago
      This seems much more likely than the scenario where they simply didn’t have any backups of the data.
  • tuwtuwtuwtuw 6y ago
    Did you ever work for an organization?
  • _Wintermute 6y ago
    It's amazing how bad some of the practices are outside of tech companies. I worked for a pharmaceutical company, one of their half million dollar microscopes was saving data locally, when I asked if it was backed up at all the answer was: "no ... but it's using one of those red hard-drives".
• surfpel 6y ago
  Don’t they have insurance for these things? A small college near me had an attack like this but paid via insurance.
  • gorgoiler 6y ago
    Would you want the insurance policy to pay out though? At some level of recklessness, insurance becomes void. I think a lack of infrastructure to restore a hacked server — with data valued at over $1M — is negligent enough to not be covered. But maybe UCSF are on MegaCo’s YOLO tier of server insurance, which is so expensive and isolated it has no impact on my MegaCo pet insurance premiums?
    • surfpel 6y ago
      But patients suffering for administrative negligence? I don’t agree with that but there does need to be some kind of incentive there.
      • gorgoiler 6y ago
        That’s a good point, I hadn’t thought of that. Perhaps the health of some businesses is so important that they should be protected by, ahem, state-backed insurance paid for by the taxpayer?
    • closeparen 6y ago
      I'll venture a guess that UCSF was nominally in compliance with some relevant bureaucratic regime (ISO 27001, SOC 2, etc), and that was good enough for a stodgy insurance company that's not very sophisticated about "cyber risk" (in case use of the term "cyber" isn't a giveaway...)
      • gorgoiler 6y ago
        The CISO’s top 5 “security tips” PR piece didn’t mention off-system backups. Policy voided.
    • 6y ago
• locusofself 6y ago
  What kind of data is worth at least a million dollars and isn't properly backed up? Unbelievable. Some heads should roll .
  • duxup 6y ago
    It's a university, university IT is often ultra political and those who win the various battles make the rules, regardless of competence... and often without IT's sign off. Some universities generally have done better about such things and are making progress... but generally there is a push and pull for IT dollars by unversity departments who want to spend that money as they wish for their given programs and then that money comes FROM IT ... who down the road are then tasked with the costs related to maintaining it and the terrible decisions a department made in the meantime... or in the worst of cases tasked with securing that data and / or making it work at all. It's the same story for IT in the private sector to some extent, but it is way worse at many universities. Imagine if your HR director got to pick the PCs to support, networking equipment, software, backup methods (if any) all on their own and wanted zero input. That's kinda how it is at many universities. I spent months helping a large university dig out from a program where they hooked up some super special microscopes worth millions of dollars ... to low grade network switches and storage. I got to try to explain why you can't put 10,000 pounds of data into a borderline consumer grade network ... in all of a couple milliseconds.
    • throw0101a 6y ago
      > It's a university, university IT is often ultra political […] While you're not necessarily wrong, another option is budget. If this is academic- / research-generated data, then it could have been paid for by grant money, and most of the cash goes to paying grad students and perhaps some computer equipment. IT may have chargebacks (they have bills / cost centres to pay too after all), and no one wants to "waste" grant money. Often these things are 'shadow IT' run in an ad hoc fashion by just throwing together some PCs. If the group's expertise is in medicine / biology, how many members want to give up their day-light research hours to run the computer infrastructure? I've spent about half my IT career in the academic sphere, and cheap solutions can be a fight to implement even if they solve the problem; even free (open source) ones can be an effort if they take time or slow down the workflow. And these people aren't stupid: they 'know' they should do some of these things. But people 'know' they should get exercise, and how many folks do that?
• justinclift 6y ago
  Wonder if the cost of backups for the affected servers would have been less than the US$1.1M they paid? ;) At a guess, they could probably have put in a some fairly high quality storage + backups for that price, and still had money left over for all expenses paid staff vacations. :)
• rkho 6y ago
  As a reminder, in 2017 UCSF offshored all of its IT staff to HCL Technologies and forced their then-employees to train their replacements before laying them off. They brought the replacements into the Bay Area on H1B temporarily while they were trained by their soon-to-be-laid-off counterparts and then sent back overseas to continue their roles once training was complete. https://sanfrancisco.cbslocal.com/2017/02/28/ucsf-tech-workers-have-last-day-on-job-after-outsourcing/ https://sanfrancisco.cbslocal.com/2017/02/28/ucsf-tech-worke...
  • scottwb 6y ago
    It wasn’t all IT, not even half.
  • mc32 6y ago
    What kind of monster of an employer makes their employees train the replacements they're getting fired for? If that were me, I'd organize and have everyone quit; let them figure things out. Screw the pittance of a severance.
    • PeterStuer 6y ago
      From what i have seen over the years this is an industry standard practice. Don't assume the employees about to be sacked are in the know. Sometimes this is hidden under the guise of big IT modernization projects. The company transitions or consolidates towards ERP platform X. It is a huge operation that touches almost every part of the business. Armies of analysts are brought in to map the AS-IS business operations, the system is developed, once stable enough, the other shoe drops and the whole IT department is now moved off- or near shore to the company that transitioned the system while other 'non-strategic' parts of the business are 'streamlined' into managed service contracts with other parties in the project consortium.
    • Ghjklov 6y ago
      Their severance pay was probably on the line...too sad
    • voodoochicken 6y ago
• Gatsky 6y ago
  I have to assume things were backed up, and the attackers were deep enough in the system to find or delete the backups. Anyway, I wonder how this payment was made...
• dangero 6y ago
  What I find crazy about this -- no guarantee that the ransom payment would unlock the machines -- did they send 1.14M in one go or was it a smaller amount for the first machine, then an additional fee for each additional machine? Also would be interested to know -- was it Bitcoin or some other cryptocurrency that was used?
  • kevinsundar 6y ago
    It’s in the best interest of the people who make these types of malware to provide the decryption tools once the ransom is paid. If news gets out that even if you pay the ransom you won’t get the decryption tools then no one would pay the ransom and the hackers get no money at all.
  • recursivecaveat 6y ago
    Apparently crypto-ransom people are actually pretty trustworthy about unlocking the machines. It doesn't really cost them anything (0% chance you were gonna send a 2nd payment if they didn't unlock), and their reputation as 'fair' is very important for securing future ransoms.
    • closeparen 6y ago
      Someone's gotta be thinking about doing a ransomware operation that doesn't unlock the data in order to poison the well.
      • Animats 6y ago
        It's been done. There was one "ransomware" attack that just erased everything.
        • LilBytes 6y ago
          NotPetra
    • alextheparrot 6y ago
      That creates an interesting grey hat scenario where a normally good actor could execute a few ransoms and not unlock the machines. Eroding trust in the entire scheme could reduce the long-term effectiveness of the scheme
• robbrown451 6y ago
  Maybe there should be a law that if you pay a ransom, you are required to pay the same amount as a fine. Because paying these ransoms is funding the criminals.... how about you have to also fund law enforcement to combat those criminals? (also, this should reduce the amount that actually goes to the bad guys, since the amount of ransom would have greater downward pressure, i.e. if they'd probably not be able to collect more than $0.57M because that would cost UCSF $1.14M)
  • ukulele 6y ago
    This would make the payers far less likely to report it, and ultimately make it much harder to track
    • robbrown451 6y ago
      Well if it was illegal to not report it, they'd be putting themselves at a lot of risk. I can't see a university or corporation making a large payment like that illegally. Also, it may not make the costs to them more. Remember the amount of ransom is based on what the ransomers think the ransomees are willing to pay. Today, the reason it was 1.4 million instead of 2.8 million, is that they didn't think UCSF would pay the latter amount. So if they knew UCSF would have to pay double the amount of the ransom, they'd have to only ask for half as much.
  • tenuousemphasis 6y ago
    Or since we already have to pay for law enforcement anyway, they could just do their jobs and catch the bad guys. But I think we all know at this point that "catching bad guys" is just a pretense, right?
    • robbrown451 6y ago
      Not sure what you are saying. We already pay for law enforcement, but their resources aren't infinite. This would serve three purposes: help fund those things that are costly, deter the bad guys (since they can't ask for as much money if paying a ransom is going to be twice as costly to their victims), as well as to add additional incentivize people to secure their systems. I don't know what you mean by ""catching bad guys" is just a pretense". Pretense for what? By who? That sounds very conspiracy minded.
• schoolornot 6y ago
  CISO: https://cio.ucop.edu/spotlight-patrick-phelan-once-a-ucla-bruin-now-ciso-at-ucsf/ https://cio.ucop.edu/spotlight-patrick-phelan-once-a-ucla-br... I can't think of any reason not to use a cloud hosted service for backup today. OneDrive, Dropbox, and Google Drive all sign BAAs and give you versioning amongst a million other security features. AWS even has offerings that let you take periodic snapshots of on-premise volumes. Point in time recoveries for the entire account would be nice add too but not having to fork over a million dollars in exchange for a few clicks sounds like a bargain. Hell we were setting up write-only S3 buckets for critical data stores 5+ years ago. Gross incompetence, I'd fire everyone. A district-wide outage for a week, fine. A 1.4 million dollar check to get data that should have been archived somewhere GTFO.
  • AnthonyMouse 6y ago
    There are reasons not to use cloud services for backups, not trusting them with your data being a major one. But then you should still be using some kind of internal backup system. There is no excuse for not having backups.
    • jtokoph 6y ago
      If you don’t trust them with your data, you can encrypt it with your own keys.
      • briffle 6y ago
        Really, someone just did that for them, for the bargain price of $1.4M. I'm guessing they saved at least that much by outsourcing their entire IT department a few years ago.
        • znpy 6y ago
          This is really the best possible answer. you just did the root cause analysis for them too (you should send them an invoice). Nothing new, in a way: cut the spending on IT, lower quality, get incidents. Meh.
• speedgoose 6y ago
  I'm surprised that paying the ransom is a thing. This is a very selfish thing to do, as you encourage the authors of such attacks.
• janwillemb 6y ago
  The article is inconsistent about the severity, which lowers its trustworthiness. It states that everything is under control, a top super leading security guru expert is on top of it, no important data was stolen or exposed, everything will be fine soon. And... oh yeah, BTW, we'll be paying an astronomic sum of money to the criminals to get our data back.
• eitland 6y ago
  I recently listened to Mikko Hyppönen as the company I work for had invited him to give a talk. It was very interesting. One thing I hadn't realized before was that ransomware criminals has developed their own backup strategy: - in addition to encrypting the data they will also exfiltrate it and threaten to publish it ob the internet for everyone to see. That way it doesn't necessarily help an organization just because they have multiple layers of offsite read-only backups.
• rubatuga 6y ago
  To understand why they didn't back up: a) they are academics, which means funding is lacking and b) it's in the field of medicine, known for its lack of tech adoption.
• Veserv 6y ago
  What a cheesy attack. It only cost them $1.14M to get their data back. I can not tell if the attack hit something inconsequential, the criminals are stupid, or they just do not understand finances for them to ask for such a tiny sum. UCSF received $1.43B in grants and contracts during 2017-2018 [1]. Assuming they are generating an equivalent amount of value in knowledge evenly distributed over time, the loss of one day of research would be ~$3.9M. So, if the the last whole organization backup was one day ago and the attackers were only able to stop access to the last day of work since they did not think to corrupt the backups before they went out, then the ROI of paying off the ransom would be ~3.43. If they were able to affect the entire organization for an entire week, then the cost would be ~$27.3M with an ransom ROI of ~24. So, assuming they did any damage of consequence, asking for ~$1.14M seems like robbing a person at gunpoint for their pocket lint. [1] https://www.ucsf.edu/news/2019/02/413396/ucsf-top-public-recipient-nih-funds-pushing-unbroken-streak https://www.ucsf.edu/news/2019/02/413396/ucsf-top-public-rec...
  • WrtCdEvrydy 6y ago
    Doesn't matter, it's the perfect crime... The encryption is done on the user's machine using their processing power, and there's virtually no downside for you (either they pay and you decrypt or they don't and you just dissapear). Ransomware will be an issue for years to come.
    • Veserv 6y ago
      I am confused by your response since it seems to agree with my post but you used words that indicate you think that they do not agree. Did you mean to reply to someone else? I am saying that that the ransom is hilariously small compared to what they would probably be able to get. So, when they realize they can actually ask for a number that is not a rounding error they will probably increase the number/outcome of attacks. They are criminals robbing people at gunpoint and only asking for their pocket lint. Once they all realize they can ask for a wallet and get it, I think the number is going to go way up.
      • rovr138 6y ago
• ppiuser 6y ago
  How incredibly negligent and incompetent of the ucsf IT staff to not have this data sufficiently backed up, not to mention protected. Does this university offer IT courses? If so their reputation has been seriously tarnished.
• noodlesUK 6y ago
  This just illustrates how incredibly important solid backup strategies can be. A big university should be able to figure out how to make WORM (write once read many) backups of their data. A million bucks buys a shitload of cloud storage or physical airgapped tapes...
  • TwoBit 6y ago
    The same weakness that resulted in lack of backups may be related to how the intruders were able to get onto the machines in the first place.
• Isinlor 6y ago
  Here is Dutch University of Maastricht paying 0.2 million euro ransom last year in December. Seems like business is blooming. https://www.reuters.com/article/us-cybercrime-netherlands-university/university-of-maastricht-says-it-paid-hackers-200000-euro-ransom-idUSKBN1ZZ2HH https://www.reuters.com/article/us-cybercrime-netherlands-un...
• Gatsky 6y ago
  Is there a way to restrict encryption at a hardware level? The conditions where you would like to voluntarily encrypt data are usually quite rarefied. Allowing any sort of encryption activity on your system seems like a hazard these days.
  • justinclift 6y ago
    Storage systems will just see a lot of data being changed. You could have some sort of alert in place, but if the malware doesn't write fast enough to trigger it, you'd still miss the problem. Generally, the approach places take is having backups (eg to tape, off site, etc), and/or having storage that makes a snapshot every few hours and retains them for days/weeks/months. Snapshotx are generally very low cost and easy these days, as it's just a pointer manipulation thing on (say) ZFS rather than a complete copy of the entire data set.
    • Gatsky 6y ago
      Backups are the default solution. But my understanding of this UCSF case is that their system was compromised and then ransomwared. In this situation, the attacker could interfere with your backup system for several weeks or even spoil existing backups. It would still be nice to restrict encryption on a system that you control. I suspect, but don't know, that encryption has a particular pattern of memory and CPU activity that could be recognised. Or if there are a few commonly used libraries you could do it that way, although an attacker could roll their own encryption.
• throwaway87634 6y ago
  I've worked in an institution for which I rewrote a public facing database to replace their old and clunky system. I had numerous emails with them to transition smoothly between the old and the new system, and the last step was for them to give me the latest snapshot of the MSSQL db that would allow me to import the last two weeks of data entry. Scripts were ready, so the down time would be 10 min which was totally acceptable for that tool. That's when I realize the backup was two weeks old, that they had already deleted the machine as soon as the transition started (against our plan), and that the automated backups at the scale of the university had not been working for months (they realized it because of this incident)...
• ponker 6y ago
  There should be a government agency that tracks down these ransomware creators and shoots them in the fucking head.