7 ms·
Backups. I cannot emphasize enough the importance of backups. Take backups, verify your ability to restore from them, and keep them segregated from the rest of
by korethr 5y ago
Backups.
I cannot emphasize enough the importance of backups. Take backups, verify your ability to restore from them, and keep them segregated from the rest of your infrastructure. It doesn't matter how inelegant and hacky your backup solution is, so long as you can restore from it. Any backup you can restore from is better than no backup.
You might get a call from one of your application engineers shortly before bed on a Friday night that the web front-ends are acting weird, and they can't get in to troubleshoot, and then 10 minutes later come to discover that the latest strain of Ryuk has laid waste to 2/3s of the servers and workstations across the company. And then all of a sudden, those VM snapshots you'd been copying off to another file share with a shell script have become your salvation. Yeah, containing Ryuk and the rest of incident response mode are going to suck, but at least now you don't have to write an apology to your customers that the data they entrusted to you has been irrevocably lost.
In case you're wondering, no, that did not literally happen to me. But it is a mild fictionalization of someone I know.
Keep backups, and test your restores regularly, people.
- kemonocode 5y agoModern ransomware gangs focus more on data exfiltration rather than actually locking down data, and it lets them remain undetected for longer too. That said, yes, correct, having good and reliable backups is vital.
- beermonster 5y agoIn combination with techniques to minimise easy/unrestricted exfiltration of data. Such as egress filtering, software access policies/controls, minimum privilege, network segregation/isolation, good user access and network access controls, decent patching policy, good and tested backups and incident response plan…yada yada. Make sure if they do get in, the damage is limited.
- likecarter 5y agoA crucial point you missed: Ransomware gangs often destroy your backup infrastructure. So it's important to create pull-only backups or backups that cannot be deleted / overwritten.
- varispeed 5y ago> backups that cannot be deleted / overwritten That gets complex if your database contains PII. If a user asks for their account to be deleted...
- tomrod 5y agoWould it be sufficient that the delete accounts script be managed and merged at restoration time?
- marcinzm 5y agoYou still have their data and anyone who bypasses the restoration script can get it so no.
- anang 5y agoThis is not true. The prevailing opinion is that backups do not need to be modified and that it’s enough to delete at restore time (and to of course not process the data again).
- varispeed 5y agoThe neatest way I heard is to store PII in an encrypted form and then delete encryption keys when a user requests deletion. That requires the keys to be in a different backup pipeline though.
- viraptor 5y agoIt's not "cannot be overwritten ever". It's more "you can overwrite it using credentials stored in a safe location and never from the production environment". Someone will always have permissions to remove or change the entries, just not easily and not in an automated daily process way.
- korethr 5y agoI thought this was sufficiently implied with "keep them segregated from the rest of your infrastructure," but yes, you are correct. It is important to have a set of backups that can't be destroyed by the attackers. You might get lucky that your hand-rolled solution is so hacky that a ransomware gang overlooks it. But better to not rely on that luck. In the past, this was achieved by having a set of tapes offsite. Today, one might configure Veeam to lie when issued a delete command, and instead send the data off to an Amazon glacier instance that requires different credentials to read, write, and delete.
- the_duke 5y agoI sometimes do infrastructure consulting. One of the first questions I ask is if they have at least one fully independent, full/incremental off-site backup that can't be corrupted from the main infrastructure, and if they have ever checked if they actually work and are restorable. I'm continuously surprised how often the answer turns out to be no after dinner digging, even in larger companies with otherwise well-run IT. No, the automatic 7 day RDS snapshots or turning on S3 versioning is not a sufficient backup. Neither is mirroring to a S3 Glacier bucket in the same org, or rsyncing to a a backup server in the same datacenter. Backups are annoying and unglamorous. Nobody wants to do them, or do the tedious work of validating them or setting up something like an automated restore test. Until the day you lose your data.
- nicbou 5y agoThis applies to average people too. I wonder who among us can say they meet your (reasonable) standard. Like you said, backups are annoying and unglamorous. Yet, the data on my laptop is the only thing I could not replace. It's more important to me than my passport or my birth certificate. Its preservation is certainly worth a bit of thought.
- disabled 5y ago> Like you said, backups are annoying and unglamorous. It’s called having a network attached storage (NAS) device. I have a Synology NAS, which I backup to, continuously at 5 minute intervals. Warning: Microsoft image and file backups sometimes do not work. I recommend Acronis True Image instead, which comes with antivirus. It pretty much always works, never falter never fail. Get the version that allows you to back up to the cloud with blockchain features. You will be happy you did.
- tenebrisalietum 5y agoCan ransomware leak on to your always-connected NAS?
- 5y ago
- thorwasdfasdf 5y agoI think we should make every sysadmin watch Mr.Robot, at the least the first season. That may drive home the point: secure backups are important. ;)
- beermonster 5y agoAnd don’t restore your cloned image using clonezilla whilst on morphine. It’s hard enough to use whilst sober.
- ericalexander0 5y agoWhy do you sweep? https://ericalexander.org/post/devops-and-ransomware/ https://ericalexander.org/post/devops-and-ransomware/
- tomrod 5y agoYou know, I'm curious how large cloud providers handle this. Obviously EC2 for AWS, but what about managed services? A bad ransomware attack on a large cloud provider could cripple a significant portion of the internet.
- beermonster 5y agoVPCs are segregated.
- amiga 5y agoThe hardest part of getting people to do this is people dont want backups. They want restores.
- unstatusthequo 5y agoBackups are nice. Until threat actor deletes them or encrypts them. Backups are also nice until you realize they exfiltrated data too. They are one mitigation but definitely not a sure fire insurance against a data breach and legal implications thereof. Source: I’m a cybersecurity lawyer.
- beermonster 5y agoBackups should be on a different segregated network (ideally off-site) with strict access controls and encryption. The threat actor should not just be able to find them lying about (possible even directly attached) with no authentication or authorisation required to access them. If they are able to access them somehow then they shouldn’t be readable. What you describe is people with a poor backup policy, which yeah I guess describes most folk. But backups aren’t the issue here, poor backups are. And they do not mitigate against the threat of the data being published online which seems to be the new(ish) threat these days with ransomeware.
- beermonster 5y agoBackups help recover your data. But gangs have evolved. They also use exfiltration of your data to embarrass you into paying up, which backups do not mitigate against: the public shaming.