4 ms·

One thing I've always wondered: How do you prevent ransomware from ruining your backups, too? Lots of ransomware tends to be "delayed" – it can't encrypt every

One thing I've always wondered: How do you prevent ransomware from ruining your backups, too? Lots of ransomware tends to be "delayed" – it can't encrypt everything instantly, so it encrypts a little bit each minute. During those minutes, isn't it conceivable that the now-encrypted files replace your old backups? I suppose this isn't really a "backup" but rather a "mirroring strategy." But for certain kinds of data -- photos, video, large media -- can you really afford to do any other kind of strategy? The other question I have is related to that first point: since ransomware can't encrypt everything all at once, how is it possible for your system to continue to function? As you can tell, I'm a ransomware noob, but it's quite interesting to me from an engineering standpoint. Does the system get into a "half encrypted" state where, if you rebooted it, it would fail to boot at all? Or does ransomware targeted at businesses tend to be more of a surgical strike, where it targets and wipes out specific datastores before anyone notices? (It's the "before anyone notices" part that I'm especially curious about. Isn't there some kind of alarm that could be raised more or less instantly, because something detects unexpected binary blobs being created on your disk?)

goldenkey 5y ago
Linux executables can be modified while still running, this is what differentiates windows updates, many requiring rebooting, from Linux updates. Once rebooted, then you realize your executables are tainted.
derekp7 5y ago
Your general backup strategy should follow something like doing full/incremental and/or snapshot based backups. So in the case of your media, if you do a daily snapshot then your daily backups would be very small. And if the media doesn't change that often you can keep weekly copies for several weeks, monthly copies for many months, and several years of yearly snapshots. The other strategy is with tape rotation. You need to have about 30x the tape storage as you have online storage, so you can keep 7 yearly backups, 12 monthly, 6 weekly (all full backups), and 7 - 14 daily incremental backups.
Dylan16807 5y ago
> I suppose this isn't really a "backup" but rather a "mirroring strategy." Correct. > But for certain kinds of data -- photos, video, large media -- can you really afford to do any other kind of strategy? Yes. Make the backup system for that big slow-changing data a moderate amount bigger than the primary data store, and then you can have months of retention at low cost. If too much data changes at once then it should go read-only and send out a barrage of alerts.
mceachen 5y ago
Other replies have backed into this, but the best solution is to 1. Use a COW (copy-on-write) filesystem like btrfs or ZFS 2. Set up snapshots to be taken periodically (hourly/daily) and sent to a different host or volume. 3. Monitor disk usage: if you get hit by a cryptolocker, your disk usage will approximately double as it rewrites all your files. 4. Manually backup snapshots or the full volume to offline storage every N days/weeks/months. In case you missed it, I wrote this up a while back: https://photostructure.com/faq/how-do-i-safely-store-files/ https://photostructure.com/faq/how-do-i-safely-store-files/ TL;DR: Lots of copies keeps stuff safe!
- cunthorpe 5y ago
  5. Don’t use (only) direct disk access to backups I think all of that is lost if the cryptolocker just formats any volumes named Backup. The copy-on-write can’t just be enforced by the filesystem. If this computer can permanently delete content on the backup system, then so can the locker.
- xmodem 5y ago
  Another important step - make the offiste backup 'pull' based - so the credentials to access the data already there do not exist on the system being backed up.
  - BLKNSLVR 5y ago
    Yes! My homelab, admittedly low complexity, uses a NAS device that powers on at a scheduled time, mounts the shares it needs to, runs the 'pull' backup, unmounts, and powers itself off. The intention being that, in the event of intrusion, it's presence and accessibility are limited to the window of time in which it's performing the backup. Additionally to that, a rotating set of removable HDDs as backups of backups that also get spread off-site amongst family members houses. I really should go into offering backup solutions to local small business...
- bruce343434 5y ago
  What's the point of COW? There are 0 tools that restore the "orignals" of the copies on write.
laurencei 5y ago
I use Amazon S3 buckets, with the web server having an IAM user key with very limited access that only allows writes. No reads/listing. So if the web server gets hacked, the hacker can only write to the bucket, but has no way to know what is already there or access anything in it.
- shortj 5y ago
  Make sure you have versioning turned on as well. Even if attackers can figure out your naming conventions and overwrite, you just go back to the first version and everything is good.
- runeks 5y ago
  Would be better if there were a “create/write to new file” permission which doesn’t allow overwriting existing files.
908B64B197 5y ago
> I suppose this isn't really a "backup" but rather a "mirroring strategy." But for certain kinds of data -- photos, video, large media -- can you really afford to do any other kind of strategy? Saving diffs/snapshots will solve the issue. As long as the file doesn't change the cost is almost 0.
deleted 5y ago
[deleted]
Rd6n6 5y ago
If you decided to use incremental backups to mitigate against this, what are your favourite tools or providers? Backblaze with duplicati? Duplicity with s3? Rsync and rclone without it being incremental?
analog31 5y ago
Granted I'm a complete amateur here, but still I wonder if my approach is helpful. My home backup drives are running on Raspberry Pi, so I have control over what other software runs on them. I've been writing Python programs that run on the Pi and monitor for changes. My hypothesis is that if I do the right analysis, I will notice changes to the files unless the ransomware is capable of actually infecting the software running on the Pi itself. I believe that I would detect unexpected binary blobs. Of course this depends on me writing the programs correctly, and a lot of other assumptions, but it might suggest a way to protect backups. My backup "drive" is anything but passive.
- easton 5y ago
  The change you detect would be your files being encrypted, this seems pointless unless you keep automatically rollback changes that don't seem okay to you (which means you need backups for your backups to be compared to...).
  - analog31 5y ago
    Indeed, this assumes the correct files are somewhere but not accessible to the family computers.
Robotbeat 5y ago
You just use cold backups. For home, I have two USB disks I use for backups and I alternate which I use. Neither is plugged in at the same time. At least one is always "cold". For larger scale, you can do the same thing with tape. One tape backup goes off-site (perhaps) or at least cold. The cost isn't that high. A USB spinning disk may cost a third to a 5th that of your SSD hard drive. And you can get hard drives up to 18TB now. But even a portable 2TB USB-powered 2.5in external hard drive is only $60, so this is a cheap and robust strategy.
- benhurmarcel 5y ago
  That mitigates the risk, but it relies on the assumption that you'll notice when files get encrypted. It's not a guarantee, malwares can hide themselves long enough for you to plug both disks before you notice.
  - Robotbeat 5y ago
    True, but I don’t rotate them every day. Maybe once a week or month. Unlikely to not notice by that time. With tape backup, you might keep a tape in cold storage for years.
- theandrewbailey 5y ago
  > For home, I have two USB disks I use for backups and I alternate which I use. Neither is plugged in at the same time. At least one is always "cold". Why not have one of those drives off site, and rotate every so often? Carry the drive with you when you swap, so that the original and all backups are not in the same place at the same time. I have three external drives. Originally I planned to keep two offsite, but I don't have an offsite office anymore.

goldenkey 5y ago

Linux executables can be modified while still running, this is what differentiates windows updates, many requiring rebooting, from Linux updates. Once rebooted, then you realize your executables are tainted.

derekp7 5y ago

Your general backup strategy should follow something like doing full/incremental and/or snapshot based backups. So in the case of your media, if you do a daily snapshot then your daily backups would be very small. And if the media doesn't change that often you can keep weekly copies for several weeks, monthly copies for many months, and several years of yearly snapshots. The other strategy is with tape rotation. You need to have about 30x the tape storage as you have online storage, so you can keep 7 yearly backups, 12 monthly, 6 weekly (all full backups), and 7 - 14 daily incremental backups.

Dylan16807 5y ago

> I suppose this isn't really a "backup" but rather a "mirroring strategy." Correct. > But for certain kinds of data -- photos, video, large media -- can you really afford to do any other kind of strategy? Yes. Make the backup system for that big slow-changing data a moderate amount bigger than the primary data store, and then you can have months of retention at low cost. If too much data changes at once then it should go read-only and send out a barrage of alerts.

mceachen 5y ago

Other replies have backed into this, but the best solution is to 1. Use a COW (copy-on-write) filesystem like btrfs or ZFS 2. Set up snapshots to be taken periodically (hourly/daily) and sent to a different host or volume. 3. Monitor disk usage: if you get hit by a cryptolocker, your disk usage will approximately double as it rewrites all your files. 4. Manually backup snapshots or the full volume to offline storage every N days/weeks/months. In case you missed it, I wrote this up a while back: https://photostructure.com/faq/how-do-i-safely-store-files/ https://photostructure.com/faq/how-do-i-safely-store-files/ TL;DR: Lots of copies keeps stuff safe!

cunthorpe 5y ago
5. Don’t use (only) direct disk access to backups I think all of that is lost if the cryptolocker just formats any volumes named Backup. The copy-on-write can’t just be enforced by the filesystem. If this computer can permanently delete content on the backup system, then so can the locker.
xmodem 5y ago
Another important step - make the offiste backup 'pull' based - so the credentials to access the data already there do not exist on the system being backed up.
- BLKNSLVR 5y ago
  Yes! My homelab, admittedly low complexity, uses a NAS device that powers on at a scheduled time, mounts the shares it needs to, runs the 'pull' backup, unmounts, and powers itself off. The intention being that, in the event of intrusion, it's presence and accessibility are limited to the window of time in which it's performing the backup. Additionally to that, a rotating set of removable HDDs as backups of backups that also get spread off-site amongst family members houses. I really should go into offering backup solutions to local small business...
bruce343434 5y ago
What's the point of COW? There are 0 tools that restore the "orignals" of the copies on write.

laurencei 5y ago

I use Amazon S3 buckets, with the web server having an IAM user key with very limited access that only allows writes. No reads/listing. So if the web server gets hacked, the hacker can only write to the bucket, but has no way to know what is already there or access anything in it.

shortj 5y ago
Make sure you have versioning turned on as well. Even if attackers can figure out your naming conventions and overwrite, you just go back to the first version and everything is good.
runeks 5y ago
Would be better if there were a “create/write to new file” permission which doesn’t allow overwriting existing files.

908B64B197 5y ago

> I suppose this isn't really a "backup" but rather a "mirroring strategy." But for certain kinds of data -- photos, video, large media -- can you really afford to do any other kind of strategy? Saving diffs/snapshots will solve the issue. As long as the file doesn't change the cost is almost 0.

deleted 5y ago

[deleted]

Rd6n6 5y ago

If you decided to use incremental backups to mitigate against this, what are your favourite tools or providers? Backblaze with duplicati? Duplicity with s3? Rsync and rclone without it being incremental?

analog31 5y ago

Granted I'm a complete amateur here, but still I wonder if my approach is helpful. My home backup drives are running on Raspberry Pi, so I have control over what other software runs on them. I've been writing Python programs that run on the Pi and monitor for changes. My hypothesis is that if I do the right analysis, I will notice changes to the files unless the ransomware is capable of actually infecting the software running on the Pi itself. I believe that I would detect unexpected binary blobs. Of course this depends on me writing the programs correctly, and a lot of other assumptions, but it might suggest a way to protect backups. My backup "drive" is anything but passive.

easton 5y ago
The change you detect would be your files being encrypted, this seems pointless unless you keep automatically rollback changes that don't seem okay to you (which means you need backups for your backups to be compared to...).
- analog31 5y ago
  Indeed, this assumes the correct files are somewhere but not accessible to the family computers.

Robotbeat 5y ago

You just use cold backups. For home, I have two USB disks I use for backups and I alternate which I use. Neither is plugged in at the same time. At least one is always "cold". For larger scale, you can do the same thing with tape. One tape backup goes off-site (perhaps) or at least cold. The cost isn't that high. A USB spinning disk may cost a third to a 5th that of your SSD hard drive. And you can get hard drives up to 18TB now. But even a portable 2TB USB-powered 2.5in external hard drive is only $60, so this is a cheap and robust strategy.

benhurmarcel 5y ago
That mitigates the risk, but it relies on the assumption that you'll notice when files get encrypted. It's not a guarantee, malwares can hide themselves long enough for you to plug both disks before you notice.
- Robotbeat 5y ago
  True, but I don’t rotate them every day. Maybe once a week or month. Unlikely to not notice by that time. With tape backup, you might keep a tape in cold storage for years.
theandrewbailey 5y ago
> For home, I have two USB disks I use for backups and I alternate which I use. Neither is plugged in at the same time. At least one is always "cold". Why not have one of those drives off site, and rotate every so often? Carry the drive with you when you swap, so that the original and all backups are not in the same place at the same time. I have three external drives. Originally I planned to keep two offsite, but I don't have an offsite office anymore.