6 ms·

Nearly all my personal photos were encrypted by the helprecover@foxmail.com ("HELP") variant of Phobos. I've been holding onto the encrypted copies for a while

Nearly all my personal photos were encrypted by the helprecover@foxmail.com ("HELP") variant of Phobos. I've been holding onto the encrypted copies for a while in hopes that some people were working on a crack, and I'm excited to read this update. Sidecar question: when automating your backups, what's a good way to make sure your rolling backups aren't simply backing up malware-encrypted files? I found out too late that all my backups were encrypted. I only had 1 week retention to save space. Is longer retention and manual checks the only sane strategy? Or has some backup software built in sanity detection for crypto attacks by now?

908B64B197 4y ago
> what's a good way to make sure your rolling backups aren't simply backing up malware-encrypted files? Compute checksums. Also, if storing diffs check out how many files the back-up think changed. All of your photo library getting re-uploaded should be a red flag.
- Helmut10001 4y ago
  I do this using an `rsync --dry-run` with `--stats` enabled, you can get the number of changed/deleted files from the stats and then decide whether you want to proceed without `--dry-run`. This is for off-site backups. For on-site, I prefer ZFS Snapshots.
throwaway78941 4y ago
Don't save just up to 1 week old snapshots. Also save a 2 week old, 3 week old and 1 month old one.
bombcar 4y ago
I do an off-line, off-site backup once a year that never gets overwritten. ZFS snapshots can help here, too, for online stuff, especially if you have monitoring about changeset size.
- mavhc 4y ago
  zfs server where you can only ssh into it with 1 user, and that user cannot zfs destroy, only zfs receive
biesnecker 4y ago
I store two weeks of daily snapshots, and then 12 months of monthly snapshots. It gives me a year of pretty good coverage for only about 2x the cost of just two weeks of backups.
ridgered4 4y ago
One technique would be to place unchanging bait files that you pre-check before allowing the backup to proceed.
- 0cf8612b2e1e 4y ago
  That’s a nifty and cheap idea. Now I am wondering if I should make the standard juicy targets (eg ~/Documents, .config, .ssh) complete decoys and put all of my real data just off to the side. Could still be hit by a generic attack, but targeted data extraction attempts would initially fail.
  - elcritch 4y ago
    Hmmm, settings things like `~/.ssh` to non standard locations too would probably block a lot of the standard dependency-chain-malware coming around as well.
heywhatupboys 4y ago
don't backup file names. Backup checksums.
- TacticalCoder 4y ago
  I agree. And for some stuff you get cryptographic checksums for free. Backup of Git repositiories: ... # git fsck --full error: unable to unpack contents of .git/objects/a2/cf1a9631658799733f43c3b3f0a799696a4b21 error: a2cf1a9631658799733f43c3b3f0a799696a4b21: object corrupt or missing: .git/objects/a2/cf1a9631658799733f43c3b3f0a799696a4b21 Oops... No matter if it's a malware, the lack of ECC which by bad luck induced a bit flip that wasn't detected (on an otherwise okay Git repo) or a disk failing, it's trivial to detect if the repo is corrupted. Same for my ripped archive of Audio CDs. The rippers save lots of information and the rips are bitperfect, cross checked with other people's rips' checksums. And the checksums are all there. For family pictures, I add a checksum to the pictures myself. Backups aren't really backups until they've been verified :)
  - jxramos 4y ago
    what do you mean adding checksum to the picture, do you add the checksum as a filename suffix eg IMG0001_<checksum>.jpg, something like that? Or do you tuck it into the exif data and have a tool that computes the checksum of the file minus the checksum part.
    - TacticalCoder 4y ago
      Yup exactly just adding a suffix. I'm not only backing .jpg files. For example I also backup a few screenshots (some are in .png and some are in .webp format). So I don't care about the different pictures (or short family movies) format. I just wrote some Clojure / babashka code to do that. I also truncate the checksum so that the filename doesn't become gigantic: it's not sensitive content, it's just to detect corruption. Then I can use another computer and generate, say, all the thumbnails of the pictures and do a quick eyeball verification. If it looks correct, later on I can just automatically have the checksums verified. Funnily enough I got a few old JPG pictures who were corrupt but I ended finding the correct version on older backups. Checksum then helps too: otherwise you have two files with the same name (say on different HDD), but only one is correct and you don't know which one without manually opening them. It's not super advanced and maybe a bit overkill but it's not complicated and works fine for my use case. P.S: I take it another way would be to use a fs that use content-based addressing or does checksumming for me.
werdnapk 4y ago
My backups got attacked by ransomware, but I only caught it about 6-8 months after it occurred. Thankfully, my backup drive is copied to another backup which never deletes files, only copies them, so the renamed files that were encrypted were eventually copied over to my second drive, but the originals remained. The attacker wasn't aware of the second drive.
dec0dedab0de 4y ago
For something like photos you could just store every revision.
jewel 4y ago
Sorry that happened to you. Using a de-duplicating backup solution (such as rdiff-backup or restic), will let you keep daily increments indefinitely with very little overhead.
fps_doug 4y ago
Don't do simple rolling backups, use something with deduplication like borg backup or ZFS/btrfs if you want to do it at the FS level. The backup size should not increase by much more than the actual size of any new files, so if suddenly, you need twice as much backup space because all your files seem to have changed, you should get suspicious.
- temp2022account 4y ago
  Seconded, I have an external 2tb HDD that stores 1 years worth of daily backups from my 256gb (~180gb used at any time) laptop hard drive. My backup script (https://gist.github.com/Jeffrey-P-McAteer/7d4b9052825914b5e0c500c04251e973 https://gist.github.com/Jeffrey-P-McAteer/7d4b9052825914b5e0...) takes maybe 30 minutes for a full backup, 5 minutes for most deltas. Files which are the same get hard-linked to the previous days backups, new files are copied over and content-de-duped by btrfs.
  - 0cf8612b2e1e 4y ago
    If nothing else, external hard drives are cheap and robust enough that I think more people should invest in having an offsite backup. Annually make a full backup, write the date on the outside, and leave it at the parents house. Make that your family holiday ritual.
    - gattr 4y ago
      As of recently, my setup for backups consists of 2x4 TB HDDs (from different manufacturers) with BTRFS in RAID 1, plugged into a small 2-drive USB3 docking station. With both checksumming and mirroring, feels pretty safe from HW-failure/bit rot standpoint (if one disk fails, you can still mount the other in "degraded" mode).
- wyager 4y ago
  Agreed. I'll shill rsync.net (no affiliation, just a happy customer) and their ZFS VM backup service. It's basically just a lightweight freebsd VM with a big ZFS volume attached, so you can `zfs send` incremental backups to it, and they support meta-snapshotting of your backup machine on their end. I wrote https://github.com/wyager/zfs-backup https://github.com/wyager/zfs-backup to manage my automatic incremental backups, and there are a number of other tools like this.
msm_ 4y ago
> Nearly all my personal photos were encrypted by the helprecover@foxmail.com ("HELP") variant of Phobos. I can't assist you with recovery, and without a lot of logs and forensics data (or a significant performance improvement), the described method is likely unfeasible. But I'll try to find a matching sample and let you know if it's vulnerable. > Sidecar question: when automating your backups, what's a good way to make sure your rolling backups aren't simply backing up malware-encrypted files? Lots of good responses, I like incremental backups without ovewriting anything (supported OOTB by all copy-on-write filesystems, like ZFS or BTRFS). Not sure how to configure this on Windows.
bauruine 4y ago
I use borgbackup to a server and once a month upload it to backblaze b2 with a 90 day retention policy on the bucket. The policy doesn't let you modify or delete the files for 90 days even if you have access to the account. Costs something like 5$ for the 3x500GB of backups I have.
GordonS 4y ago
> when automating your backups, what's a good way to make sure your rolling backups aren't simply backing up malware-encrypted files? Maybe you could check the level of entropy (measure of randomness) of files before backing up - very high entropy could suggest encrypted data?
- KMag 4y ago
  This is good. Some antivirus programs run this check, but some ransomware adapted by encrypting 16-byte AES blocks every so often in the file, so that the file becomes useless without entropy increasing too much. Also, JPEG, PNG, .jar, .xlsx, etc. are already compressed, so pretty high entropy to begin with. As others have pointed out, the growth rate of your de-duplicated backup size is probably the best way to detect ransomware.
proactivesvcs 4y ago
I use restic which has very flexible retention policies. Rather than retain simply "x days", one can have it "keep the last x hourlies" all the way up to "keep the last x annuals", and XOR many criteria in one policy. Since it uses snapshotting, compression and deduplication it's very storage-efficient. They also offer a simple REST server which allows one to backup with append-only permissions, which can alleviate a ransomware attack.
powersnail 4y ago
I use rsync manually, and always with "--dry-run" first. Unless the ransome-ware is smart enough to rot my backup very, very, very slowly, I should be able to detect any problem by simply reading which files are to be overwritten. 99.99% of the files I back up rarely change. I also have most of my non-sensitive data on Onedrive, which keeps old versions of files.
macjohnmcc 4y ago
I worked in backup software for 22 years until a few years ago. My suggestion is to make a backup and store it for long term on an external drive. The to the cloud full followed incremental backup daily so you can go back to any point along the way at any time. Currently I use Acronis and it works well. I backup to their cloud to avoid having my local backups also encrypted preventing restoring in the event of failure or malware. Good backup strategy would not overwrite old backups until you must and backup regularly, so you don't lose any more data than you can afford to lose. Remember recovery time as well.

908B64B197 4y ago

> what's a good way to make sure your rolling backups aren't simply backing up malware-encrypted files? Compute checksums. Also, if storing diffs check out how many files the back-up think changed. All of your photo library getting re-uploaded should be a red flag.

Helmut10001 4y ago
I do this using an `rsync --dry-run` with `--stats` enabled, you can get the number of changed/deleted files from the stats and then decide whether you want to proceed without `--dry-run`. This is for off-site backups. For on-site, I prefer ZFS Snapshots.

throwaway78941 4y ago

Don't save just up to 1 week old snapshots. Also save a 2 week old, 3 week old and 1 month old one.

bombcar 4y ago

I do an off-line, off-site backup once a year that never gets overwritten. ZFS snapshots can help here, too, for online stuff, especially if you have monitoring about changeset size.

mavhc 4y ago
zfs server where you can only ssh into it with 1 user, and that user cannot zfs destroy, only zfs receive

biesnecker 4y ago

I store two weeks of daily snapshots, and then 12 months of monthly snapshots. It gives me a year of pretty good coverage for only about 2x the cost of just two weeks of backups.

ridgered4 4y ago

One technique would be to place unchanging bait files that you pre-check before allowing the backup to proceed.

0cf8612b2e1e 4y ago
That’s a nifty and cheap idea. Now I am wondering if I should make the standard juicy targets (eg ~/Documents, .config, .ssh) complete decoys and put all of my real data just off to the side. Could still be hit by a generic attack, but targeted data extraction attempts would initially fail.
- elcritch 4y ago
  Hmmm, settings things like `~/.ssh` to non standard locations too would probably block a lot of the standard dependency-chain-malware coming around as well.

heywhatupboys 4y ago

don't backup file names. Backup checksums.

TacticalCoder 4y ago
I agree. And for some stuff you get cryptographic checksums for free. Backup of Git repositiories: ... # git fsck --full error: unable to unpack contents of .git/objects/a2/cf1a9631658799733f43c3b3f0a799696a4b21 error: a2cf1a9631658799733f43c3b3f0a799696a4b21: object corrupt or missing: .git/objects/a2/cf1a9631658799733f43c3b3f0a799696a4b21 Oops... No matter if it's a malware, the lack of ECC which by bad luck induced a bit flip that wasn't detected (on an otherwise okay Git repo) or a disk failing, it's trivial to detect if the repo is corrupted. Same for my ripped archive of Audio CDs. The rippers save lots of information and the rips are bitperfect, cross checked with other people's rips' checksums. And the checksums are all there. For family pictures, I add a checksum to the pictures myself. Backups aren't really backups until they've been verified :)
- jxramos 4y ago
  what do you mean adding checksum to the picture, do you add the checksum as a filename suffix eg IMG0001_<checksum>.jpg, something like that? Or do you tuck it into the exif data and have a tool that computes the checksum of the file minus the checksum part.
  - TacticalCoder 4y ago
    Yup exactly just adding a suffix. I'm not only backing .jpg files. For example I also backup a few screenshots (some are in .png and some are in .webp format). So I don't care about the different pictures (or short family movies) format. I just wrote some Clojure / babashka code to do that. I also truncate the checksum so that the filename doesn't become gigantic: it's not sensitive content, it's just to detect corruption. Then I can use another computer and generate, say, all the thumbnails of the pictures and do a quick eyeball verification. If it looks correct, later on I can just automatically have the checksums verified. Funnily enough I got a few old JPG pictures who were corrupt but I ended finding the correct version on older backups. Checksum then helps too: otherwise you have two files with the same name (say on different HDD), but only one is correct and you don't know which one without manually opening them. It's not super advanced and maybe a bit overkill but it's not complicated and works fine for my use case. P.S: I take it another way would be to use a fs that use content-based addressing or does checksumming for me.

werdnapk 4y ago

My backups got attacked by ransomware, but I only caught it about 6-8 months after it occurred. Thankfully, my backup drive is copied to another backup which never deletes files, only copies them, so the renamed files that were encrypted were eventually copied over to my second drive, but the originals remained. The attacker wasn't aware of the second drive.

dec0dedab0de 4y ago

For something like photos you could just store every revision.

jewel 4y ago

Sorry that happened to you. Using a de-duplicating backup solution (such as rdiff-backup or restic), will let you keep daily increments indefinitely with very little overhead.

fps_doug 4y ago

Don't do simple rolling backups, use something with deduplication like borg backup or ZFS/btrfs if you want to do it at the FS level. The backup size should not increase by much more than the actual size of any new files, so if suddenly, you need twice as much backup space because all your files seem to have changed, you should get suspicious.

temp2022account 4y ago
Seconded, I have an external 2tb HDD that stores 1 years worth of daily backups from my 256gb (~180gb used at any time) laptop hard drive. My backup script (https://gist.github.com/Jeffrey-P-McAteer/7d4b9052825914b5e0c500c04251e973 https://gist.github.com/Jeffrey-P-McAteer/7d4b9052825914b5e0...) takes maybe 30 minutes for a full backup, 5 minutes for most deltas. Files which are the same get hard-linked to the previous days backups, new files are copied over and content-de-duped by btrfs.
- 0cf8612b2e1e 4y ago
  If nothing else, external hard drives are cheap and robust enough that I think more people should invest in having an offsite backup. Annually make a full backup, write the date on the outside, and leave it at the parents house. Make that your family holiday ritual.
  - gattr 4y ago
    As of recently, my setup for backups consists of 2x4 TB HDDs (from different manufacturers) with BTRFS in RAID 1, plugged into a small 2-drive USB3 docking station. With both checksumming and mirroring, feels pretty safe from HW-failure/bit rot standpoint (if one disk fails, you can still mount the other in "degraded" mode).
wyager 4y ago
Agreed. I'll shill rsync.net (no affiliation, just a happy customer) and their ZFS VM backup service. It's basically just a lightweight freebsd VM with a big ZFS volume attached, so you can `zfs send` incremental backups to it, and they support meta-snapshotting of your backup machine on their end. I wrote https://github.com/wyager/zfs-backup https://github.com/wyager/zfs-backup to manage my automatic incremental backups, and there are a number of other tools like this.

msm_ 4y ago

> Nearly all my personal photos were encrypted by the helprecover@foxmail.com ("HELP") variant of Phobos. I can't assist you with recovery, and without a lot of logs and forensics data (or a significant performance improvement), the described method is likely unfeasible. But I'll try to find a matching sample and let you know if it's vulnerable. > Sidecar question: when automating your backups, what's a good way to make sure your rolling backups aren't simply backing up malware-encrypted files? Lots of good responses, I like incremental backups without ovewriting anything (supported OOTB by all copy-on-write filesystems, like ZFS or BTRFS). Not sure how to configure this on Windows.

bauruine 4y ago

I use borgbackup to a server and once a month upload it to backblaze b2 with a 90 day retention policy on the bucket. The policy doesn't let you modify or delete the files for 90 days even if you have access to the account. Costs something like 5$ for the 3x500GB of backups I have.

GordonS 4y ago

> when automating your backups, what's a good way to make sure your rolling backups aren't simply backing up malware-encrypted files? Maybe you could check the level of entropy (measure of randomness) of files before backing up - very high entropy could suggest encrypted data?

KMag 4y ago
This is good. Some antivirus programs run this check, but some ransomware adapted by encrypting 16-byte AES blocks every so often in the file, so that the file becomes useless without entropy increasing too much. Also, JPEG, PNG, .jar, .xlsx, etc. are already compressed, so pretty high entropy to begin with. As others have pointed out, the growth rate of your de-duplicated backup size is probably the best way to detect ransomware.

proactivesvcs 4y ago

I use restic which has very flexible retention policies. Rather than retain simply "x days", one can have it "keep the last x hourlies" all the way up to "keep the last x annuals", and XOR many criteria in one policy. Since it uses snapshotting, compression and deduplication it's very storage-efficient. They also offer a simple REST server which allows one to backup with append-only permissions, which can alleviate a ransomware attack.

powersnail 4y ago

I use rsync manually, and always with "--dry-run" first. Unless the ransome-ware is smart enough to rot my backup very, very, very slowly, I should be able to detect any problem by simply reading which files are to be overwritten. 99.99% of the files I back up rarely change. I also have most of my non-sensitive data on Onedrive, which keeps old versions of files.

macjohnmcc 4y ago

I worked in backup software for 22 years until a few years ago. My suggestion is to make a backup and store it for long term on an external drive. The to the cloud full followed incremental backup daily so you can go back to any point along the way at any time. Currently I use Acronis and it works well. I backup to their cloud to avoid having my local backups also encrypted preventing restoring in the event of failure or malware. Good backup strategy would not overwrite old backups until you must and backup regularly, so you don't lose any more data than you can afford to lose. Remember recovery time as well.