8 ms·

This is a question I've thought alot about, so many some Sys Admins can give a good idea about how to approach it: How do you create a backup server that is re

This is a question I've thought alot about, so many some Sys Admins can give a good idea about how to approach it: How do you create a backup server that is reachable by production servers (so that they can back up to it) without then being vulnerable to the same kind of ransomware attacks that infect the production servers? You can't exactly make them read-only, or else they can't accept the "legitimate" writes that might occur during the normal backup process.

computer 7y ago
Make the backup servers pull?
- PowerfulWizard 7y ago
  That is what I do. The backup server can access the production server, but production can't access the backup. But my setup requires the backup to go to disk on the production server and not stream out directly, so that aspect is annoying.
donmcronald 7y ago
Append only?
- adrianmsmith 7y ago
  Right, that's what I use, https://www.tarsnap.com/ https://www.tarsnap.com/ has the option of allowing an access key to only be able to create a new backup but not read or alter or delete any existing backups. I blogged about my setup here: https://www.databasesandlife.com/write-only-backups/ https://www.databasesandlife.com/write-only-backups/
deleted 7y ago
[deleted]
deleted 7y ago
[deleted]
tomschlick 7y ago
Two separate S3 buckets with different credentials. The server pushes its own backups to the "quarantine" bucket and later a process moves the backup to the long term storage bucket.
- dev_dull 7y ago
  I’m surprised this answer is so far down. Production machines write to an S3 bucket with a lambda trigger. As soon as the upload is finished, the lambda either lifts the file to another location or changes the acl so the production machines can no longer access it.
- luhn 7y ago
  I use a single bucket with versioning enabled. Production server has access to write as it pleases, but isn't allowed to touch past versions of a file.
  - eropple 7y ago
    S3 now allows permanent object locking, too, which makes this even harder to screw up.
  - brazzledazzle 7y ago
    I think having the bucket in a different account with different root/MFA might make sense here too. Your production AWS account is more likely to get owned than your account dedicated to backups.
    - luhn 7y ago
      Yes! The bucket is located in a different account with separate billing and very limited on who has root/admin access.
- reaperducer 7y ago
  Two separate S3 buckets with different credentials This gives me the heebie-jeebies. It seems better to me to have one on Amazon, and one on another cloud provider entirely. Plus regular sneakernet archives, if your data is of a size that would permit that sort of thing. I'm fortunate that this kind of thing is handled by other people where I work. It's my understanding that we have production data on one service on the east coast, a mirror with a different company in the south, and weekly backup archives stored offline on the west coast.
luma 7y ago
The easiest solution to this is out-of-band backups happening at the storage layer. The details depend heavily upon your platform and storage hardware, but most SAN and virtualization solutions today allow one to backup hosted systems without being accessible by those systems (and no agents installed into the guest OS).
srfilipek 7y ago
I use FreeNAS (FreeBSD NAS with ZFS). The ZFS snapshotting feature effectively gives me protections from this type of issue. My home computers push backups regularly throughout the day, and every day I create snapshots of each volume (how long to keep the snapshots is another question). This snapshot can only be accessed or managed on the NAS itself. This effectively creates an append-only backup NAS thanks to the periodic snapshots.
- geoka9 7y ago
  Do those snapshots somehow benefit from the incremental nature of daily backups?
  - corwin7 7y ago
    Yes, ZFS snapshots only contain data that changed. ZFS is very efficient, elegant, and reliable.
    - geoka9 7y ago
      Even with ZFS deduplication off?
      - srfilipek 7y ago
        Yes.
        CrazyMusicians 7y ago
        These are all good ideas. It seems to me part of the problem with these ransom attacks is the intruders aren't pouncing the minute they get inside the victim's network. They take their time, figure out where everything is (including backups) and then only force-spread the ransom malware after they have staged everything just-so.
    - jacobgreenleaf 7y ago
      To be more precise, ZFS snapshots are mostly just a timestamp reference, similar to how a git commit is just a reference to a git object. Writes don't change the value of the files in the snapshot because they're stored in new places rather than overwriting the existing files on disk. https://blogs.oracle.com/ahrens/is-it-magic https://blogs.oracle.com/ahrens/is-it-magic
benjohnson 7y ago
ZFS Snapshots are really handy - it takes root access to remove them and you can't delete them with the normal UNIX delete. IF you're sharing with Samba or NFS, there's no way for any remote user to be able to delete. You just go back to the last good version.
fuzz4lyfe 7y ago
Drawbridge, you configure a firewall to allow access only at certain times. Also you can configure a NAS so that deleted items are preserved without a admin removing them.
johngalt 7y ago
A generally safe backup process looks like this: Production has no access to backup. Backup has read only access to production. Backup writes are append and not overwrites. Deletes/archival are governed by a retention process.
- Damogran6 7y ago
  Monitor dedupe statistics. If they go pear shaped, Copy N is much different than Copy N+1.
  - lifeisstillgood 7y ago
    By good and easy to do point- something as simple as size will help but a diff between backups would tell much - i had not thought of that - then again it comes for free in something like tarsnap
- pstuart 7y ago
  A solid list. It's nice to have a restore step in there too, so one has both a validation that the backup is usable, and gives a "playground" where one can have a safe day old environment for testing/training/whatnot against production data.
  - __jal 7y ago
    I don't consider that nice-to-have. That is essential. If you do not regularly test restoring, you do not know if you have backups. I normally overstate this just a little for effect, "If you don't test your backups you don't have backups", but that isn't precisely true. And yes, it frequently makes sense to base test/fixture data and whatnot on backup data flows, but unless you don't deal with any PII, you're hopefully not using that data without a sanitization step.
    - JustSomeNobody 7y ago
      Do you restore to prod? Or to Non-Prod and compare to prod?
dmoy 7y ago
Backup could (should?) be on tape, which once written is physically removed from the system and put in cold storage for some period of time (years?). That's how we did it at my first job, at least. Of course IANAL so idk how this jives with various EU laws.
perlgeek 7y ago
You can write backups to an NFS share, and create regular snapshots on the file system server that are not writable from the outside (you have to log into the file system server, which hopefully is extra secured / limited access, and has a different OS than what you use elsewhere). It's not perfect, but at least a single vulnerable client can't encrypt or delete your backups (in the snapshots) that way. As usually with security, the principles of least privs and segregating as much as possible are important.
0xEFF 7y ago
In the past I had the backup server ssh via public key into the production server and pull the data via file system snapshots. It’d be a matter of restoring the most recent snapshot prior to the attack.
jacquesm 7y ago
You pull from the production servers rather than push to the backup servers. That way even if the production servers are compromised your backups are still safe.
mc32 7y ago
Also how do you prove integrity of your backups if you don’t have certainty of initial perhaps temporarily dormant breach? I mean your backups to tape could still have bad data.
MrStonedOne 7y ago
Windows strategy: one way dfs replication to onsite backup server hooked up to an offsite cloud backup provider.
syn0byte 7y ago
Unless you have persistent mount to network FSs like NFS or SMB how do you think the ransomware would spread? You sure don't need network mounts for backups. Cronjob to an (S)FTP server and an upload script trigger to chown/chmod all incoming files making the whole thing WORM (Write Once Read Many). Once its submitted the same user account can't alter it. Even if the malware is clever and scans for .netrc and .id_rsa and manages to create its own connection to the backup server it doesn't have access to anything anyway.
MrGilbert 7y ago
I'm not a sysadmin, and I'm pretty sure that <big company> does it different, but here is how I do it at home: - I have a UnRaid machine, and a backup machine. The backup machine is a small itx board, and has a single HDD attached. - A NodeRED instance has a so-called "Flow" on the UnRaid machine that is waking up the backup machine every 7 days. - Thanks to anachron, with a 10 minute delay, rsnapshot connects to the UnRaid machine, pulls the data, and then issues a shutdown to the backup machine. This setup let me sleep pretty well.
theamk 7y ago
Make S3 bucket on a different account, grant it write privileges from your main one. Enable versioning and setup lifecycle to purge old versions in 90 cays. That’s it - no matter what your main account does, you will bave 3 months to undo it
jabart 7y ago
Most attacks are done via local file access. Ship things over an API. EC2 -> S3 bucket with only write access and versioning enabled. EC2 -> EFS and it's a rotating set of 7 with 7 different security groups that rotate.
merb 7y ago
you can use s3/gcloud files with "create"/"read" only rights.

computer 7y ago

Make the backup servers pull?

PowerfulWizard 7y ago
That is what I do. The backup server can access the production server, but production can't access the backup. But my setup requires the backup to go to disk on the production server and not stream out directly, so that aspect is annoying.

donmcronald 7y ago

Append only?

adrianmsmith 7y ago
Right, that's what I use, https://www.tarsnap.com/ https://www.tarsnap.com/ has the option of allowing an access key to only be able to create a new backup but not read or alter or delete any existing backups. I blogged about my setup here: https://www.databasesandlife.com/write-only-backups/ https://www.databasesandlife.com/write-only-backups/

deleted 7y ago

[deleted]

deleted 7y ago

[deleted]

tomschlick 7y ago

Two separate S3 buckets with different credentials. The server pushes its own backups to the "quarantine" bucket and later a process moves the backup to the long term storage bucket.

dev_dull 7y ago
I’m surprised this answer is so far down. Production machines write to an S3 bucket with a lambda trigger. As soon as the upload is finished, the lambda either lifts the file to another location or changes the acl so the production machines can no longer access it.
luhn 7y ago
I use a single bucket with versioning enabled. Production server has access to write as it pleases, but isn't allowed to touch past versions of a file.
- eropple 7y ago
  S3 now allows permanent object locking, too, which makes this even harder to screw up.
- brazzledazzle 7y ago
  I think having the bucket in a different account with different root/MFA might make sense here too. Your production AWS account is more likely to get owned than your account dedicated to backups.
  - luhn 7y ago
    Yes! The bucket is located in a different account with separate billing and very limited on who has root/admin access.
reaperducer 7y ago
Two separate S3 buckets with different credentials This gives me the heebie-jeebies. It seems better to me to have one on Amazon, and one on another cloud provider entirely. Plus regular sneakernet archives, if your data is of a size that would permit that sort of thing. I'm fortunate that this kind of thing is handled by other people where I work. It's my understanding that we have production data on one service on the east coast, a mirror with a different company in the south, and weekly backup archives stored offline on the west coast.

luma 7y ago

The easiest solution to this is out-of-band backups happening at the storage layer. The details depend heavily upon your platform and storage hardware, but most SAN and virtualization solutions today allow one to backup hosted systems without being accessible by those systems (and no agents installed into the guest OS).

srfilipek 7y ago

I use FreeNAS (FreeBSD NAS with ZFS). The ZFS snapshotting feature effectively gives me protections from this type of issue. My home computers push backups regularly throughout the day, and every day I create snapshots of each volume (how long to keep the snapshots is another question). This snapshot can only be accessed or managed on the NAS itself. This effectively creates an append-only backup NAS thanks to the periodic snapshots.

geoka9 7y ago
Do those snapshots somehow benefit from the incremental nature of daily backups?
- corwin7 7y ago
  Yes, ZFS snapshots only contain data that changed. ZFS is very efficient, elegant, and reliable.
  - geoka9 7y ago
    Even with ZFS deduplication off?
    - srfilipek 7y ago
      Yes.
      - CrazyMusicians 7y ago
        These are all good ideas. It seems to me part of the problem with these ransom attacks is the intruders aren't pouncing the minute they get inside the victim's network. They take their time, figure out where everything is (including backups) and then only force-spread the ransom malware after they have staged everything just-so.
  - jacobgreenleaf 7y ago
    To be more precise, ZFS snapshots are mostly just a timestamp reference, similar to how a git commit is just a reference to a git object. Writes don't change the value of the files in the snapshot because they're stored in new places rather than overwriting the existing files on disk. https://blogs.oracle.com/ahrens/is-it-magic https://blogs.oracle.com/ahrens/is-it-magic

benjohnson 7y ago

ZFS Snapshots are really handy - it takes root access to remove them and you can't delete them with the normal UNIX delete. IF you're sharing with Samba or NFS, there's no way for any remote user to be able to delete. You just go back to the last good version.

fuzz4lyfe 7y ago

Drawbridge, you configure a firewall to allow access only at certain times. Also you can configure a NAS so that deleted items are preserved without a admin removing them.

johngalt 7y ago

A generally safe backup process looks like this: Production has no access to backup. Backup has read only access to production. Backup writes are append and not overwrites. Deletes/archival are governed by a retention process.

Damogran6 7y ago
Monitor dedupe statistics. If they go pear shaped, Copy N is much different than Copy N+1.
- lifeisstillgood 7y ago
  By good and easy to do point- something as simple as size will help but a diff between backups would tell much - i had not thought of that - then again it comes for free in something like tarsnap
pstuart 7y ago
A solid list. It's nice to have a restore step in there too, so one has both a validation that the backup is usable, and gives a "playground" where one can have a safe day old environment for testing/training/whatnot against production data.
- __jal 7y ago
  I don't consider that nice-to-have. That is essential. If you do not regularly test restoring, you do not know if you have backups. I normally overstate this just a little for effect, "If you don't test your backups you don't have backups", but that isn't precisely true. And yes, it frequently makes sense to base test/fixture data and whatnot on backup data flows, but unless you don't deal with any PII, you're hopefully not using that data without a sanitization step.
  - JustSomeNobody 7y ago
    Do you restore to prod? Or to Non-Prod and compare to prod?

dmoy 7y ago

Backup could (should?) be on tape, which once written is physically removed from the system and put in cold storage for some period of time (years?). That's how we did it at my first job, at least. Of course IANAL so idk how this jives with various EU laws.

perlgeek 7y ago

You can write backups to an NFS share, and create regular snapshots on the file system server that are not writable from the outside (you have to log into the file system server, which hopefully is extra secured / limited access, and has a different OS than what you use elsewhere). It's not perfect, but at least a single vulnerable client can't encrypt or delete your backups (in the snapshots) that way. As usually with security, the principles of least privs and segregating as much as possible are important.

0xEFF 7y ago

In the past I had the backup server ssh via public key into the production server and pull the data via file system snapshots. It’d be a matter of restoring the most recent snapshot prior to the attack.

jacquesm 7y ago

You pull from the production servers rather than push to the backup servers. That way even if the production servers are compromised your backups are still safe.

mc32 7y ago

Also how do you prove integrity of your backups if you don’t have certainty of initial perhaps temporarily dormant breach? I mean your backups to tape could still have bad data.

MrStonedOne 7y ago

Windows strategy: one way dfs replication to onsite backup server hooked up to an offsite cloud backup provider.

syn0byte 7y ago

Unless you have persistent mount to network FSs like NFS or SMB how do you think the ransomware would spread? You sure don't need network mounts for backups. Cronjob to an (S)FTP server and an upload script trigger to chown/chmod all incoming files making the whole thing WORM (Write Once Read Many). Once its submitted the same user account can't alter it. Even if the malware is clever and scans for .netrc and .id_rsa and manages to create its own connection to the backup server it doesn't have access to anything anyway.

MrGilbert 7y ago

I'm not a sysadmin, and I'm pretty sure that <big company> does it different, but here is how I do it at home: - I have a UnRaid machine, and a backup machine. The backup machine is a small itx board, and has a single HDD attached. - A NodeRED instance has a so-called "Flow" on the UnRaid machine that is waking up the backup machine every 7 days. - Thanks to anachron, with a 10 minute delay, rsnapshot connects to the UnRaid machine, pulls the data, and then issues a shutdown to the backup machine. This setup let me sleep pretty well.

theamk 7y ago

Make S3 bucket on a different account, grant it write privileges from your main one. Enable versioning and setup lifecycle to purge old versions in 90 cays. That’s it - no matter what your main account does, you will bave 3 months to undo it

jabart 7y ago

Most attacks are done via local file access. Ship things over an API. EC2 -> S3 bucket with only write access and versioning enabled. EC2 -> EFS and it's a rotating set of 7 with 7 different security groups that rotate.

merb 7y ago

you can use s3/gcloud files with "create"/"read" only rights.