5 ms·

> while it works, simpler solutions also provide the same level of reliability. It exists and is important because many backup strategies are broken and people

> while it works, simpler solutions also provide the same level of reliability. It exists and is important because many backup strategies are broken and people don't realize it. For example your own strategy treats a PC's local storage and an external drive as distinct backups when in reality you've only evaluated hardware failures when formulating it and not malicious actors. In 3-2-1, in particular the media type + off-site thing, tries to "trick" you into having a backup which isn't accessible from the same system that it is backing up (i.e. offline backups). Your backup strategy has been used almost verbatim by multiple institutions who got cryto-locked. The external drive was hit and then the cloud backup service happily synced the now encrypted files. They went from "backed up in three places" to backed up in zero places, and are now calling the cloud provider hoping that their backups had unencrypted copies in them. 3-2-1 isn't simple, but it is good, and that's what it tries to be.

toastedwedge 5y ago
Backup versioning is the important distinction here. A backup on its own is only good against loss, but a backup with multiple snapshots in varying states in place protects against the threat of crypto-locked cloud synchronization. Of course if the provider is locked then it's a moot point.
- Someone1234 5y ago
  Some backup versioning schemes are disabled on purpose by some cryto-malware.
gregsadetsky 5y ago
In my DIY backup setups, I'm relying on rsync.net's snapshots [0] and also the fact that the AWS S3 bucket I'm automatically copying stuff to has 'versioning' enabled. [1] Is that good enough in your point of view? Thanks [0] https://www.rsync.net/resources/howto/snapshots.html https://www.rsync.net/resources/howto/snapshots.html (not affiliated, just a happy customer) [1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versio... (ditto)
- slownews45 5y ago
  For automated ransomewhere this works. For targeted - it often doesn't because often your AWS keys are on the system doing the back up and have permissions to delete items etc. Of course, this is why S3 allows you to set and object lock rule (ie, 30 days is plenty) that means even you (or your computer) can't go and delete those online backups.
  - gregsadetsky 5y ago
    That's great to hear about the object lock rule, thanks a lot. I'll add that to my S3 setups.
    - titaniczero 5y ago
      Also, if this were for a company I'd suggest to test the backup at least once during the object lock duration: e.g. if the object lock is set to 30 days, you have to test them at least once (e.g.: day 15). That's because a targeted attack with a ransomware could gain access to your servers and wait 30 days while silently encrypting your backups until the 30th day, when the attackers could just complete the attack encrypting the rest of the files and showing the message. So my recommendation for extremely critical data would be: 1) Test whole data thoroughly at least once during the object lock period. 2) Setup an automatic task that retrieves X random data every day (or the longest period of time you can afford to lose it) and perform checks with checksums or other methods. If something is corrupt and/or encrypted you will realize before it is too late.
      - gregsadetsky 5y ago
        Thanks a lot! I appreciate the insight
  - rsync 5y ago
    "For targeted - it often doesn't because often your AWS keys are on the system doing the back up and have permissions to delete items etc." Just a reminder - snapshots at rsync.net are immutable.[1] Even if Mallory gains access to all of your access credentials, she cannot delete/change the snapshots in your rsync.net account. [1] https://twitter.com/rsyncnet/status/1453044746213990405 https://twitter.com/rsyncnet/status/1453044746213990405
CGamesPlay 5y ago
What you're saying is that a critical piece of the 3-2-1 rule is a piece that isn't actually prescribed by the 3-2-1 rule: that you have "a backup which isn't accessible from the same system that it is backing up (i.e. offline backups)." Another reason that I don't think this "rule of thumb" as useful as it purports to be. Presumably the rule was invented before ransomware was a thing, so perhaps it gets a pass for not anticipating versions, but yeah: the rule of thumb for the modern world probably includes something about backup versions.
- gerdesj 5y ago
  It should be known as the "3-2-1 rule with knobs on" for that precise reason, not to mention several others. However, you need to start somewhere with a pithy pitch. Veeam for example (other backup systems are available) have several "immutable offerings" whereby you get backups that can't be deleted for a while. These are not appliances but configuration recipes. One of them is a generic Linux box with XFS with reflinks, a particular set of file perms and what they call a one off credential (a sort of app password I think), forward incrementals and a few other things. Get it all in line then even if a baddy gets in they can't delete your backups for a while. This does require detection inside say a week. The reflinks thing is really useful for storage space and things like "synthetic fulls" where you roll up incrementals into the last full backup. You can also use AWS etc object storage for immutable. In all of these things you can lengthen the timescales available by applying more cash. For me: tape in a safe is the only decent way to protect against ransomware. Even then some tape hygiene is needed!

toastedwedge 5y ago

Backup versioning is the important distinction here. A backup on its own is only good against loss, but a backup with multiple snapshots in varying states in place protects against the threat of crypto-locked cloud synchronization. Of course if the provider is locked then it's a moot point.

Someone1234 5y ago
Some backup versioning schemes are disabled on purpose by some cryto-malware.

gregsadetsky 5y ago

In my DIY backup setups, I'm relying on rsync.net's snapshots [0] and also the fact that the AWS S3 bucket I'm automatically copying stuff to has 'versioning' enabled. [1] Is that good enough in your point of view? Thanks [0] https://www.rsync.net/resources/howto/snapshots.html https://www.rsync.net/resources/howto/snapshots.html (not affiliated, just a happy customer) [1] https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versio... (ditto)

slownews45 5y ago
For automated ransomewhere this works. For targeted - it often doesn't because often your AWS keys are on the system doing the back up and have permissions to delete items etc. Of course, this is why S3 allows you to set and object lock rule (ie, 30 days is plenty) that means even you (or your computer) can't go and delete those online backups.
- gregsadetsky 5y ago
  That's great to hear about the object lock rule, thanks a lot. I'll add that to my S3 setups.
  - titaniczero 5y ago
    Also, if this were for a company I'd suggest to test the backup at least once during the object lock duration: e.g. if the object lock is set to 30 days, you have to test them at least once (e.g.: day 15). That's because a targeted attack with a ransomware could gain access to your servers and wait 30 days while silently encrypting your backups until the 30th day, when the attackers could just complete the attack encrypting the rest of the files and showing the message. So my recommendation for extremely critical data would be: 1) Test whole data thoroughly at least once during the object lock period. 2) Setup an automatic task that retrieves X random data every day (or the longest period of time you can afford to lose it) and perform checks with checksums or other methods. If something is corrupt and/or encrypted you will realize before it is too late.
    - gregsadetsky 5y ago
      Thanks a lot! I appreciate the insight
- rsync 5y ago
  "For targeted - it often doesn't because often your AWS keys are on the system doing the back up and have permissions to delete items etc." Just a reminder - snapshots at rsync.net are immutable.[1] Even if Mallory gains access to all of your access credentials, she cannot delete/change the snapshots in your rsync.net account. [1] https://twitter.com/rsyncnet/status/1453044746213990405 https://twitter.com/rsyncnet/status/1453044746213990405

CGamesPlay 5y ago

What you're saying is that a critical piece of the 3-2-1 rule is a piece that isn't actually prescribed by the 3-2-1 rule: that you have "a backup which isn't accessible from the same system that it is backing up (i.e. offline backups)." Another reason that I don't think this "rule of thumb" as useful as it purports to be. Presumably the rule was invented before ransomware was a thing, so perhaps it gets a pass for not anticipating versions, but yeah: the rule of thumb for the modern world probably includes something about backup versions.

gerdesj 5y ago
It should be known as the "3-2-1 rule with knobs on" for that precise reason, not to mention several others. However, you need to start somewhere with a pithy pitch. Veeam for example (other backup systems are available) have several "immutable offerings" whereby you get backups that can't be deleted for a while. These are not appliances but configuration recipes. One of them is a generic Linux box with XFS with reflinks, a particular set of file perms and what they call a one off credential (a sort of app password I think), forward incrementals and a few other things. Get it all in line then even if a baddy gets in they can't delete your backups for a while. This does require detection inside say a week. The reflinks thing is really useful for storage space and things like "synthetic fulls" where you roll up incrementals into the last full backup. You can also use AWS etc object storage for immutable. In all of these things you can lengthen the timescales available by applying more cash. For me: tape in a safe is the only decent way to protect against ransomware. Even then some tape hygiene is needed!