10 ms·

Ransomware-resistant backups with duplicity and AWS S3

87 points by alanfranz 5y ago

Proven 5y ago
hughrr 5y ago
I would never use this after being burned badly. Duplicity hits a scalability brick wall on large file volumes which consumes ridiculous amounts of CPU and RAM on the host machine and leads to irrecoverable failure where it can’t backup or restore anything. Fortunately I caught this before we had a DR scenario. I am using rdiff-backup over SSH to replace it now. This has been reliable so far but recovery times are extensive.
- czl 5y ago
  Assuming zfs is reasonable for the usecase: incremental zfs snapshots are likely efficient since these are byte level vs file level.
  - hughrr 5y ago
    Depends on the recovery cost but yes I agree they are probably a better solution.
- rsync 5y ago
  "I would never use this after being burned badly. Duplicity hits a scalability brick wall on large file volumes which consumes ridiculous amounts of CPU and RAM on the host machine and leads to irrecoverable failure where it can’t backup or restore anything." I believe you are correct and I believe that in my private correspondence with the duplicity maintainer (we sometimes sponsor duplicity development[1]) he sort of conceded that borg backup[2] is a better solution. If the cloud platform you point your borg backups to can configure immutable snapshots (that is, they create, rotate, and destroy them) then a good solution would be using borg backup over SSH and configuring some of those snapshots[3]. [1] https://www.rsync.net/resources/notices/2007cb.html https://www.rsync.net/resources/notices/2007cb.html [2] https://www.stavros.io/posts/holy-grail-backups/ https://www.stavros.io/posts/holy-grail-backups/ [3] https://twitter.com/rsyncnet/status/1453044746213990405 https://twitter.com/rsyncnet/status/1453044746213990405
- nonbirithm 5y ago
  Duplicity also can't be run natively on Windows, so I was stuck having to migrate all of my backups to another program (Duplicati) once I changed operating systems.
- alanfranz 5y ago
  Do you have an idea of how "large" it is? I used it for a previous server of mine (about 250GB/300GB), I did daily backups and multiple restores over the years, they would always succeed, even though, admittedly, I didn't care about speed.
  - jcrawfordor 5y ago
    I found Duplicity to be untenable for a volume of about 2TB on a machine with relatively low resources (4GB of RAM). It would consistently fail due to OOM before completing the job. This is an unfortunately common issue with backup tools, which it seems get surprisingly little testing on multi-TB tasks. I'd guesstimate maybe half of open source backup tools I've tried either seriously struggle or reliably fail to complete an 8TB job I have on another machine, with similar memory constraints. So far Rustic has been my best bet. It seems to complete very large jobs with static and relatively low memory consumption. Enumeration takes about 30 minutes and a full backup nearly a month on the 8TB job (limited bandwidth available) but it reliably completes. Rustic also resumes interrupted jobs (e.g. due to reboot) in around 10-15 minutes on the 8TB volume and with it seems minimal rework. I'm sure there are other tools that handle this as well, but I've definitely gotten frustrated with finding them. I wish more backup tools gave you some kind of assurance in the marketing materials that they've been validated on, say, 10TB.
    - alanfranz 5y ago
      > I wish more backup tools gave you some kind of assurance in the marketing materials that they've been validated on, say, 10TB. I'd wish that too. Also, I'd like to pay for a backup tool, since it's so critical, so that I can get a sort of support, but I have found issues with many of them. I must say that, with my small population (N=1) for testing, it was hard for me to settle on a backup system. I tried duplicity, then I tried borg, then I used duplicati. I had considered attic and restic as well, but I don't remember why I didn't choose them right now. I experienced issues with most of them and I reverted to duplicity. With borg, I had a persistent issues where backups would stop and said something like "backup destination is newer than source" (I forgot the exact message, but happened multiple times across versions). Duplicati seemed a very large and complex codebase, and periodically stopped working, mostly because dotnet runtime issues AFAICR.
justin_oaks 5y ago
AWS S3 object versioning makes it pretty easy to allow a server to add backup data without the ability to permanently modify or delete data that existed previously. For my backups I use restic and sync the restic repository data to S3. Even if the source data is corrupted, I can always roll back to the set of S3 object versions from a particular time. The downside to using S3 object versioning is that I haven't found any good tools to work with object versions across multiple objects. For example, I need to prune old backups from my restic repository. To do that I have to delete object versions that are no longer current (i.e. the latest version of an object). To accomplish this I had to write a script that uses Boto3 (AWS SDK for Python) that lets me delete non-current object versions and deletes any delete-marker versions. The code was pretty straightforward, but I wish there was a tool that made it easier to do that.
- dageshi 5y ago
  Not 100% sure but can't you setup rules for that stuff in s3 itself? Like delete everything older than x date but never delete current version? I have a feeling I set something like this up but it's been a while since I did it.
  - nijave 5y ago
    Yeah you can use lifecycle policies https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/object...
  - andrewguenther 5y ago
    You are correct, S3 lifecycle policies support deleting non-current object versions once they're X days old as well as X versions old: https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configuration-examples.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecy...
  - justin_oaks 5y ago
    You can, but automatically doing it would defeat the whole purpose of having versions I could roll back to. I need to confirm that that my recent backups are valid before deleting old backups. Otherwise I might as well not have used S3 versioning.
deleted 5y ago
[deleted]
czl 5y ago
Having point in time backups is a good start but I can see ransom-ware adopt to slowly corrupt your data in a way that is reversible but may take months to detect. Your backups going back months will then have this corruption. To detect this application level tripwires may be needed like checksums etc. Finally there is always reputation damage and threats to expose the attack an your data to the public via blackmail. Just because you have backups does not make you safe.
- gjs278 5y ago
- EvanAnderson 5y ago
  Application "tripwires" are just another obstacle for an attacker to overcome. If audits aren't external to the system being audited they're just as vulnerable to manipulation. A Customer of mine in the financial sector sent their backups to a third-party for independent verification quarterly. The third-party restored data into a clean system reconciled against the production system. That would be the kind of auditing that would be more apt to detect the "low and slow" attack.
andrewguenther 5y ago
At a minimum, you should include enabling MFA for the IAM user. Generally, I'd suggest against using IAM users entirely. Ideally you would use an IAM Role via federation or SSO. For my personal accounts I use AWS SSO even though I'm just one person since it enables me to do all my work through role-based authentication and is still protected by MFA on top.
- alanfranz 5y ago
  Uhm, I don't understand how this would work. We need the backups to run unattended. The IAM user I configure has got no console access at all. Do you have any pointer or example?
  - andrewguenther 5y ago
    That's true. You could still do IAM role authentication via web identity which would be much safer than a user and could still be automated.
nijave 5y ago
Seems easier to just do incremental snapshots unless you're on bare metal. Many hypervisors support them and they're built into EC2/EBS If you want to limit data, you can create additional drives and mount them at the appropriate location (or change your application config to save to the auxiliary drives)
ignoramous 5y ago
Incredible that folks have to jump through so many hoops. At this point, AWS should offer versioned S3 backups accessible only offline or through customer support, enabled with a click of a button.
- different_sort 5y ago
  AWS Backup does exist.
  - ignoramous 5y ago
    I knew about AWS Backup but I previously never saw an option to enable it for S3. It looks like it is in limited (1TB) preview in the Oregon region, and doesn't support backing up buckets encrypted with client-provided keys. That said, AWS Backup is the answer to ransomware woes and it can't GA soon enough (3P solutions like Rubrik, Druva notwithstanding) https://docs.aws.amazon.com/aws-backup/latest/devguide/s3-backups.html https://docs.aws.amazon.com/aws-backup/latest/devguide/s3-ba...
rmbyrro 5y ago
> you'll need to make sure that your master access to AWS S3 is never compromised Your master access to S3 should never go into your servers. Create an IAM access with authorization to only PUT objects into S3. > For the purpose we have, Governance mode is OK Maybe not (?), since Governance mode allows for deletion of previous versions. One careless mistake handling your access key/secret and you're exposed to bye bye backups. End note: this is still not enough. An attacker could compromise your backup script and wait for 40 days before locking yourself out of your data. When you try to recover a backup, you'll notice you have none. Perhaps most attackers won't have the patience and will just forget about you, but who knows? A naive protection from that would be to store at least one version of the backups forever. But we're still not covered, since the attacker could compromise that particular version and boom. I can't think of a comprehensive, fully closed-loop solution right now...
- bradknowles 5y ago
  There is an immutable backup solution to S3: https://www.retrospect.com/en/support/kb/immutable_backups_on_amazon_s3 https://www.retrospect.com/en/support/kb/immutable_backups_o...
- rsync 5y ago
  "I can't think of a comprehensive, fully closed-loop solution right now..." The ZFS snapshots that you may configure on an rsync.net account are immutable. There are no credentials that can be presented to destroy them or alter their rotation other than our account manager whose changes always pass by a set of human eyes. Which is to say, no 'zfs' commands are ever automated. So the short answer is you simply point borg backup to rsync.net and configure some days/weeks/months of snapshots. The long answer - if you're interested: https://twitter.com/rsyncnet/status/1470669611200770048 https://twitter.com/rsyncnet/status/1470669611200770048 ... skip to 52:20 for "how to destroy an rsync.net account": "... Another thing that lights up big and red on our screen is ... someone's got a big schedule of snapshots ... and then they change it to zero ... you've got seven days and four weeks and six months but we want to change that to zero days of snapshots. We see those things ... and we reach out to people."
  - 5y ago
antaviana 5y ago
One good alternative is to upload to an S3 bucket with object lock enabled. This way you can store immutable objects. You can make them immutable for everyone if you wish and the only way to delete them is to close the AWS account. I cannot think of a safest place for a backup than a bucket with object lock.
- WatchDog 5y ago
  The article is about object lock
  - mlac 5y ago
    One good alternative to reading the articles is just skimming the comments.
    - ignoramous 5y ago
      Yet another alternative is to misread the title.
- momothereal 5y ago
  Can you still configure object retention when they are locked? i.e. automatically delete objects after X days.
joe_the_user 5y ago
As I understand it, the problem enterprises hit isn't the lack of backups or that backups are compromised. The problem seems to be that the restore process has stopped being scalable - restore of all enterprise data can take weeks or longer, and so paying the hacker is cheaper than doing the restore even though the restore would work.
- justin_oaks 5y ago
  Some companies I've worked for have "disaster recovery" drills periodically. Those should involve restoring backups. From that you should have a timeline of how long it takes to perform a restore. If it takes weeks to perform a restore then that should be a major red flag and be addressed. Of course, if a company has the kind of security that allows them to be hit by ransomware, then they probably aren't diligent about their disaster recovery scenarios.
  - spydum 5y ago
    obvious problem here is that most dont perform dr test across all systems simultaneously - they do it by major component, and usually have a working system.. ransomware can take down ALL of that, so you are left with bare metal recovery, and no infra to build off of.
- travem 5y ago
  That depends, with modern enterprise backup software it can be faster to recover that way than rely on the (sometimes slow) decryption software. This was the case with Colonial Pipeline for example, https://www.theregister.com/2021/05/13/colonial_pipeline_ransom/ https://www.theregister.com/2021/05/13/colonial_pipeline_ran...
WatchDog 5y ago
Object lock helps tick some regulatory boxes, but at least in the described scenario, enabling it isn't really going to protect you against many additional threat models. Simply using object versioning, and an IAM role with no delete permission is sufficient. The biggest risk with using AWS for backups, is being compromised via a high-privilege IAM role. If a role has s3:* permission, it can just remove the locks and delete the file.
- justin_oaks 5y ago
  Indeed. It'd be best to have another copy of the backup somewhere other than S3, or in S3 under a different AWS account. I worked for a company that had all their databases in AWS RDS and all their database backups as RDS backups stored in the same AWS account. A compromise of a single high-privilege account could cause that company to no longer have ANY data. I warned the higher-ups repeatedly that they needed more than one backup location. They didn't seem to care too much.
  - alanfranz 5y ago
    Yes. Cross account replication is the way to go for better protection.
ghostoftesla 5y ago
ddtaylor 5y ago
I use duplicity (via the DejaDup GUI) and I really like it. However, there is one thing I think most people need to be aware of: Duplicity does NOT encrypt the names of your files. Some might not care about this much, but for me I really don't like my encrypted backups containing so much metadata. EDIT: FWIW I don't use the AWS backup options. I have separate offsite backup connections to cloud services as well as NAS.
- aborsy 5y ago
  Any half-ass backup software encrypts the file names in the repository, and it seems duplicity does too.
  - _zooted 5y ago
    It actually does not as you can see all of the file names before you enter your encryption password.
    - aborsy 5y ago
      That’s most likely local cache (it takes file names from your local disk)!! The file names and content are encrypted. No decent backup software will leave file names in plaintext.
- alanfranz 5y ago
  Author here. > Duplicity does NOT encrypt the names of your files. I, personally, do not care, but you're right. For the purpose of this article I could even leave encryption off, though. "I don't trust the backup destination" wouldn't be part of my threat model here.
m3nu 5y ago
Short plug: At https://BorgBase.com https://BorgBase.com, we also do append-only backups and every repo is isolated. The majority or repos is append-only at this point.
- alanfranz 5y ago
  Thanks for your plug :-) Your company seems interesting, but could you give us some background? How many people are working there? What are your assurances? I cited rsync.net in my article, and I leverage AWS S3 as an example, because they're well-known players. I would be more hesitant to use a small and unknown (hence potentially unreliable) backup provider.
  - m3nu 5y ago
    BorgBase has been around since 2018 and we also maintain Vorta, a popular desktop backup client (also included in Debian since Buster). In addition we sponsor Borg and Borgmatic to keep the eco system healthy. Most providers don't go as far. So I'm quite proud of that. BorgBase itself doesn't have staff, but runs under Peakford.com, which offers a variety of hosting- and consulting offers. Myself, I mainly take care of our open source offerings and community.
  - AdaX 5y ago
alanfranz 5y ago
Author here. I'd like to thank the HN folks for some great feedback (as usual). I'll update the article in the next hours to add some caveats.

87 points by alanfranz 5y ago

I would never use this after being burned badly. Duplicity hits a scalability brick wall on large file volumes which consumes ridiculous amounts of CPU and RAM on the host machine and leads to irrecoverable failure where it can’t backup or restore anything. Fortunately I caught this before we had a DR scenario. I am using rdiff-backup over SSH to replace it now. This has been reliable so far but recovery times are extensive.

czl 5y ago
Assuming zfs is reasonable for the usecase: incremental zfs snapshots are likely efficient since these are byte level vs file level.
- hughrr 5y ago
  Depends on the recovery cost but yes I agree they are probably a better solution.
rsync 5y ago
"I would never use this after being burned badly. Duplicity hits a scalability brick wall on large file volumes which consumes ridiculous amounts of CPU and RAM on the host machine and leads to irrecoverable failure where it can’t backup or restore anything." I believe you are correct and I believe that in my private correspondence with the duplicity maintainer (we sometimes sponsor duplicity development[1]) he sort of conceded that borg backup[2] is a better solution. If the cloud platform you point your borg backups to can configure immutable snapshots (that is, they create, rotate, and destroy them) then a good solution would be using borg backup over SSH and configuring some of those snapshots[3]. [1] https://www.rsync.net/resources/notices/2007cb.html https://www.rsync.net/resources/notices/2007cb.html [2] https://www.stavros.io/posts/holy-grail-backups/ https://www.stavros.io/posts/holy-grail-backups/ [3] https://twitter.com/rsyncnet/status/1453044746213990405 https://twitter.com/rsyncnet/status/1453044746213990405
nonbirithm 5y ago
Duplicity also can't be run natively on Windows, so I was stuck having to migrate all of my backups to another program (Duplicati) once I changed operating systems.
alanfranz 5y ago
Do you have an idea of how "large" it is? I used it for a previous server of mine (about 250GB/300GB), I did daily backups and multiple restores over the years, they would always succeed, even though, admittedly, I didn't care about speed.
- jcrawfordor 5y ago
  I found Duplicity to be untenable for a volume of about 2TB on a machine with relatively low resources (4GB of RAM). It would consistently fail due to OOM before completing the job. This is an unfortunately common issue with backup tools, which it seems get surprisingly little testing on multi-TB tasks. I'd guesstimate maybe half of open source backup tools I've tried either seriously struggle or reliably fail to complete an 8TB job I have on another machine, with similar memory constraints. So far Rustic has been my best bet. It seems to complete very large jobs with static and relatively low memory consumption. Enumeration takes about 30 minutes and a full backup nearly a month on the 8TB job (limited bandwidth available) but it reliably completes. Rustic also resumes interrupted jobs (e.g. due to reboot) in around 10-15 minutes on the 8TB volume and with it seems minimal rework. I'm sure there are other tools that handle this as well, but I've definitely gotten frustrated with finding them. I wish more backup tools gave you some kind of assurance in the marketing materials that they've been validated on, say, 10TB.
  - alanfranz 5y ago
    > I wish more backup tools gave you some kind of assurance in the marketing materials that they've been validated on, say, 10TB. I'd wish that too. Also, I'd like to pay for a backup tool, since it's so critical, so that I can get a sort of support, but I have found issues with many of them. I must say that, with my small population (N=1) for testing, it was hard for me to settle on a backup system. I tried duplicity, then I tried borg, then I used duplicati. I had considered attic and restic as well, but I don't remember why I didn't choose them right now. I experienced issues with most of them and I reverted to duplicity. With borg, I had a persistent issues where backups would stop and said something like "backup destination is newer than source" (I forgot the exact message, but happened multiple times across versions). Duplicati seemed a very large and complex codebase, and periodically stopped working, mostly because dotnet runtime issues AFAICR.

justin_oaks 5y ago

AWS S3 object versioning makes it pretty easy to allow a server to add backup data without the ability to permanently modify or delete data that existed previously. For my backups I use restic and sync the restic repository data to S3. Even if the source data is corrupted, I can always roll back to the set of S3 object versions from a particular time. The downside to using S3 object versioning is that I haven't found any good tools to work with object versions across multiple objects. For example, I need to prune old backups from my restic repository. To do that I have to delete object versions that are no longer current (i.e. the latest version of an object). To accomplish this I had to write a script that uses Boto3 (AWS SDK for Python) that lets me delete non-current object versions and deletes any delete-marker versions. The code was pretty straightforward, but I wish there was a tool that made it easier to do that.

dageshi 5y ago
Not 100% sure but can't you setup rules for that stuff in s3 itself? Like delete everything older than x date but never delete current version? I have a feeling I set something like this up but it's been a while since I did it.
- nijave 5y ago
  Yeah you can use lifecycle policies https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lifecycle-mgmt.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/object...
- andrewguenther 5y ago
  You are correct, S3 lifecycle policies support deleting non-current object versions once they're X days old as well as X versions old: https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecycle-configuration-examples.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/lifecy...
- justin_oaks 5y ago
  You can, but automatically doing it would defeat the whole purpose of having versions I could roll back to. I need to confirm that that my recent backups are valid before deleting old backups. Otherwise I might as well not have used S3 versioning.

deleted 5y ago

[deleted]

czl 5y ago

Having point in time backups is a good start but I can see ransom-ware adopt to slowly corrupt your data in a way that is reversible but may take months to detect. Your backups going back months will then have this corruption. To detect this application level tripwires may be needed like checksums etc. Finally there is always reputation damage and threats to expose the attack an your data to the public via blackmail. Just because you have backups does not make you safe.

gjs278 5y ago
EvanAnderson 5y ago
Application "tripwires" are just another obstacle for an attacker to overcome. If audits aren't external to the system being audited they're just as vulnerable to manipulation. A Customer of mine in the financial sector sent their backups to a third-party for independent verification quarterly. The third-party restored data into a clean system reconciled against the production system. That would be the kind of auditing that would be more apt to detect the "low and slow" attack.

andrewguenther 5y ago

At a minimum, you should include enabling MFA for the IAM user. Generally, I'd suggest against using IAM users entirely. Ideally you would use an IAM Role via federation or SSO. For my personal accounts I use AWS SSO even though I'm just one person since it enables me to do all my work through role-based authentication and is still protected by MFA on top.

alanfranz 5y ago
Uhm, I don't understand how this would work. We need the backups to run unattended. The IAM user I configure has got no console access at all. Do you have any pointer or example?
- andrewguenther 5y ago
  That's true. You could still do IAM role authentication via web identity which would be much safer than a user and could still be automated.

nijave 5y ago

Seems easier to just do incremental snapshots unless you're on bare metal. Many hypervisors support them and they're built into EC2/EBS If you want to limit data, you can create additional drives and mount them at the appropriate location (or change your application config to save to the auxiliary drives)

ignoramous 5y ago

Incredible that folks have to jump through so many hoops. At this point, AWS should offer versioned S3 backups accessible only offline or through customer support, enabled with a click of a button.

different_sort 5y ago
AWS Backup does exist.
- ignoramous 5y ago
  I knew about AWS Backup but I previously never saw an option to enable it for S3. It looks like it is in limited (1TB) preview in the Oregon region, and doesn't support backing up buckets encrypted with client-provided keys. That said, AWS Backup is the answer to ransomware woes and it can't GA soon enough (3P solutions like Rubrik, Druva notwithstanding) https://docs.aws.amazon.com/aws-backup/latest/devguide/s3-backups.html https://docs.aws.amazon.com/aws-backup/latest/devguide/s3-ba...

rmbyrro 5y ago

> you'll need to make sure that your master access to AWS S3 is never compromised Your master access to S3 should never go into your servers. Create an IAM access with authorization to only PUT objects into S3. > For the purpose we have, Governance mode is OK Maybe not (?), since Governance mode allows for deletion of previous versions. One careless mistake handling your access key/secret and you're exposed to bye bye backups. End note: this is still not enough. An attacker could compromise your backup script and wait for 40 days before locking yourself out of your data. When you try to recover a backup, you'll notice you have none. Perhaps most attackers won't have the patience and will just forget about you, but who knows? A naive protection from that would be to store at least one version of the backups forever. But we're still not covered, since the attacker could compromise that particular version and boom. I can't think of a comprehensive, fully closed-loop solution right now...

bradknowles 5y ago
There is an immutable backup solution to S3: https://www.retrospect.com/en/support/kb/immutable_backups_on_amazon_s3 https://www.retrospect.com/en/support/kb/immutable_backups_o...
rsync 5y ago
"I can't think of a comprehensive, fully closed-loop solution right now..." The ZFS snapshots that you may configure on an rsync.net account are immutable. There are no credentials that can be presented to destroy them or alter their rotation other than our account manager whose changes always pass by a set of human eyes. Which is to say, no 'zfs' commands are ever automated. So the short answer is you simply point borg backup to rsync.net and configure some days/weeks/months of snapshots. The long answer - if you're interested: https://twitter.com/rsyncnet/status/1470669611200770048 https://twitter.com/rsyncnet/status/1470669611200770048 ... skip to 52:20 for "how to destroy an rsync.net account": "... Another thing that lights up big and red on our screen is ... someone's got a big schedule of snapshots ... and then they change it to zero ... you've got seven days and four weeks and six months but we want to change that to zero days of snapshots. We see those things ... and we reach out to people."
- 5y ago

antaviana 5y ago

One good alternative is to upload to an S3 bucket with object lock enabled. This way you can store immutable objects. You can make them immutable for everyone if you wish and the only way to delete them is to close the AWS account. I cannot think of a safest place for a backup than a bucket with object lock.

WatchDog 5y ago
The article is about object lock
- mlac 5y ago
  One good alternative to reading the articles is just skimming the comments.
  - ignoramous 5y ago
    Yet another alternative is to misread the title.
momothereal 5y ago
Can you still configure object retention when they are locked? i.e. automatically delete objects after X days.

joe_the_user 5y ago

As I understand it, the problem enterprises hit isn't the lack of backups or that backups are compromised. The problem seems to be that the restore process has stopped being scalable - restore of all enterprise data can take weeks or longer, and so paying the hacker is cheaper than doing the restore even though the restore would work.

justin_oaks 5y ago
Some companies I've worked for have "disaster recovery" drills periodically. Those should involve restoring backups. From that you should have a timeline of how long it takes to perform a restore. If it takes weeks to perform a restore then that should be a major red flag and be addressed. Of course, if a company has the kind of security that allows them to be hit by ransomware, then they probably aren't diligent about their disaster recovery scenarios.
- spydum 5y ago
  obvious problem here is that most dont perform dr test across all systems simultaneously - they do it by major component, and usually have a working system.. ransomware can take down ALL of that, so you are left with bare metal recovery, and no infra to build off of.
travem 5y ago
That depends, with modern enterprise backup software it can be faster to recover that way than rely on the (sometimes slow) decryption software. This was the case with Colonial Pipeline for example, https://www.theregister.com/2021/05/13/colonial_pipeline_ransom/ https://www.theregister.com/2021/05/13/colonial_pipeline_ran...

WatchDog 5y ago

Object lock helps tick some regulatory boxes, but at least in the described scenario, enabling it isn't really going to protect you against many additional threat models. Simply using object versioning, and an IAM role with no delete permission is sufficient. The biggest risk with using AWS for backups, is being compromised via a high-privilege IAM role. If a role has s3:* permission, it can just remove the locks and delete the file.

justin_oaks 5y ago
Indeed. It'd be best to have another copy of the backup somewhere other than S3, or in S3 under a different AWS account. I worked for a company that had all their databases in AWS RDS and all their database backups as RDS backups stored in the same AWS account. A compromise of a single high-privilege account could cause that company to no longer have ANY data. I warned the higher-ups repeatedly that they needed more than one backup location. They didn't seem to care too much.
- alanfranz 5y ago
  Yes. Cross account replication is the way to go for better protection.

ghostoftesla 5y ago

ddtaylor 5y ago

I use duplicity (via the DejaDup GUI) and I really like it. However, there is one thing I think most people need to be aware of: Duplicity does NOT encrypt the names of your files. Some might not care about this much, but for me I really don't like my encrypted backups containing so much metadata. EDIT: FWIW I don't use the AWS backup options. I have separate offsite backup connections to cloud services as well as NAS.

aborsy 5y ago
Any half-ass backup software encrypts the file names in the repository, and it seems duplicity does too.
- _zooted 5y ago
  It actually does not as you can see all of the file names before you enter your encryption password.
  - aborsy 5y ago
    That’s most likely local cache (it takes file names from your local disk)!! The file names and content are encrypted. No decent backup software will leave file names in plaintext.
alanfranz 5y ago
Author here. > Duplicity does NOT encrypt the names of your files. I, personally, do not care, but you're right. For the purpose of this article I could even leave encryption off, though. "I don't trust the backup destination" wouldn't be part of my threat model here.

m3nu 5y ago

Short plug: At https://BorgBase.com https://BorgBase.com, we also do append-only backups and every repo is isolated. The majority or repos is append-only at this point.

alanfranz 5y ago
Thanks for your plug :-) Your company seems interesting, but could you give us some background? How many people are working there? What are your assurances? I cited rsync.net in my article, and I leverage AWS S3 as an example, because they're well-known players. I would be more hesitant to use a small and unknown (hence potentially unreliable) backup provider.
- m3nu 5y ago
  BorgBase has been around since 2018 and we also maintain Vorta, a popular desktop backup client (also included in Debian since Buster). In addition we sponsor Borg and Borgmatic to keep the eco system healthy. Most providers don't go as far. So I'm quite proud of that. BorgBase itself doesn't have staff, but runs under Peakford.com, which offers a variety of hosting- and consulting offers. Myself, I mainly take care of our open source offerings and community.
- AdaX 5y ago

alanfranz 5y ago

Author here. I'd like to thank the HN folks for some great feedback (as usual). I'll update the article in the next hours to add some caveats.