8 ms·

Append-only backups with restic and rclone

75 points by edward 8y ago

dorfsmay 8y ago
This is great. I find myself having to explain and argue this point way too often: "One issue with most backup solutions is that an attacker controlling the local system can also wipe its old backups."
- regecks 8y ago
  S3 also makes this kind of protection pretty easy - enable bucket versioning and allow only PutObject.
  - juskrey 8y ago
    write only FTP rights
  - bpye 8y ago
    B2 can do this too if S3 is a bit too spendy.
- yroc92 8y ago
  What kind of solution to this problem would you propose?
  - Jaepa 8y ago
    Well, append-only backups with something like restic and rclone seems like the obvious one.
  - mprovost 8y ago
    Tapes sitting on a shelf is the classic solution.
    - m3nu 8y ago
      Write-once backup media, like DVDs were even better. But they are getting out of fashion.
jeltz 8y ago
Borg Backup also has this feature, and so far my experience with it has been great.
- witten 8y ago
  borgmatic, a Borg Backup wrapper with a declarative config file, also supports creating append-only Borg repositories. More at https://torsion.org/borgmatic/ https://torsion.org/borgmatic/
- kklimonda 8y ago
  With borg, make sure that you read and considered drawbacks of the append-only mode: https://borgbackup.readthedocs.io/en/stable/usage/notes.html#drawbacks https://borgbackup.readthedocs.io/en/stable/usage/notes.html... I'm curious whether restic's append-only fares better here.
  - Tharre 8y ago
    How could that possibly be improved? Append-only means no deleting or freeing up space by definition.
    - indigo945 8y ago
      Not necessarily, because the backup server could squash backups itself, even when the client is not allowed to do so.
jsiepkes 8y ago
Worth noting that if you run your own Restic server you can run it in append only mode.
hendry 8y ago
Seems a bit complex. How about the remote server adding the immutable flag on the uploaded files?
ioquatix 8y ago
Or just use ZFS and send/recv over the network? It's super simple, guarantees reliability, and very easy to replicate, e.g. offline backups.
- cyphar 8y ago
  The nice thing about restic is that you don't need to have a smart server, you can store your backups anywhere. And it doesn't require a particular filesystem on your host. Not to mention that deduplication is very cheap due its design (and unlike extent-based dedup, content-defined dedup also helps dedup some file changes a lot better). ZFS, while it is definitely a great project, doesn't provide those features.
  - cryptonector 8y ago
    You can store zfs send output as a file. You don't need a ZFS filesystem to receive into if all you're doing is backups.
    - bpye 8y ago
      I've never considered doing this, can you do it differentially?
      - cryptonector 8y ago
        Do you mean incrementally? Yes, zfs send sends the contents that's different between snapshots (or the entire dataset up to a snapshot, if you want).
        bpye 8y ago
        Hm, I think I understand. So you could save the entire dataset up to snapshot x, then for x+1 you would save just a diff. Might have to play around with this. I currently use ZFS snapshots to protect against accidental data loss and I have daily backups off site but these are managed independently. This might make my life easier.
m3nu 8y ago
This bugged me too last year. So I built a hosting service for Borg backups that can have append-only access keys. So the client machine can't change old backups. Ever. In addition you can lock your settings down with 2FA. https://www.borgbase.com/ https://www.borgbase.com/
- bartman 8y ago
  Great service, especially with the addition of the monitoring features. How do you store the data on your end?
  - m3nu 8y ago
    Borg stores all data as plain files in 500MB segments. Every segment has a checksum, which you can validate using the `borg check` command. The actual data lies on a plain vanilla RAID array.
- tomjen3 8y ago
  That is going to be an issue for anyone with GDPR data, since you are required to be able to delete it upon request.
  - m3nu 8y ago
    You can still use a full access key to prune old data/old archives as needed. But you use it from a trusted machine or temporarily. Also: you still need to keep some data even under GDPR. Like billing data.
casylum 8y ago
Does restic support sparse files yet?
nine_k 8y ago
Noting the obvious: if you're backing up customer data, an append-only backup is not GDPR-compliant.
- Xeago 8y ago
  An append-only backup is not by definition non-compliant with GDPR. It's important that the individual can be assured that their personal data will not be restored back to production systems (except in certain rare instances, e.g., the need to recover from a natural disaster or serious security breach). In such cases, the user’s personal data may be restored from backups, but the controller will take the necessary steps to honor the initial request and erase the primary instance of the data again. For example: https://www.acronis.com/en-us/blog/posts/backups-and-gdpr-right-be-forgotten-recommendations https://www.acronis.com/en-us/blog/posts/backups-and-gdpr-ri...
h1d 8y ago
duplicacy is another alternative. It has paid plans but free for personal CLI usage. Their study on various cloud storage performance helped me decide tremendously. https://github.com/gilbertchen/cloud-storage-comparison https://github.com/gilbertchen/cloud-storage-comparison I make backups using multiple implementations (restic and duplicacy) to avoid unfortunate case of an implementation corrupting encrypted / compressed data which may become unrecoverable completely or any other bug to partially corrupt it. I hate it when backups don't work when I need it and it's not easy to do 100% verification on all backups regularly.

75 points by edward 8y ago

This is great. I find myself having to explain and argue this point way too often: "One issue with most backup solutions is that an attacker controlling the local system can also wipe its old backups."

regecks 8y ago
S3 also makes this kind of protection pretty easy - enable bucket versioning and allow only PutObject.
- juskrey 8y ago
  write only FTP rights
- bpye 8y ago
  B2 can do this too if S3 is a bit too spendy.
yroc92 8y ago
What kind of solution to this problem would you propose?
- Jaepa 8y ago
  Well, append-only backups with something like restic and rclone seems like the obvious one.
- mprovost 8y ago
  Tapes sitting on a shelf is the classic solution.
  - m3nu 8y ago
    Write-once backup media, like DVDs were even better. But they are getting out of fashion.

jeltz 8y ago

Borg Backup also has this feature, and so far my experience with it has been great.

witten 8y ago
borgmatic, a Borg Backup wrapper with a declarative config file, also supports creating append-only Borg repositories. More at https://torsion.org/borgmatic/ https://torsion.org/borgmatic/
kklimonda 8y ago
With borg, make sure that you read and considered drawbacks of the append-only mode: https://borgbackup.readthedocs.io/en/stable/usage/notes.html#drawbacks https://borgbackup.readthedocs.io/en/stable/usage/notes.html... I'm curious whether restic's append-only fares better here.
- Tharre 8y ago
  How could that possibly be improved? Append-only means no deleting or freeing up space by definition.
  - indigo945 8y ago
    Not necessarily, because the backup server could squash backups itself, even when the client is not allowed to do so.

jsiepkes 8y ago

Worth noting that if you run your own Restic server you can run it in append only mode.

hendry 8y ago

Seems a bit complex. How about the remote server adding the immutable flag on the uploaded files?

ioquatix 8y ago

Or just use ZFS and send/recv over the network? It's super simple, guarantees reliability, and very easy to replicate, e.g. offline backups.

cyphar 8y ago
The nice thing about restic is that you don't need to have a smart server, you can store your backups anywhere. And it doesn't require a particular filesystem on your host. Not to mention that deduplication is very cheap due its design (and unlike extent-based dedup, content-defined dedup also helps dedup some file changes a lot better). ZFS, while it is definitely a great project, doesn't provide those features.
- cryptonector 8y ago
  You can store zfs send output as a file. You don't need a ZFS filesystem to receive into if all you're doing is backups.
  - bpye 8y ago
    I've never considered doing this, can you do it differentially?
    - cryptonector 8y ago
      Do you mean incrementally? Yes, zfs send sends the contents that's different between snapshots (or the entire dataset up to a snapshot, if you want).
      - bpye 8y ago
        Hm, I think I understand. So you could save the entire dataset up to snapshot x, then for x+1 you would save just a diff. Might have to play around with this. I currently use ZFS snapshots to protect against accidental data loss and I have daily backups off site but these are managed independently. This might make my life easier.

m3nu 8y ago

This bugged me too last year. So I built a hosting service for Borg backups that can have append-only access keys. So the client machine can't change old backups. Ever. In addition you can lock your settings down with 2FA. https://www.borgbase.com/ https://www.borgbase.com/

bartman 8y ago
Great service, especially with the addition of the monitoring features. How do you store the data on your end?
- m3nu 8y ago
  Borg stores all data as plain files in 500MB segments. Every segment has a checksum, which you can validate using the `borg check` command. The actual data lies on a plain vanilla RAID array.
tomjen3 8y ago
That is going to be an issue for anyone with GDPR data, since you are required to be able to delete it upon request.
- m3nu 8y ago
  You can still use a full access key to prune old data/old archives as needed. But you use it from a trusted machine or temporarily. Also: you still need to keep some data even under GDPR. Like billing data.

casylum 8y ago

Does restic support sparse files yet?

nine_k 8y ago

Noting the obvious: if you're backing up customer data, an append-only backup is not GDPR-compliant.

Xeago 8y ago
An append-only backup is not by definition non-compliant with GDPR. It's important that the individual can be assured that their personal data will not be restored back to production systems (except in certain rare instances, e.g., the need to recover from a natural disaster or serious security breach). In such cases, the user’s personal data may be restored from backups, but the controller will take the necessary steps to honor the initial request and erase the primary instance of the data again. For example: https://www.acronis.com/en-us/blog/posts/backups-and-gdpr-right-be-forgotten-recommendations https://www.acronis.com/en-us/blog/posts/backups-and-gdpr-ri...

h1d 8y ago

duplicacy is another alternative. It has paid plans but free for personal CLI usage. Their study on various cloud storage performance helped me decide tremendously. https://github.com/gilbertchen/cloud-storage-comparison https://github.com/gilbertchen/cloud-storage-comparison I make backups using multiple implementations (restic and duplicacy) to avoid unfortunate case of an implementation corrupting encrypted / compressed data which may become unrecoverable completely or any other bug to partially corrupt it. I hate it when backups don't work when I need it and it's not easy to do 100% verification on all backups regularly.