7 ms·

I was recently looking into options for offsite, encrypted (at-rest & in-transit), deduplicated, snapshotted, backups to S3. S3QL was one of the options I inves

by indiv0 11y ago

I was recently looking into options for offsite, encrypted (at-rest & in-transit), deduplicated, snapshotted, backups to S3. S3QL was one of the options I investigated, but unfortunately their crypto is MAC-then-encrypt [0], which is no good. A small sample of the other options considered: * EncFS + rsnapshot/rdiff-backup * Weak cryptography in EncFS [1] * Duplicity * Uses forward snapshots, so recovery takes longer with each successive snapshot, unless you upload a "fresh" copy of the data * ext4 on LVM on LUKS on a loopback device provided by s3backer [2] * OK I'll admit this one is pretty far out there, but it actually works decently well (ignoring the massive FUSE overhead) * Downside: a slight hiccup in network connectivity guarantees data corruption * Attic * Known data corruption issues with large datasets [3] Does anyone know of anything that can provide the above requirements? At this point it's really more of a thought experiment, as I've decided to go with Tarsnap [4] for my use-case. [0]: http://www.rath.org/s3ql-docs/impl_details.html http://www.rath.org/s3ql-docs/impl_details.html [1]: https://defuse.ca/audits/encfs.htm https://defuse.ca/audits/encfs.htm [2]: https://github.com/archiecobbs/s3backer https://github.com/archiecobbs/s3backer [3]: http://librelist.com/browser/attic/2015/3/31/comparison-of-attic-vs-bup-vs-obnam/#cbbe599389a20c787a74b137dc78fb1a http://librelist.com/browser/attic/2015/3/31/comparison-of-a... [4]: https://www.tarsnap.com/ https://www.tarsnap.com/

lfam 11y ago
> offsite, encrypted (at-rest & in-transit), deduplicated, snapshotted, backups to S3 I think restic[1] is going for that. I don't think it's "ready" yet, though. [1] https://github.com/restic/restic https://github.com/restic/restic
- Freaky 11y ago
  Gave it a quick go - plugged it into an existing Attic backup script so it scanned/excluded the same dirs. Result: 7.9MB Attic etc backup -> 22MB Restic backup. 43GB Attic homes backup -> 71GB Restic backup.
  - lfam 11y ago
    That's dramatic. I'm not sure if restic has compression yet. If not, that could explain it.
    - Freaky 11y ago
      Looks like you're right: https://github.com/restic/restic/issues/21 https://github.com/restic/restic/issues/21 And of course you can't do it after the fact because you can't turn off the crypto.
pslam 11y ago
I run a slightly crazy setup for about 200GB of data: * raid1 on all data. This isn't a backup - it's for end-to-end detection of bit flips in (some of) the storage path (and at rest). This needs a better solution (zfs?) * zbackup of all selected files to external hard disk, overnight. This handles the de-duplication. * duplicity of the zbackup dataset to S3, immediately following. This is usually a small upload - zbackup diffs are tiny and it doesn't touch files it doesn't need to. * Rate limited so full backup is about a week, incrementals usually only a few MB. It seems to work well so far, and I'm prepared to do a fresh full backup every year, swapping disks periodically. This general idea is to keep data in 3 physical places: local, external and remote. Local can be recovered from external. External can be thrown away and recreated. Remote recreated. Wish there was an all-in-one solution with all the checkboxes checked. It feels silly to have to get there with a bunch of scripts.
- pja 11y ago
  raid1 on all data. This isn't a backup - it's for end-to-end detection of bit flips in (some of) the storage path (and at rest). This needs a better solution (zfs?) Yeah, btrfs or zfs are the better choice for this than RAID1 in the modern world I think. ZFS anywhere it runs natively, btrfs on Linux. (My experience with btrfs is that it’s fine on a server, but on laptops I’ve ended up with unmountable, unfixable filesystems a number of times. To be fair to btrfs I’ve always been able to recover the data though.)
  - riquito 11y ago
    Anecdotal, but I use btrfs on my laptop since august 2014, had a good number of hard power off not related to btrfs and the filesystem never failed me once. I had some problem with space though using intensively docker: in some situations btrfs thought that the free space was finished and I had to manually rebuild the metadata.
    - vidarh 11y ago
      The Docker-on-Btrfs problems are/were severe enough that CoreOS now defaults to overlayfs on Ext4. I have a bunch of EC2 servers on CoreOS, and we ended up wiping Docker partitions regularly until that switch...
nabla9 11y ago
>MAC-then-encrypt [0], which is no good. He must have red Practical Cryptography from Ferguson and Schneier. In any case, it's not big issue. Encrypt-then-MAC is easier to get wrong.
- tptacek 11y ago
  I don't know about the constructions and attack scenarios pertaining to this app, but MAC-then-encrypt is generally insecure; it's not "easier to get wrong", but is itself a flaw (typically: of exposing the cipher to malicious ciphertext, rather than screening it out with a MAC, which is designed to distinguish between valid and invalid ciphertext).
  - zx2c4 11y ago
    Rogaway's bringing it back with a vengeance though: http://web.cs.ucdavis.edu/~rogaway/aez/aez.pdf http://web.cs.ucdavis.edu/~rogaway/aez/aez.pdf With AEZ, the "MAC" is actually just a block of zeros.
    - JoachimSchipper 11y ago
      That's not MAC-then-encrypt, that's "ECB mode is pretty okay if your message and a nonce fits in your block" plus "let's make an arbitrary-length block cipher". In any case, the result is AEAD, not MAC-then-encrypt. (That said, AEZ is really interesting.)
    - pbsd 11y ago
      Rogaway brought it back with a vengeance back in 2006, with SIV: http://web.cs.ucdavis.edu/~rogaway/papers/keywrap.pdf http://web.cs.ucdavis.edu/~rogaway/papers/keywrap.pdf Essentially all 'misuse resistant' modes are some form of MtE, since in those modes one necessarily needs to process the entire message before beginning encryption. AEZ is MtE in the loosest sense, basically another way to say it's a two-pass/offline mode.
      - tptacek 11y ago
zimbatm 11y ago
Camlistore can use S3 as a backend and does deduplication. Not sure about the encryption though. [0]: https://camlistore.org/ https://camlistore.org/
- darkr 11y ago
  It's supported encryption since last year or thereabouts. I really like the look of camlistore and have done a bit of playing around with it. Was considering deploying it a while back but the docs are [still] lacking and far as I can tell doesn't seem to have gotten to a production ready state yet. Here's a gist showing how to configure an encrypted store on s3: https://gist.github.com/cknave/3ddae29cc466663cb40e https://gist.github.com/cknave/3ddae29cc466663cb40e
aidos 11y ago
I'm glad this post has made it to the top of HN. I've also been looking around at these sorts of solutions over the last couple of weeks. I hadn't rolled anything out yet but s3ql was the leader for me at the moment. I don't know so much about the encryption side but to be honest it's lower down my list of priorities. Another solution I've seen people using is attic[0] as a centralised backup for multiple machines and then running s3ql to push that over to s3. [0] https://attic-backup.org https://attic-backup.org
mappu 11y ago
The attic corruption in question is apparently caused by a bug in the version of msgpack the reporter was using, can be mitigated by updating dependencies - see https://github.com/jborg/attic/issues/264 https://github.com/jborg/attic/issues/264
diggs 11y ago
If you're open to using something proprietary and commercially supported then depending on your scale you should check out http://storreduce.com http://storreduce.com. We're a growing startup in this space. We do real-time compression, encryption (on the wire and at rest) and deduplication for object storage. We currently support an s3-compatible (including a full policy engine) api out the front end, and on the backend we can store to anything that exposes an s3 api (S3, Glacier etc.). Because of the s3-compatible interface we work with any existing client tools that work with s3. We pride ourselves on our speed and scale. We can do 600mb/s sustained throughput and easily scale to multi-petabyte datasets. We typically see 95%-97% dedupe ratios on backup data. We support high availability clustering and replication (for example, replicate between regions for DR). We don't currently support snapshotting but it's something we can implement relatively easily if people need it. Our deployment model is based on a virtual appliance and can be deployed in the cloud or on premise. We can also do things like an on-premise writer (that only uploads unique, deduped data over the network), and a reader in the cloud to support cloud workloads or DR. We have a real focus on backup to cloud in addition to supporting real time big-data use cases in the cloud. Disclaimer: I work here - if you would like to contact me please feel free tpower@storreduce.com.
- k3d3 11y ago
  Sorry to say, but if the source is not available for a crypto tool, it's probably not worth looking at. Best of luck though.
  - Tepix 11y ago
    Perhaps they can move the encryption into an extra open source module in the future.
    - edwintorok 11y ago
      Encryption should happen client-side [1], an S3-to-S3 gateway wouldn't help unless you deploy one on all client nodes. s3cmd and duplicity has support via GPG, but not all S3 clients will know what to do with those files. [1] http://www.skylable.com/blog/2014/09/transparency-reports-server-side-encryption-the-placebo-of-privacy/ http://www.skylable.com/blog/2014/09/transparency-reports-se... Disclaimer: I'm co-founder of Skylable
drdaeman 11y ago
Although git-annex is not a backup system, it works for me as one to backup photos, disk images, some documents etc (although I use it with hubiC and self-hosted SFTP storages, not S3). Maybe it could meet your requirements, too. Files (or chunks, if it's enabled) are gpg-encrypted (to a key or using a symmetric passphrase) and could be stored in nearly anything that could store named blobs. Supports various modes (archival, backup, RAID-like). Performance is fair for me. Downside is that it's not a "real" filesystem, but a git repo that tracks metadata, plus blob storage. Assistant daemon saves a lot of keystrokes, though. There is also some FUSE-based implementation, but I haven't tried it. Getting started is easy, so I suggest taking a look and toying around for half an hour. Another downside is that it - being a git repo - may have problems with storing another repositories inside. Haven't tried this, though.
- rsync 11y ago
  Many hundreds of folks are pointing very similar git-annex based backup schemes to us and we're always happy to have an ultra-clueful customer such as that. Joey and I put together a special discount rate for git-annex folks that want to point at rsync.net: http://www.rsync.net/products/git-annex-pricing.html http://www.rsync.net/products/git-annex-pricing.html
witten 11y ago
Try Borg, a fork of Attic with more active development: https://borgbackup.github.io/borgweb/ https://borgbackup.github.io/borgweb/ Also check out Atticmatic, a wrapper script for both Attic and Borg that makes them much nicer to configure and use: https://torsion.org/atticmatic/ https://torsion.org/atticmatic/
rsync 11y ago
Of the four methods you list, and their "warts", I would say that duplicity seems the easiest to overcome: "Uses forward snapshots, so recovery takes longer with each successive snapshot, unless you upload a "fresh" copy of the data" (unless you have a relatively slow network connection) duplicity will interoperate with plain old SFTP, so we[1] have supported it since it was released. We have also contributed funds to the development of duplicity. In these 10 or so years that it has been around, we've never seen any complaints from customers or evidence of things breaking - it is quite solid. That being said, we do really want to support attic since so many people ask for it. It requires a server-side component to be run, and we try to keep our environment squeaky clean and dead simple[2] - but I think we can cxFreeze it and run it as a binary executable on our end... [1] rsync.net, with HN-readers discount. Just email. [2] There are no interpreters in the jail that each account is locked in, so we can't run a python component on our end.
hobarrera 11y ago
Tarsnap was exactly what I was going to suggest. Exactly what don't you like about it//why do you want to replace it?
Nikratio 11y ago
If MAC-then-encrypt is no good, why don't you report it as a bug against S3QL (with trustworthy references of course)? It may even get fixed.

by indiv0 11y ago

lfam 11y ago

> offsite, encrypted (at-rest & in-transit), deduplicated, snapshotted, backups to S3 I think restic[1] is going for that. I don't think it's "ready" yet, though. [1] https://github.com/restic/restic https://github.com/restic/restic

Freaky 11y ago
Gave it a quick go - plugged it into an existing Attic backup script so it scanned/excluded the same dirs. Result: 7.9MB Attic etc backup -> 22MB Restic backup. 43GB Attic homes backup -> 71GB Restic backup.
- lfam 11y ago
  That's dramatic. I'm not sure if restic has compression yet. If not, that could explain it.
  - Freaky 11y ago
    Looks like you're right: https://github.com/restic/restic/issues/21 https://github.com/restic/restic/issues/21 And of course you can't do it after the fact because you can't turn off the crypto.

pslam 11y ago

I run a slightly crazy setup for about 200GB of data: * raid1 on all data. This isn't a backup - it's for end-to-end detection of bit flips in (some of) the storage path (and at rest). This needs a better solution (zfs?) * zbackup of all selected files to external hard disk, overnight. This handles the de-duplication. * duplicity of the zbackup dataset to S3, immediately following. This is usually a small upload - zbackup diffs are tiny and it doesn't touch files it doesn't need to. * Rate limited so full backup is about a week, incrementals usually only a few MB. It seems to work well so far, and I'm prepared to do a fresh full backup every year, swapping disks periodically. This general idea is to keep data in 3 physical places: local, external and remote. Local can be recovered from external. External can be thrown away and recreated. Remote recreated. Wish there was an all-in-one solution with all the checkboxes checked. It feels silly to have to get there with a bunch of scripts.

pja 11y ago
raid1 on all data. This isn't a backup - it's for end-to-end detection of bit flips in (some of) the storage path (and at rest). This needs a better solution (zfs?) Yeah, btrfs or zfs are the better choice for this than RAID1 in the modern world I think. ZFS anywhere it runs natively, btrfs on Linux. (My experience with btrfs is that it’s fine on a server, but on laptops I’ve ended up with unmountable, unfixable filesystems a number of times. To be fair to btrfs I’ve always been able to recover the data though.)
- riquito 11y ago
  Anecdotal, but I use btrfs on my laptop since august 2014, had a good number of hard power off not related to btrfs and the filesystem never failed me once. I had some problem with space though using intensively docker: in some situations btrfs thought that the free space was finished and I had to manually rebuild the metadata.
  - vidarh 11y ago
    The Docker-on-Btrfs problems are/were severe enough that CoreOS now defaults to overlayfs on Ext4. I have a bunch of EC2 servers on CoreOS, and we ended up wiping Docker partitions regularly until that switch...

nabla9 11y ago

>MAC-then-encrypt [0], which is no good. He must have red Practical Cryptography from Ferguson and Schneier. In any case, it's not big issue. Encrypt-then-MAC is easier to get wrong.

tptacek 11y ago
I don't know about the constructions and attack scenarios pertaining to this app, but MAC-then-encrypt is generally insecure; it's not "easier to get wrong", but is itself a flaw (typically: of exposing the cipher to malicious ciphertext, rather than screening it out with a MAC, which is designed to distinguish between valid and invalid ciphertext).
- zx2c4 11y ago
  Rogaway's bringing it back with a vengeance though: http://web.cs.ucdavis.edu/~rogaway/aez/aez.pdf http://web.cs.ucdavis.edu/~rogaway/aez/aez.pdf With AEZ, the "MAC" is actually just a block of zeros.
  - JoachimSchipper 11y ago
    That's not MAC-then-encrypt, that's "ECB mode is pretty okay if your message and a nonce fits in your block" plus "let's make an arbitrary-length block cipher". In any case, the result is AEAD, not MAC-then-encrypt. (That said, AEZ is really interesting.)
  - pbsd 11y ago
    Rogaway brought it back with a vengeance back in 2006, with SIV: http://web.cs.ucdavis.edu/~rogaway/papers/keywrap.pdf http://web.cs.ucdavis.edu/~rogaway/papers/keywrap.pdf Essentially all 'misuse resistant' modes are some form of MtE, since in those modes one necessarily needs to process the entire message before beginning encryption. AEZ is MtE in the loosest sense, basically another way to say it's a two-pass/offline mode.
    - tptacek 11y ago

zimbatm 11y ago

Camlistore can use S3 as a backend and does deduplication. Not sure about the encryption though. [0]: https://camlistore.org/ https://camlistore.org/

darkr 11y ago
It's supported encryption since last year or thereabouts. I really like the look of camlistore and have done a bit of playing around with it. Was considering deploying it a while back but the docs are [still] lacking and far as I can tell doesn't seem to have gotten to a production ready state yet. Here's a gist showing how to configure an encrypted store on s3: https://gist.github.com/cknave/3ddae29cc466663cb40e https://gist.github.com/cknave/3ddae29cc466663cb40e

aidos 11y ago

I'm glad this post has made it to the top of HN. I've also been looking around at these sorts of solutions over the last couple of weeks. I hadn't rolled anything out yet but s3ql was the leader for me at the moment. I don't know so much about the encryption side but to be honest it's lower down my list of priorities. Another solution I've seen people using is attic[0] as a centralised backup for multiple machines and then running s3ql to push that over to s3. [0] https://attic-backup.org https://attic-backup.org

mappu 11y ago

The attic corruption in question is apparently caused by a bug in the version of msgpack the reporter was using, can be mitigated by updating dependencies - see https://github.com/jborg/attic/issues/264 https://github.com/jborg/attic/issues/264

diggs 11y ago

If you're open to using something proprietary and commercially supported then depending on your scale you should check out http://storreduce.com http://storreduce.com. We're a growing startup in this space. We do real-time compression, encryption (on the wire and at rest) and deduplication for object storage. We currently support an s3-compatible (including a full policy engine) api out the front end, and on the backend we can store to anything that exposes an s3 api (S3, Glacier etc.). Because of the s3-compatible interface we work with any existing client tools that work with s3. We pride ourselves on our speed and scale. We can do 600mb/s sustained throughput and easily scale to multi-petabyte datasets. We typically see 95%-97% dedupe ratios on backup data. We support high availability clustering and replication (for example, replicate between regions for DR). We don't currently support snapshotting but it's something we can implement relatively easily if people need it. Our deployment model is based on a virtual appliance and can be deployed in the cloud or on premise. We can also do things like an on-premise writer (that only uploads unique, deduped data over the network), and a reader in the cloud to support cloud workloads or DR. We have a real focus on backup to cloud in addition to supporting real time big-data use cases in the cloud. Disclaimer: I work here - if you would like to contact me please feel free tpower@storreduce.com.

k3d3 11y ago
Sorry to say, but if the source is not available for a crypto tool, it's probably not worth looking at. Best of luck though.
- Tepix 11y ago
  Perhaps they can move the encryption into an extra open source module in the future.
  - edwintorok 11y ago
    Encryption should happen client-side [1], an S3-to-S3 gateway wouldn't help unless you deploy one on all client nodes. s3cmd and duplicity has support via GPG, but not all S3 clients will know what to do with those files. [1] http://www.skylable.com/blog/2014/09/transparency-reports-server-side-encryption-the-placebo-of-privacy/ http://www.skylable.com/blog/2014/09/transparency-reports-se... Disclaimer: I'm co-founder of Skylable

drdaeman 11y ago

Although git-annex is not a backup system, it works for me as one to backup photos, disk images, some documents etc (although I use it with hubiC and self-hosted SFTP storages, not S3). Maybe it could meet your requirements, too. Files (or chunks, if it's enabled) are gpg-encrypted (to a key or using a symmetric passphrase) and could be stored in nearly anything that could store named blobs. Supports various modes (archival, backup, RAID-like). Performance is fair for me. Downside is that it's not a "real" filesystem, but a git repo that tracks metadata, plus blob storage. Assistant daemon saves a lot of keystrokes, though. There is also some FUSE-based implementation, but I haven't tried it. Getting started is easy, so I suggest taking a look and toying around for half an hour. Another downside is that it - being a git repo - may have problems with storing another repositories inside. Haven't tried this, though.

rsync 11y ago
Many hundreds of folks are pointing very similar git-annex based backup schemes to us and we're always happy to have an ultra-clueful customer such as that. Joey and I put together a special discount rate for git-annex folks that want to point at rsync.net: http://www.rsync.net/products/git-annex-pricing.html http://www.rsync.net/products/git-annex-pricing.html

witten 11y ago

Try Borg, a fork of Attic with more active development: https://borgbackup.github.io/borgweb/ https://borgbackup.github.io/borgweb/ Also check out Atticmatic, a wrapper script for both Attic and Borg that makes them much nicer to configure and use: https://torsion.org/atticmatic/ https://torsion.org/atticmatic/

rsync 11y ago

Of the four methods you list, and their "warts", I would say that duplicity seems the easiest to overcome: "Uses forward snapshots, so recovery takes longer with each successive snapshot, unless you upload a "fresh" copy of the data" (unless you have a relatively slow network connection) duplicity will interoperate with plain old SFTP, so we[1] have supported it since it was released. We have also contributed funds to the development of duplicity. In these 10 or so years that it has been around, we've never seen any complaints from customers or evidence of things breaking - it is quite solid. That being said, we do really want to support attic since so many people ask for it. It requires a server-side component to be run, and we try to keep our environment squeaky clean and dead simple[2] - but I think we can cxFreeze it and run it as a binary executable on our end... [1] rsync.net, with HN-readers discount. Just email. [2] There are no interpreters in the jail that each account is locked in, so we can't run a python component on our end.

hobarrera 11y ago

Tarsnap was exactly what I was going to suggest. Exactly what don't you like about it//why do you want to replace it?

Nikratio 11y ago

If MAC-then-encrypt is no good, why don't you report it as a bug against S3QL (with trustworthy references of course)? It may even get fixed.