4 ms·
As for "cloud provider and encrypted" - how did you do that? Basically, what I want is a way to back up about 8 TB of data on my (LUKS-encrypted) home server to
by mschuster91 5y ago
As for "cloud provider and encrypted" - how did you do that? Basically, what I want is a way to back up about 8 TB of data on my (LUKS-encrypted) home server to the cloud, given the following constraints:
- data retrieval is not needed except for recovery case
- ideally, the backup process on the home server side should not be more difficult than a cron job running "rsync -avz --delete /mnt/raid user@server:/mnt/storage"
- integrity of everything should be assured - there's a couple of filesystem-level backups made with "rsync -av" on the data store which means UID/GID, chmod, symlinks, special files (e.g. device files) and whatever Samba uses to store Time Machine xattr metadata must be kept, and there should (but not must) be a way to verify if the file content in the backup is still intact.
- there must be absolutely no way for the cloud or server hosting provider or someone gaining access to the server e.g. via an RCE in the SSH or other sync daemon to access any data (both content and metadata like file name) both in transit and at rest
- it should be somewhat affordable (e.g. Amazon Glacier is ~33 $ a month, Backblaze ~40$ a month)
- ideally, there should be some form of asymmetric encryption be used so that decrypting the data requires the possession of one of three off-site YubiKeys with each having a distinct on-key-generated RSA4096 key and the corresponding password.
The easiest way to accomplish the first three targets would be to simply spin up a tiny AWS EC2 instance with an attached LUKS-encrypted EBS volume or rent a dedicated/colo server somewhere with the same setup and run "rsync -avAHX --delete" on the home server, but that's not affordable and there is a risk of the provider/a hacker accessing/manipulating the cloud server while it is running.
Duplicity plus any "dumb storage" seems to fulfill almost all constraints, but it seems to require either symmetric encryption or access to the private key on the home server - otherwise, how would it be able to decrypt and read the existing backup to find out what has already been backed up?
- _h9mb 5y ago> there must be absolutely no way for the cloud or server hosting provider or someone gaining access to the server e.g. via an RCE in the SSH or other sync daemon to access any data (both content and metadata like file name) both in transit and at rest Gocryptfs.
- tentacleuno 5y ago> As for "cloud provider and encrypted" - how did you do that? Basically, what I want is a way to back up about 8 TB of data on my (LUKS-encrypted) home server to the cloud I personally just throw it all into a .7z file and then upload it to a cloud service (Mega, specifically). I chose 7z as it has the capability to encrypt file names, which would otherwise leak a lot of metadata to the cloud storage provider. The main downside of this is that Mega deletes your stuff after 3 months of idle time, so I need to periodically check in on it to make it think I'm still using it. I am really not sure how 8TB of data would fare with this setup though. For one, you'd definitely want to set up some sort of streaming thing instead of buffering the .7z file on the server and then uploading it.
- mschuster91 5y agoRe-uploading 8 TB each month is impossible, here in Germany all you can get on a residential VDSL line is ~10-20 MBit/s, which means you're stuck at 24/7 uploading for way over a month.
- KingOfCoders 5y agoGermany: I get 100mbit upload consistently and with 1gbit fiber would get 200mbit+. If in dire need with a second provider that has the fiber in my home already I could get 400mbit+ upload.
- tentacleuno 5y ago> Re-uploading 8 TB each month is impossible Woah, each month? I was assuming you would be syncing deltas, i.e. only upload what's actually changed. An ideal system would be to keep a list of files and their respective hashes on each 7z archive task, then do a final checksum of the 7z file so you can ensure its integrity when you download it from the cloud. A quick look at the manpage[0] shows there's an update option, although wouldn't you have to keep the whole archive locally? [0]: https://linux.die.net/man/1/7z https://linux.die.net/man/1/7z
- sekh60 5y agoLook into restic or Borg backup. Restic with rclone can directly backup to a lot of cloud providers, with Borg you may have to remount the remote share.
- KingOfCoders 5y agoUsed rsync.net with borg successfully for years, +1
- hx833001 5y agoAws Glacier deep archive would be $8 a month
- rectang 5y agoAt least until you need to restore, then I understand that you will pay much more.
- withinboredom 5y agoOr just throw Garage on a few nodes and duplicate the data all over the place.
- cesarb 5y agoAn option which seems to match most if not all of your requirements is borgbackup. As for the private key, borgbackup has several modes, the default being one where the private key is stored on the server but encrypted with a passphrase; if you don't like it, you could use the mode where the private key encrypted with the passphrase is kept locally and then backup that (very small) key file separately.
- msh 5y agoRclone does the encryption and incremental backups
- mindslight 5y ago> ideally, there should be some form of asymmetric encryption be used so that decrypting the data requires the possession of one of three off-site YubiKeys with each having a distinct on-key-generated RSA4096 key and the corresponding password This last one seems to drastically change the design into a place that hasn't really been developed. It would be quite neat, in addition to adding other properties like a compromise of the backed up server couldn't be used to destroy a backup. But assuming your goal is to pragmatically get it done, I would forgo this and accept that your backed up server is going to have full control over the backups. FWIW I view online/cloud backups as having the strength of being synced often, and therefore up to date in case physical damage happens to the server. For longer term storage I use offsite drives in a safe deposit box, that take care of things like infrastructure being compromised and deliberately deleted along with online backups. For long term data integrity I use ZFS, borg verify, and ad-hoc sha512sums (for things that don't change much, like music collection). In general keep in mind that your 8TB to backup likely has a power law distribution where your really important filesets are much smaller and can therefore be inexpensively backed up with more copies. The best pricing I've seen for block devices is Kimsufi (10 USD / 2TB-mo) and buyvm.net (5 USD / 1TB-mo) although I haven't used the latter.
- _h9mb 5y ago> In general keep in mind that your 8TB to backup likely has a power law distribution where your really important filesets are much smaller and can therefore be inexpensively backed up with more copies. In my personal experience a simple ignore list can reduce size by an order of magnitude. Things that doesn't make sense to backup like thumbnails, cache files, development dependencies. And smaller is more robust, not only can it be inexpensively backed up with more copies but it can also be synced more frequently with a lower risk of partition and also is quicker to download and restore.
- useragent86 5y agoFriendly, pedantic, hopefully helpful reminder: > - ideally, the backup process on the home server side should not be more difficult than a cron job running "rsync -avz --delete /mnt/raid user@server:/mnt/storage" A mirror is not the same thing as a backup.
- jmmv 5y agoI think https://www.tarsnap.com/ https://www.tarsnap.com/ might do the trick for you.
- philjohn 5y agoCheck out Crashplan small business - $10 a month per endpoint. Not officially supported on a headless server, but there are docker containers that work.