5 ms·

Yesterday, while discussing backups, someone said end users should basically trust the cloud. This is why it is a bad idea. The risk is data loss, irrespectiv

Yesterday, while discussing backups, someone said end users should basically trust the cloud. This is why it is a bad idea. The risk is data loss, irrespective if it is because of hardware failure or because of cloud failure connected to overzealous legal enforcement, algorithm decision making failure,or service deprecation. Part of the risk equation is how big the chance is, and I would really love to see the numbers of hardware faulure vs cloud failure. Another part is impact. Hardware loss might be recoverable via specific services. Cloud failure, basically you're on your own, and they might revoke access to your email a.k.a. digital identity certificate, or even sick the copyright cartel on you, worst case ending in a police visit taking out the other backups.

viktorcode 5y ago
Actually, this shows why using Google Drive as a backup storage may be a bad idea.
philjohn 5y ago
And this is why I put together a NAS for my home, which then backs up to a cloud provider, encrypted - and if that provider decides to play silly shenanigans I can just as easily colo another box and back up to that as well as any other cloud provider. ZFS on Linux is stupidly easy to setup, something like URBackup for all your clients to backup onto the NAS and you've got a nice 1-2-3 (with 3 being offsite) backup scheme.
- sofixa 5y ago
  > ZFS on Linux is stupidly easy to setup For a user familiar with Linux. Even many macOS and Windows based developers would need some time and research to set something like that up, let alone a non technical person.
  - j16sdiz 5y ago
    If this is a problem… Just buy a ready to use box from synology, qnap, western digital, etc… or just use a usb drive.
    - mschuster91 5y ago
      With pre-made boxes, you're viable to run into issues when they graciously decide to force a security update on you. QNAP makes extremely decent boxes but last year's behavior that broke a lot of stuff [1] has all but killed my trust in them. [1]: https://www.zdnet.com/article/decryptor-released-for-deadbolt-ransomware-affecting-qnap-nas-devices/ https://www.zdnet.com/article/decryptor-released-for-deadbol...
    - csydas 5y ago
      TBH, synology and qnap have awful track records when it comes to data integrity and cause a ton of issues even when using the software the "intended" way. SMB is a chatty protocol in general which makes it hard to troubleshoot, and because of default lack of write-through, you have a non-trivial chance of a situation where the SMB stack returns "this write was a-okay!" but in fact the data never landed on disk and only hit the cache. My team works with a lot of clients for backups and we've had so many silent corruption cases with lousy synology and qnap boxes that we just don't support them when using SMB or NFS anymore; the built-in stack is just too unreliable. iSCSI is a bit better, but iSCSI isn't how a lot of people want their NAS to work, so it's always a tough discussion. General purpose servers with proper endpoints have always been better, but it's a hard sell to clients. Directly-attached USB is marginally better in that you don't deal with network protocols, but a lot of companies cheap out on USB controllers and it's a challenge sometimes to convince clients that the on-board controller is the issue when single (small) file writes work without issue. But long story short, I'd propose a small general purpose server (even Raspberry Pi!) over a QNAP/Synology any day of the week. The latter does everything simply but with excess mediocrity. From my point of view, they hit MVP across a dozen + items, but competency in none.
      - adra 5y ago
        People relying on reliable backups building their own storage servers from scratch. Clearly this is universal sage advice that will certainly not end badly for the vast majority of people -_-. I'm on my second Synology, the first was a cheapish 4-bay; now I'm using a much pricier 8-bay. This is for personal use, but the time and effort saved by using this over hand building has saved me factors more money. I do recommend NFS over SMB for the cheaper models as smbd is a lot more CPU heavy and transfers actually became CPU bound pretty quickly. I've never had data corruption issues in my 10ish years of use, but one subjective data point is pretty crap. I don't imagine that there are many personal users who need/want ISCSI or other more raw block I/O protocols. As for businesses, off the shelf solutions in your requirements range are often available, and unless you're looking for very specific optimized workloads, there's probably a bit excessively expensive vendor solution that can meet those requirements.
  - datavirtue 5y ago
    Western Digital, Netgear, or Buffalo will do it for you. The cost is a trade of your time...plus the added value of a device that does one thing and does it well.
- mschuster91 5y ago
  As for "cloud provider and encrypted" - how did you do that? Basically, what I want is a way to back up about 8 TB of data on my (LUKS-encrypted) home server to the cloud, given the following constraints: - data retrieval is not needed except for recovery case - ideally, the backup process on the home server side should not be more difficult than a cron job running "rsync -avz --delete /mnt/raid user@server:/mnt/storage" - integrity of everything should be assured - there's a couple of filesystem-level backups made with "rsync -av" on the data store which means UID/GID, chmod, symlinks, special files (e.g. device files) and whatever Samba uses to store Time Machine xattr metadata must be kept, and there should (but not must) be a way to verify if the file content in the backup is still intact. - there must be absolutely no way for the cloud or server hosting provider or someone gaining access to the server e.g. via an RCE in the SSH or other sync daemon to access any data (both content and metadata like file name) both in transit and at rest - it should be somewhat affordable (e.g. Amazon Glacier is ~33 $ a month, Backblaze ~40$ a month) - ideally, there should be some form of asymmetric encryption be used so that decrypting the data requires the possession of one of three off-site YubiKeys with each having a distinct on-key-generated RSA4096 key and the corresponding password. The easiest way to accomplish the first three targets would be to simply spin up a tiny AWS EC2 instance with an attached LUKS-encrypted EBS volume or rent a dedicated/colo server somewhere with the same setup and run "rsync -avAHX --delete" on the home server, but that's not affordable and there is a risk of the provider/a hacker accessing/manipulating the cloud server while it is running. Duplicity plus any "dumb storage" seems to fulfill almost all constraints, but it seems to require either symmetric encryption or access to the private key on the home server - otherwise, how would it be able to decrypt and read the existing backup to find out what has already been backed up?
  - _h9mb 5y ago
    > there must be absolutely no way for the cloud or server hosting provider or someone gaining access to the server e.g. via an RCE in the SSH or other sync daemon to access any data (both content and metadata like file name) both in transit and at rest Gocryptfs.
  - tentacleuno 5y ago
    > As for "cloud provider and encrypted" - how did you do that? Basically, what I want is a way to back up about 8 TB of data on my (LUKS-encrypted) home server to the cloud I personally just throw it all into a .7z file and then upload it to a cloud service (Mega, specifically). I chose 7z as it has the capability to encrypt file names, which would otherwise leak a lot of metadata to the cloud storage provider. The main downside of this is that Mega deletes your stuff after 3 months of idle time, so I need to periodically check in on it to make it think I'm still using it. I am really not sure how 8TB of data would fare with this setup though. For one, you'd definitely want to set up some sort of streaming thing instead of buffering the .7z file on the server and then uploading it.
KingOfCoders 5y ago
I tell all coachees if they are on AWS, also put backups on GCP/Rsync.net/Herzner/... and vice versa. If your data and backups are with one provider it's a non zero risk of losing everything because of account removal.
- rectang 5y ago
  This is sound advice but it's such a pain in neck to implement. Cloud providers have every incentive to make it difficult for you to duplicate your actions to a competitor, and to make it difficult for any third party who wants to provide such duplication as a convenience.
  - beagle3 5y ago
    As long as it is plain files, rclone takes a few minutes to set up, and removes those difficulties.
- Aachen 5y ago
  Especially if it's encrypted, e.g. restic/bupstash/duplicity+pgp/tarsnap/perkeep/borg/kopia/... Good luck finding the hash of a nipple, copyright-claimed material, or other TOS violations in that data.
datavirtue 5y ago
I haven't had a hardware failure in so long, it's scary. I keep wondering when my next failure is going to be but my countless laptops, hard drives, NAS, and desktop systems keep chugging along without issue. I know of a Netgear consumer NAS I installed in 2011 is still running a business, untouched. It uses two enterprise drives in a mirror config.
ajsnigrutin 5y ago
Hardware loss is easily avoidable with more hardware... create backups, store them on multiple external drives (for most users, a single modern external drive is enough for all the backups, so the additional ones are for copies), and you can then keep one at home, one at work, one at your parents place, and replace them every now and then to 'refresh' the backups... Most home users don't need hourly or daily backups, they just nee their photo and home video collections saved, their documents scanned etc. With cloud providers, storing stuff in different datacenters doesn't help you, if you get banned from google, because someone didn't like your youtube comment. Multiple cloud providers might be using the same amazon/azure/... datacenter in the backend. Then there are different L0-L8 problems, from political, where your country gets embargoed due to some ongoing political struggle, or you can move to another location, where the internet is slow. And now, it seems that you can lose data due to stupidity as in the original post.
- Tempest1981 5y ago
  > easily avoidable > and you can then keep one at home, one at work, one at your parents place, and replace them every now and then to 'refresh' the backups Yes, many of us do this. But I wouldn't consider it "easy", esp for the average user. What % of home computer users own 2+ external drives? Or even 1? I bet it's under 20%.

viktorcode 5y ago

Actually, this shows why using Google Drive as a backup storage may be a bad idea.

philjohn 5y ago

And this is why I put together a NAS for my home, which then backs up to a cloud provider, encrypted - and if that provider decides to play silly shenanigans I can just as easily colo another box and back up to that as well as any other cloud provider. ZFS on Linux is stupidly easy to setup, something like URBackup for all your clients to backup onto the NAS and you've got a nice 1-2-3 (with 3 being offsite) backup scheme.

sofixa 5y ago
> ZFS on Linux is stupidly easy to setup For a user familiar with Linux. Even many macOS and Windows based developers would need some time and research to set something like that up, let alone a non technical person.
- j16sdiz 5y ago
  If this is a problem… Just buy a ready to use box from synology, qnap, western digital, etc… or just use a usb drive.
  - mschuster91 5y ago
    With pre-made boxes, you're viable to run into issues when they graciously decide to force a security update on you. QNAP makes extremely decent boxes but last year's behavior that broke a lot of stuff [1] has all but killed my trust in them. [1]: https://www.zdnet.com/article/decryptor-released-for-deadbolt-ransomware-affecting-qnap-nas-devices/ https://www.zdnet.com/article/decryptor-released-for-deadbol...
  - csydas 5y ago
    TBH, synology and qnap have awful track records when it comes to data integrity and cause a ton of issues even when using the software the "intended" way. SMB is a chatty protocol in general which makes it hard to troubleshoot, and because of default lack of write-through, you have a non-trivial chance of a situation where the SMB stack returns "this write was a-okay!" but in fact the data never landed on disk and only hit the cache. My team works with a lot of clients for backups and we've had so many silent corruption cases with lousy synology and qnap boxes that we just don't support them when using SMB or NFS anymore; the built-in stack is just too unreliable. iSCSI is a bit better, but iSCSI isn't how a lot of people want their NAS to work, so it's always a tough discussion. General purpose servers with proper endpoints have always been better, but it's a hard sell to clients. Directly-attached USB is marginally better in that you don't deal with network protocols, but a lot of companies cheap out on USB controllers and it's a challenge sometimes to convince clients that the on-board controller is the issue when single (small) file writes work without issue. But long story short, I'd propose a small general purpose server (even Raspberry Pi!) over a QNAP/Synology any day of the week. The latter does everything simply but with excess mediocrity. From my point of view, they hit MVP across a dozen + items, but competency in none.
    - adra 5y ago
      People relying on reliable backups building their own storage servers from scratch. Clearly this is universal sage advice that will certainly not end badly for the vast majority of people -_-. I'm on my second Synology, the first was a cheapish 4-bay; now I'm using a much pricier 8-bay. This is for personal use, but the time and effort saved by using this over hand building has saved me factors more money. I do recommend NFS over SMB for the cheaper models as smbd is a lot more CPU heavy and transfers actually became CPU bound pretty quickly. I've never had data corruption issues in my 10ish years of use, but one subjective data point is pretty crap. I don't imagine that there are many personal users who need/want ISCSI or other more raw block I/O protocols. As for businesses, off the shelf solutions in your requirements range are often available, and unless you're looking for very specific optimized workloads, there's probably a bit excessively expensive vendor solution that can meet those requirements.
- datavirtue 5y ago
  Western Digital, Netgear, or Buffalo will do it for you. The cost is a trade of your time...plus the added value of a device that does one thing and does it well.
mschuster91 5y ago
As for "cloud provider and encrypted" - how did you do that? Basically, what I want is a way to back up about 8 TB of data on my (LUKS-encrypted) home server to the cloud, given the following constraints: - data retrieval is not needed except for recovery case - ideally, the backup process on the home server side should not be more difficult than a cron job running "rsync -avz --delete /mnt/raid user@server:/mnt/storage" - integrity of everything should be assured - there's a couple of filesystem-level backups made with "rsync -av" on the data store which means UID/GID, chmod, symlinks, special files (e.g. device files) and whatever Samba uses to store Time Machine xattr metadata must be kept, and there should (but not must) be a way to verify if the file content in the backup is still intact. - there must be absolutely no way for the cloud or server hosting provider or someone gaining access to the server e.g. via an RCE in the SSH or other sync daemon to access any data (both content and metadata like file name) both in transit and at rest - it should be somewhat affordable (e.g. Amazon Glacier is ~33 $ a month, Backblaze ~40$ a month) - ideally, there should be some form of asymmetric encryption be used so that decrypting the data requires the possession of one of three off-site YubiKeys with each having a distinct on-key-generated RSA4096 key and the corresponding password. The easiest way to accomplish the first three targets would be to simply spin up a tiny AWS EC2 instance with an attached LUKS-encrypted EBS volume or rent a dedicated/colo server somewhere with the same setup and run "rsync -avAHX --delete" on the home server, but that's not affordable and there is a risk of the provider/a hacker accessing/manipulating the cloud server while it is running. Duplicity plus any "dumb storage" seems to fulfill almost all constraints, but it seems to require either symmetric encryption or access to the private key on the home server - otherwise, how would it be able to decrypt and read the existing backup to find out what has already been backed up?
- _h9mb 5y ago
  > there must be absolutely no way for the cloud or server hosting provider or someone gaining access to the server e.g. via an RCE in the SSH or other sync daemon to access any data (both content and metadata like file name) both in transit and at rest Gocryptfs.
- tentacleuno 5y ago
  > As for "cloud provider and encrypted" - how did you do that? Basically, what I want is a way to back up about 8 TB of data on my (LUKS-encrypted) home server to the cloud I personally just throw it all into a .7z file and then upload it to a cloud service (Mega, specifically). I chose 7z as it has the capability to encrypt file names, which would otherwise leak a lot of metadata to the cloud storage provider. The main downside of this is that Mega deletes your stuff after 3 months of idle time, so I need to periodically check in on it to make it think I'm still using it. I am really not sure how 8TB of data would fare with this setup though. For one, you'd definitely want to set up some sort of streaming thing instead of buffering the .7z file on the server and then uploading it.

KingOfCoders 5y ago

I tell all coachees if they are on AWS, also put backups on GCP/Rsync.net/Herzner/... and vice versa. If your data and backups are with one provider it's a non zero risk of losing everything because of account removal.

rectang 5y ago
This is sound advice but it's such a pain in neck to implement. Cloud providers have every incentive to make it difficult for you to duplicate your actions to a competitor, and to make it difficult for any third party who wants to provide such duplication as a convenience.
- beagle3 5y ago
  As long as it is plain files, rclone takes a few minutes to set up, and removes those difficulties.
Aachen 5y ago
Especially if it's encrypted, e.g. restic/bupstash/duplicity+pgp/tarsnap/perkeep/borg/kopia/... Good luck finding the hash of a nipple, copyright-claimed material, or other TOS violations in that data.

datavirtue 5y ago

I haven't had a hardware failure in so long, it's scary. I keep wondering when my next failure is going to be but my countless laptops, hard drives, NAS, and desktop systems keep chugging along without issue. I know of a Netgear consumer NAS I installed in 2011 is still running a business, untouched. It uses two enterprise drives in a mirror config.

ajsnigrutin 5y ago

Hardware loss is easily avoidable with more hardware... create backups, store them on multiple external drives (for most users, a single modern external drive is enough for all the backups, so the additional ones are for copies), and you can then keep one at home, one at work, one at your parents place, and replace them every now and then to 'refresh' the backups... Most home users don't need hourly or daily backups, they just nee their photo and home video collections saved, their documents scanned etc. With cloud providers, storing stuff in different datacenters doesn't help you, if you get banned from google, because someone didn't like your youtube comment. Multiple cloud providers might be using the same amazon/azure/... datacenter in the backend. Then there are different L0-L8 problems, from political, where your country gets embargoed due to some ongoing political struggle, or you can move to another location, where the internet is slow. And now, it seems that you can lose data due to stupidity as in the original post.

Tempest1981 5y ago
> easily avoidable > and you can then keep one at home, one at work, one at your parents place, and replace them every now and then to 'refresh' the backups Yes, many of us do this. But I wouldn't consider it "easy", esp for the average user. What % of home computer users own 2+ external drives? Or even 1? I bet it's under 20%.