4 ms·

I don't like Backblaze because they require you to hand over your encryption key to their website to restore which kills any hope of it being a 0-knowledge solu

I don't like Backblaze because they require you to hand over your encryption key to their website to restore which kills any hope of it being a 0-knowledge solution. Right now I'm using Arq Backup + S3 and have been happy.

Trias11 5y ago
Same. Arq + Wasabi
1123581321 5y ago
I'm a fan of Arq and DIY storage, but how do you handle versioning?
- andrepew 5y ago
  Arq has baked in file versioning unless you mean something else?
  - 1123581321 5y ago
    Ah, of course. Sorry. :)
tzs 5y ago
I was about to use Backblaze Personal and then noticed that. Was then about to use Arq + Backblaze B2 when I realized that (1) Arq supports OneDrive, (2) I have a TB of OneDrive as part of my Microsoft 365 subscription, and (3) I only use cloud storage for file transfer between mobile devices and desktop, so my OneDrive space was almost all unused, so went with Arq + OneDrive. 3 years of weekly backups later and I've still only used about half of my 1 TB. The only thing I'm not happy about with Arq is that a "verify" downloads all the backup data to checksum it. That takes 3 or 4 days on my connection. I thought I read that many cloud storage provider APIs provide a way to ask the server for a checksum of a stored blob. I'd have expected Arq to make use of that, but maybe it is not reliable (the server might just report what the checksum is supposed to be, not actually read the data and checksum it?). Arq documents their storage format. I wonder if it would be possible to use a VM on Azure to access my OneDrive storage and do the checksumming on the VM?
- prirun 5y ago
  Yes, online storage services do store a checksum of the files. They can use that internally to periodically scrub their stored data. For example, if S3 has stored 3 copies of your file, they can use the hash to distinguish good and bad copies, and replace bad copies. For erasure-coded services like Backblaze B2, they can reconstruct a file using any 17 of 20 erasure segments, so with a known checksum they can figure out which segment is bad, delete it, and use the other segments to generate a replacement. I was told by Backblaze that they do periodically scrub B2 files. HashBackup (I'm the author) has a lot of ways to verify backups. For B2 uploads, it generates a SHA1 hash before sending the data, stores that in a local database, sends the data with the hash, B2 generates an independent hash of the data, and compares it to the one HB sent. If there is a mismatch, an upload error occurs and the upload is retried. On download, B2 sends the data and SHA1 hash (I'm assuming it verifies the hash during erasure reconstruction also), HB generates a SHA1 hash for the received data, verifies it against the SHA1 B2 sent, and verifies it against the SHA1 stored when the file was uploaded. After the upload, files can be verified by: - downloading the complete file as above, then verifying every block in the file - getting the file size and SHA1 hash from B2 and verifying those against the local database (no data download) - downloading N random samples from each file and verifying the block-level SHA1 for each sample - downloading a small percentage of the files every day to verify the backup over time (this can be done with whole files or random samples) If a problem is detected and you have other copies of the backup on another storage service or locally, HashBackup's selftest will fetch all copies of a file to reconstruct the original backup data to correct any errors, then upload the corrected file to all destinations.
  - WayToDoor 5y ago
    Thanks a lot for your work on HashBackup. I can't recommend it enough. It has been great and working for me for years, no data loss and multiple restores. Top notch documentation too, just a bit sad that it isn't open source so that I could contribute. But the quality is there and it has been really solid for me.
- oefrha 5y ago
  Arq backup to OneDrive used to be very slow since IIRC they were using an old version of the OneDrive API with pretty ridiculous latencies. How is it these days?
0xCMP 5y ago
I do the same, but with B2 which is nice cause it can be very cheap.
- farski 5y ago
  Could you give a ballpark figure for how much that costs you, and how much data you're dealing with?
  - 0xCMP 5y ago
    like $1-2/mn. For a very long time it was like $1-2/year. for roughly 188gb. has hourly incremental backups of certain files like all my source files (includes things like node modules and etc) i do a full-full-everyfile backup on occasion to a NAS.
anamexis 5y ago
I use Arq with B2, which seems like a happy compromise.
deadbunny 5y ago
Has it ever been marketed as zero knowledge?
summm 5y ago
How do they think this could be acceptable?
apple4ever 5y ago
I switched to Arq after a very bad customer service experience (one they easily could've solved but decided to be jerks about instead)

Trias11 5y ago

Same. Arq + Wasabi

1123581321 5y ago

I'm a fan of Arq and DIY storage, but how do you handle versioning?

andrepew 5y ago
Arq has baked in file versioning unless you mean something else?
- 1123581321 5y ago
  Ah, of course. Sorry. :)

tzs 5y ago

I was about to use Backblaze Personal and then noticed that. Was then about to use Arq + Backblaze B2 when I realized that (1) Arq supports OneDrive, (2) I have a TB of OneDrive as part of my Microsoft 365 subscription, and (3) I only use cloud storage for file transfer between mobile devices and desktop, so my OneDrive space was almost all unused, so went with Arq + OneDrive. 3 years of weekly backups later and I've still only used about half of my 1 TB. The only thing I'm not happy about with Arq is that a "verify" downloads all the backup data to checksum it. That takes 3 or 4 days on my connection. I thought I read that many cloud storage provider APIs provide a way to ask the server for a checksum of a stored blob. I'd have expected Arq to make use of that, but maybe it is not reliable (the server might just report what the checksum is supposed to be, not actually read the data and checksum it?). Arq documents their storage format. I wonder if it would be possible to use a VM on Azure to access my OneDrive storage and do the checksumming on the VM?

prirun 5y ago
Yes, online storage services do store a checksum of the files. They can use that internally to periodically scrub their stored data. For example, if S3 has stored 3 copies of your file, they can use the hash to distinguish good and bad copies, and replace bad copies. For erasure-coded services like Backblaze B2, they can reconstruct a file using any 17 of 20 erasure segments, so with a known checksum they can figure out which segment is bad, delete it, and use the other segments to generate a replacement. I was told by Backblaze that they do periodically scrub B2 files. HashBackup (I'm the author) has a lot of ways to verify backups. For B2 uploads, it generates a SHA1 hash before sending the data, stores that in a local database, sends the data with the hash, B2 generates an independent hash of the data, and compares it to the one HB sent. If there is a mismatch, an upload error occurs and the upload is retried. On download, B2 sends the data and SHA1 hash (I'm assuming it verifies the hash during erasure reconstruction also), HB generates a SHA1 hash for the received data, verifies it against the SHA1 B2 sent, and verifies it against the SHA1 stored when the file was uploaded. After the upload, files can be verified by: - downloading the complete file as above, then verifying every block in the file - getting the file size and SHA1 hash from B2 and verifying those against the local database (no data download) - downloading N random samples from each file and verifying the block-level SHA1 for each sample - downloading a small percentage of the files every day to verify the backup over time (this can be done with whole files or random samples) If a problem is detected and you have other copies of the backup on another storage service or locally, HashBackup's selftest will fetch all copies of a file to reconstruct the original backup data to correct any errors, then upload the corrected file to all destinations.
- WayToDoor 5y ago
  Thanks a lot for your work on HashBackup. I can't recommend it enough. It has been great and working for me for years, no data loss and multiple restores. Top notch documentation too, just a bit sad that it isn't open source so that I could contribute. But the quality is there and it has been really solid for me.
oefrha 5y ago
Arq backup to OneDrive used to be very slow since IIRC they were using an old version of the OneDrive API with pretty ridiculous latencies. How is it these days?

0xCMP 5y ago

I do the same, but with B2 which is nice cause it can be very cheap.

farski 5y ago
Could you give a ballpark figure for how much that costs you, and how much data you're dealing with?
- 0xCMP 5y ago
  like $1-2/mn. For a very long time it was like $1-2/year. for roughly 188gb. has hourly incremental backups of certain files like all my source files (includes things like node modules and etc) i do a full-full-everyfile backup on occasion to a NAS.

anamexis 5y ago

I use Arq with B2, which seems like a happy compromise.

deadbunny 5y ago

Has it ever been marketed as zero knowledge?

summm 5y ago

How do they think this could be acceptable?

apple4ever 5y ago

I switched to Arq after a very bad customer service experience (one they easily could've solved but decided to be jerks about instead)