4 ms·
The per-request fees ($.01 per 1000 PUTs and $.01 per 10,000 GETs) aren't a problem for 2 reasons: 1) Arq packs small files (under 64k) together into "packs",
by sreitshamer 17y ago
The per-request fees ($.01 per 1000 PUTs and $.01 per 10,000 GETs) aren't a problem for 2 reasons:
1) Arq packs small files (under 64k) together into "packs", a bit like git does. So instead of thousands of PUTs of tiny files (which would also be very slow) it does fewer larger PUTs.
2) Arq caches indexes of those pack files locally, and caches the list of larger S3 objects as well. So it doesn't have to do a GET to figure out whether an object with a given SHA1 is already in the S3 account. (Arq stores every object by its SHA1 hash, like git does).
The file format is open (it's your data after all): http://www.haystacksoftware.com/arq/s3_data_format.txt http://www.haystacksoftware.com/arq/s3_data_format.txt
- cperciva 17y agoArq packs small files (under 64k) together into "packs"... Ok, so if you have lots of 64 kB files they're all PUT separately, using 15625 PUTs/GB (i.e., $0.15/GB)? I guess that's not too bad. The file format is open Nice! Good to see someone who believes in openness. (How about the source code, so that people can confirm that you're doing what the spec says?) One thing caught my eye glancing through that document, though: "The name of each blob is its SHA1 hash." That's the SHA1 hash of the encrypted data, right? (If not, it definitely should be!)
- sreitshamer 17y agoOk, so if you have lots of 64 kB files they're all PUT separately, using 15625 PUTs/GB (i.e., $0.15/GB)? I guess that's not too bad. I guess I could ask you to choose the threshold instead of picking 64k, but generally I've tried to choose reasonable defaults so that the user doesn't have to make those kinds of decisions. Good to see someone who believes in openness. (How about the source code, so that people can confirm that you're doing what the spec says?) Well, you could actually write your own code to read the files, since the format is open. That's the SHA1 hash of the encrypted data, right? It's the SHA1 of the contents of the object, and the object is encrypted data, so yes.
- statictype 17y ago>Well, you could actually write your own code to read the files, since the format is open. I think the parent's point was not vendor lock-in but rather, that publishing the source code to your particular implementation would lend more credibility to it since anyone can inspect it and verify that it does what it says it does.
- sreitshamer 17y agoRight, but with Arq, unlike most online backup offerings, you have direct access to the files being stored online (they're in your S3 account). So you can verify that Arq says what it does because you can look at the results.
- DavidSJ 17y agoWith source code, you can reason about edge cases. No matter how many tests you do, you can never catch all of these.
- towndrunk 17y agoHonestly, I don't mind the extra fees as long as I can restore successfully. I just want a solution that works for backup and restore with out issue.