3 ms·
Great post! I don't understand the paragraph about the cost of the PUTs and GETs though. The saving you get by batching writes seems marginal. Can you give some
by bentoner 18y ago
Great post! I don't understand the paragraph about the cost of the PUTs and GETs though. The saving you get by batching writes seems marginal. Can you give some numbers?
- andr 18y agoHow exactly do you batch writes? Do you save 2 blobs in 1 file? Or some kind of keep-alive that Amazon counts as a single request?
- cperciva 18y agoI aggregate blobs together (up to 4 MB at once) and store them to S3 as a single S3 object. The tarsnap server keeps track of which part of the object corresponds to which blob.
- cperciva 18y agoThe average block size the tarsnap server sees is about 30 kB (the tarsnap client tries to produce blocks of 64 kB on average, but then it compresses them individually before sending them to the server). This means that for every GB uploaded, there are about 33 thousand blocks. S3 PUTs cost $0.01 per thousand PUTs, so writing each of the blocks as an individual S3 object would cost $0.33 / GB for PUTs (plus the normal $0.10 / GB for bandwidth).
- jrockway 18y agoJust out of curiosity, why such small blocks?
- cperciva 18y agoThere are several advantages and disadvantages to different block sizes; but most significantly, larger blocks would make tarsnap less efficient at identifying duplicate data in the (very common) case where part of a file is modified. In the end it came down to weighing all the factors and picking a value which worked well.