3 ms·
Right now your only option would be to "buffer" packets of say 1 MByte in RAM, calculate the SHA-1, then store them as separate files in Backblaze B2. We do pl
by brianwski 11y ago
Right now your only option would be to "buffer" packets of say 1 MByte in RAM, calculate the SHA-1, then store them as separate files in Backblaze B2.
We do plan to add file offset access and larger file support very soon, so you would be able to append a 1 MByte chunk to an existing file in Backblaze with a SHA-1 of only the 1 MByte chunk. That should allow you to stream?
All great feedback, by the way. We really want to hear about these shortcomings in our API right away.
- aidenn0 11y ago> We do plan to add file offset access and larger file support very soon, so you would be able to append a 1 MByte chunk to an existing file in Backblaze with a SHA-1 of only the 1 MByte chunk. That should allow you to stream? Yeah, that would be sufficient.
- nickcw 11y agoHaving to know the SHA1 in advance would be a show stopper for rclone ( http://rclone.org http://rclone.org ) as it uses a streaming model internally (it can stream between cloud providers). Being able to append to a file in 1 MByte chunks (or larger) would be perfect - that is exactly the way Amazon S3 multipart uploads and google drive multipart uploads work.
- voltagex_ 11y agoI'm planning on storing files as encrypted X megabyte chunks (where X is TBD) - would calculating per-chunk and then uploading solve it? There's metadata support which could store original filenames etc.
- dfinlay 11y agoAdjustable chunk sizes would be beneficial as well.