3 ms·
If you use the large_file API (needed for multithreaded uploads), you do only hash chunks not the file.
by thisacctforreal 8y ago
If you use the large_file API (needed for multithreaded uploads), you do only hash chunks not the file.
- voltagex_ 8y agoOkay, I'll have to go back and have a look at some of the client libraries I tried - it may have been the machine I was using wasn't quick enough to hash ~30GB in a reasonable amount of time.
- brianwski 8y agoDisclaimer: I work at Backblaze. > If you use the large_file API (needed for multithreaded uploads) We recommend for small files that you use multi-threaded where each thread sends a totally separate file. So if you have to upload both cat.jpg and dog.jpg, you upload cat.jpg in one thread and dog.jpg in another thread. Based on the Backblaze architecture, that means cat.jpg will be sent to one "vault" in the Backblaze datacenter with one thread, and dog.jpg will be sent to a totally different "vault" in the Backblaze datacenter with another thread. This scales incredibly well, in that it should be twice as fast for two files, and 20 times as fast for 20 files if you do it correctly. Source: I wrote a lot of the Backblaze Personal Backup client, which uses this philosophy.