5 ms·

Are you able to try multithreaded uploads too? I found that single stream uploads were too slow (< 10 megabytes per second) but I could get ~35 megabytes per se

Are you able to try multithreaded uploads too? I found that single stream uploads were too slow (< 10 megabytes per second) but I could get ~35 megabytes per second from packet.net to B2 by using 4 threads. Edit: removed incorrect stuff.

thisacctforreal 8y ago
If you use the large_file API (needed for multithreaded uploads), you do only hash chunks not the file.
- voltagex_ 8y ago
  Okay, I'll have to go back and have a look at some of the client libraries I tried - it may have been the machine I was using wasn't quick enough to hash ~30GB in a reasonable amount of time.
- brianwski 8y ago
  Disclaimer: I work at Backblaze. > If you use the large_file API (needed for multithreaded uploads) We recommend for small files that you use multi-threaded where each thread sends a totally separate file. So if you have to upload both cat.jpg and dog.jpg, you upload cat.jpg in one thread and dog.jpg in another thread. Based on the Backblaze architecture, that means cat.jpg will be sent to one "vault" in the Backblaze datacenter with one thread, and dog.jpg will be sent to a totally different "vault" in the Backblaze datacenter with another thread. This scales incredibly well, in that it should be twice as fast for two files, and 20 times as fast for 20 files if you do it correctly. Source: I wrote a lot of the Backblaze Personal Backup client, which uses this philosophy.
gsibble 8y ago
I haven't added multi threaded stuff yet because I wanted it to be compatible with single threaded web servers like flask and django. I can and will add it if you want to add an issue.
- voltagex_ 8y ago
  OK - if that's your target I'm not going to hassle you. I'm probably going to choose something written in Golang for other reasons in the end.
  - gsibble 8y ago
    I'm definitely going to add multi-threading. I wrote this library in 4 days (look at the commit history). It was just something fun to do.

thisacctforreal 8y ago

If you use the large_file API (needed for multithreaded uploads), you do only hash chunks not the file.

voltagex_ 8y ago
Okay, I'll have to go back and have a look at some of the client libraries I tried - it may have been the machine I was using wasn't quick enough to hash ~30GB in a reasonable amount of time.
brianwski 8y ago
Disclaimer: I work at Backblaze. > If you use the large_file API (needed for multithreaded uploads) We recommend for small files that you use multi-threaded where each thread sends a totally separate file. So if you have to upload both cat.jpg and dog.jpg, you upload cat.jpg in one thread and dog.jpg in another thread. Based on the Backblaze architecture, that means cat.jpg will be sent to one "vault" in the Backblaze datacenter with one thread, and dog.jpg will be sent to a totally different "vault" in the Backblaze datacenter with another thread. This scales incredibly well, in that it should be twice as fast for two files, and 20 times as fast for 20 files if you do it correctly. Source: I wrote a lot of the Backblaze Personal Backup client, which uses this philosophy.

gsibble 8y ago

I haven't added multi threaded stuff yet because I wanted it to be compatible with single threaded web servers like flask and django. I can and will add it if you want to add an issue.

voltagex_ 8y ago
OK - if that's your target I'm not going to hassle you. I'm probably going to choose something written in Golang for other reasons in the end.
- gsibble 8y ago
  I'm definitely going to add multi-threading. I wrote this library in 4 days (look at the commit history). It was just something fun to do.