4 ms·
> Generally speaking, it's very inefficient to deal with numerous small files because of latency I want to second this. Backblaze's Personal Backup uses the s
by brianwski 8y ago
> Generally speaking, it's very inefficient to deal with numerous small files because of latency
I want to second this. Backblaze's Personal Backup uses the same storage that B2 uses, and what we see is that tiny 10 byte files transmitted from "far away" (New Zealand to the United States) can be as slow as 15 kbits/sec transmission rate (yes, dial up modem speeds) because throughput is slaughtered by the latency to push each file.
Due to internet philosophies such as "TCP Slow Start" we do not see full throughput rates until after 40 MBytes in a single HTTPS POST if a customer is as far away as New Zealand.
One way we circumvent TCP Slow Start is to launch multiple threads. If TCP Slow Start results in only 1 Mbit/sec of throughput, we launch 20 threads to achieve 20 Mbits/sec of throughput. But you have to ask yourself what is going so horribly wrong on the internet if EVERYBODY has to write multiple threads on the same identical path through the internet in order to take advantage of the bandwidth available. Why not just allow 1 thread to get the full 20 Mbits/sec of throughput?