3 ms·

> And customers may or may not have the RAM required to hold a 10 GByte file in RAM, and none of them have enough RAM required to hold a 500 GByte file in RAM.

> And customers may or may not have the RAM required to hold a 10 GByte file in RAM, and none of them have enough RAM required to hold a 500 GByte file in RAM. Backblaze needs to store these "snapshots" someplace, so it stores them on disk. But at the same time, customers are very likely to have enough RAM to hold a 100 MB file, and definitely have enough for a 10 MB file in RAM. It might be better (faster, easier, less abusive to the drive) to store small files in RAM (for some well-chosen value of “small”, of course. Likely one that depends on the system RAM size and bandwidth.)

brianwski 6y ago
> to store small files in RAM (for some well-chosen value of “small”, of course... The Backblaze client code currently puts that dividing line at 100 MBytes. Any file less than 100 MBytes we call a "small file" and those don't get subdivided into 10 MByte chunks on disk, they get slurped up into RAM and just held there for the entire duration of being sent. For that code path, Backblaze can be configured to send up to 30 of these 100 MByte files in 30 separate threads which means (if the customer configures it this way) it can eat up 3 GBytes of RAM at peak. If you have that much RAM, it can really rip. If you don't, we recommend using fewer threads. :-) The sub-division into 10 MByte chunks taking a one time "Snapshot" of the large file is for all files larger than 100 MBytes.