3 ms·

> For B2, one call is made to the "dispatch server" to ask for where there is spare space, then there is no redirect. An API call that retrieves the address of

> For B2, one call is made to the "dispatch server" to ask for where there is spare space, then there is no redirect. An API call that retrieves the address of another machine to contact is functionally the same thing as an HTTP redirect, just with different syntax, right? > The client must disconnect, and contact the final destination three times. This seems like the key idea that wasn't quite explained in the blog post: the client is expected to cache and reuse the same "vault" server for future requests, not just for multiple pieces of a single upload. If the client doesn't conform to that expectation, your approach would end up with exactly the same performance characteristics as S3. What I find interesting is that unless I'm misunderstanding the API documentation [1], file downloads apparently do go through a load balancer. That is, you make an HTTP request to a single endpoint that's the same for all files in the account, and it fetches the data from whichever "vault" server is actually storing it. So does that mean that Backblaze's rate of incoming uploads is much larger than the rate of downloads? Otherwise it seems like you could just reuse the same load-balancing infrastructure for both. [1]: https://www.backblaze.com/b2/docs/b2_download_file_by_name.html https://www.backblaze.com/b2/docs/b2_download_file_by_name.h...

brianwski 8y ago
> An API call that retrieves the address of another machine to contact is functionally the same thing as an HTTP redirect If you are only uploading exactly one file -> yes. But if you are uploading 1 million files, a redirect system would result in 1 million redirects. The B2 system would result in less load on the dispatch server. > the client is expected to cache and reuse the same "vault" server for future requests Yes, exactly! This is the very core part of the B2 architecture. In fact, it is a waste of time and performance to keep asking the dispatch server for a location for every file. Just assume the last vault you got continues to be "valid" until the vault kicks you off with a 503. In B2 world, the 503 is NOT a fatal error, it means "go back to the dispatching server and ask for a new location to upload to". > file downloads apparently do go through a load balancer Correct. Because we wanted the ability to serve up static web content such as this picture (this is served out of B2): https://f001.backblazeb2.com/file/bucket9/cute3.jpg https://f001.backblazeb2.com/file/bucket9/cute3.jpg and do it in a highly available and highly scalable way if the content goes viral. The way we do that is we have load balanced "download servers" that also act as a caching layer. The FIRST time somebody fetches a file from the vault, the caching layer has to ask the vault to reassemble the file from parts, and then the caching layer caches it on fast SSDs inside the download servers. The second time (in 24 hours) that a customer requests the file, it comes out of the cache and not out of the vault. Otherwise, if a video went viral the vault would get crushed trying to reassemble the video for all 10 million views. :-) The load balancers we used are described in a different blog post here: https://www.backblaze.com/blog/load-balancing-and-b2-cloud-storage/ https://www.backblaze.com/blog/load-balancing-and-b2-cloud-s... > does that mean that Backblaze's rate of incoming uploads is much larger than the rate of downloads? Yes. VERY MUCH yes. Backblaze has more than 10x the incoming bandwidth as outgoing bandwidth. Backblaze started as an online backup company with the "Backblaze Personal Backup" solution. Online Backup is highly inbound heavy.

brianwski 8y ago

> An API call that retrieves the address of another machine to contact is functionally the same thing as an HTTP redirect If you are only uploading exactly one file -> yes. But if you are uploading 1 million files, a redirect system would result in 1 million redirects. The B2 system would result in less load on the dispatch server. > the client is expected to cache and reuse the same "vault" server for future requests Yes, exactly! This is the very core part of the B2 architecture. In fact, it is a waste of time and performance to keep asking the dispatch server for a location for every file. Just assume the last vault you got continues to be "valid" until the vault kicks you off with a 503. In B2 world, the 503 is NOT a fatal error, it means "go back to the dispatching server and ask for a new location to upload to". > file downloads apparently do go through a load balancer Correct. Because we wanted the ability to serve up static web content such as this picture (this is served out of B2): https://f001.backblazeb2.com/file/bucket9/cute3.jpg https://f001.backblazeb2.com/file/bucket9/cute3.jpg and do it in a highly available and highly scalable way if the content goes viral. The way we do that is we have load balanced "download servers" that also act as a caching layer. The FIRST time somebody fetches a file from the vault, the caching layer has to ask the vault to reassemble the file from parts, and then the caching layer caches it on fast SSDs inside the download servers. The second time (in 24 hours) that a customer requests the file, it comes out of the cache and not out of the vault. Otherwise, if a video went viral the vault would get crushed trying to reassemble the video for all 10 million views. :-) The load balancers we used are described in a different blog post here: https://www.backblaze.com/blog/load-balancing-and-b2-cloud-storage/ https://www.backblaze.com/blog/load-balancing-and-b2-cloud-s... > does that mean that Backblaze's rate of incoming uploads is much larger than the rate of downloads? Yes. VERY MUCH yes. Backblaze has more than 10x the incoming bandwidth as outgoing bandwidth. Backblaze started as an online backup company with the "Backblaze Personal Backup" solution. Online Backup is highly inbound heavy.