5 ms·

Disclaimer: I'm the original author of the blog post. :-) > In fact, S3's REST API requires callers to follow HTTP redirects... Yes. Let's take the example o

Disclaimer: I'm the original author of the blog post. :-) > In fact, S3's REST API requires callers to follow HTTP redirects... Yes. Let's take the example of uploading 3 small files. With S3, every upload call must start by hitting the same URL that is then redirected. To upload the same 3 small files in B2, it is a little different. For B2, one call is made to the "dispatch server" to ask for where there is spare space, then there is no redirect. The client must disconnect, and contact the final destination three times. So for this example, S3 would need to do 3 redirects (3 total network calls) and Backblaze B2 would do zero redirects but make 4 total network calls to upload three files. I hope that makes sense. I'm not saying one is better than the other, and I freely admit S3 can be a little more intuitive. But it saves Backblaze money to not have such high loads and high uptime demand on the original URL. > Again, this could be done directly with HTTP. Yes I agree it could have been. > There's no need to have a separate API call to get the "real" upload URL. The "need" was to save money. See example above. Amazon S3 would require 3 redirects from the original URL, while Backblaze B2 chose a different tradeoff that is zero redirects and 4 total requests where 3 out of 4 requests were NOT redirected. The 3 out of 4 requests go to the final location DIRECTLY. If you are only uploading one file, the Amazon S3 API is about as efficient as Backblaze and I agree with you. But if you upload 1 million files, the Backblaze B2 architecture is cheaper/simpler. But either way, it is the tradeoff we chose. > as of August 2015, us-standard... has read-after-write consistency for new objects TIL. We launched the B2 API before August of 2015, sorry if I propagated old info!

teraflop 8y ago
> For B2, one call is made to the "dispatch server" to ask for where there is spare space, then there is no redirect. An API call that retrieves the address of another machine to contact is functionally the same thing as an HTTP redirect, just with different syntax, right? > The client must disconnect, and contact the final destination three times. This seems like the key idea that wasn't quite explained in the blog post: the client is expected to cache and reuse the same "vault" server for future requests, not just for multiple pieces of a single upload. If the client doesn't conform to that expectation, your approach would end up with exactly the same performance characteristics as S3. What I find interesting is that unless I'm misunderstanding the API documentation [1], file downloads apparently do go through a load balancer. That is, you make an HTTP request to a single endpoint that's the same for all files in the account, and it fetches the data from whichever "vault" server is actually storing it. So does that mean that Backblaze's rate of incoming uploads is much larger than the rate of downloads? Otherwise it seems like you could just reuse the same load-balancing infrastructure for both. [1]: https://www.backblaze.com/b2/docs/b2_download_file_by_name.html https://www.backblaze.com/b2/docs/b2_download_file_by_name.h...
- brianwski 8y ago
  > An API call that retrieves the address of another machine to contact is functionally the same thing as an HTTP redirect If you are only uploading exactly one file -> yes. But if you are uploading 1 million files, a redirect system would result in 1 million redirects. The B2 system would result in less load on the dispatch server. > the client is expected to cache and reuse the same "vault" server for future requests Yes, exactly! This is the very core part of the B2 architecture. In fact, it is a waste of time and performance to keep asking the dispatch server for a location for every file. Just assume the last vault you got continues to be "valid" until the vault kicks you off with a 503. In B2 world, the 503 is NOT a fatal error, it means "go back to the dispatching server and ask for a new location to upload to". > file downloads apparently do go through a load balancer Correct. Because we wanted the ability to serve up static web content such as this picture (this is served out of B2): https://f001.backblazeb2.com/file/bucket9/cute3.jpg https://f001.backblazeb2.com/file/bucket9/cute3.jpg and do it in a highly available and highly scalable way if the content goes viral. The way we do that is we have load balanced "download servers" that also act as a caching layer. The FIRST time somebody fetches a file from the vault, the caching layer has to ask the vault to reassemble the file from parts, and then the caching layer caches it on fast SSDs inside the download servers. The second time (in 24 hours) that a customer requests the file, it comes out of the cache and not out of the vault. Otherwise, if a video went viral the vault would get crushed trying to reassemble the video for all 10 million views. :-) The load balancers we used are described in a different blog post here: https://www.backblaze.com/blog/load-balancing-and-b2-cloud-storage/ https://www.backblaze.com/blog/load-balancing-and-b2-cloud-s... > does that mean that Backblaze's rate of incoming uploads is much larger than the rate of downloads? Yes. VERY MUCH yes. Backblaze has more than 10x the incoming bandwidth as outgoing bandwidth. Backblaze started as an online backup company with the "Backblaze Personal Backup" solution. Online Backup is highly inbound heavy.
3pt14159 8y ago
I fully admit that 301s / 308s have problems in practice that means they need to be treated very carefully, but in theory they could be utilized here and you wouldn't need to do 6 requests, only 4. Though I really don't get what all this jerking off is about. I use B2. It took me like 2 fucking minutes to get working. It's ridiculously simple. My concern is more that they'll be acquired by some shitty company and I'll have to migrate off of them. It's not how long it takes me to setup an API.
- brianwski 8y ago
  Disclaimer: I work for Backblaze. > It took me like 2 fucking minutes to get working. It's ridiculously simple. I'm so glad you said that, thank you! Our confidence is sometimes shaken when a customer says they would prefer it to be S3 compatible. It honestly isn't that hard, I always wonder what the disconnect in communication is. > My concern is more that they'll be acquired by some shitty company Backblaze is not for sale. And definitely not for sale to shitty companies. And even for non-shitty companies Backblaze is not for sale at prices any company would probably want to spend (or could justify to their board). I don't know if this is common knowledge but Backblaze is employee owned. We never took any significant VC funding, so the only people with voting rights on the board of directors are the original 5 Backblaze founders (including myself). We like what we are doing and we COMPLETELY control our own destiny and work environment. Literally nobody (except customers) can tell us what to do, how to price our products, or what features to build. Backblaze is a fun company to work at, and we all make a good living. So it would be inordinately expensive to buy us out just to put us out of business and dissolve our tight-knit group here and ruin all our fun. Our plan is to stay independent forever. More background: the original 5 Backblaze founders have worked together for 19 years and have worked together at three startups. The first two starts were acquired by shitty companies. I do not want that to happen ever again, as long as I live. You know what it is like spending YEARS of your life building something beautiful, then hand it over to a shitty company and stand helplessly by while the beautiful product is first twisted, mangled, then withers and dies? It sucks. It sucks dead bunnies through a bent straw. So Backblaze was born different. Backblaze was born LEAN. The original founders and a few demi-partners going 18 months without salary, and another year on minimum wage, so that we wouldn't lose control of it. Employee-owned. Employee-run. Not. For. Sale.
willglynn 8y ago
> With S3, every upload call must start by hitting the same URL that is then redirected. Well… hold on. The docs say that S3 _can_ redirect PUTs and clients need to handle it correctly. This means a service could redirect every single incoming PUT and be S3-compatible. For example, this is how Internet Archive's S3ish service works: https://archive.org/help/abouts3.txt https://archive.org/help/abouts3.txt In practice, S3 doesn't seem to redirect very much at all, and I've never actually seen it redirect small objects. My speculation is that S3's frontend service is responsible for erasure coding and replicating inbound PUTs. Remember that S3's default storage class needs to send bits to multiple machines in multiple datacenters; it makes sense to me that this happens while you're uploading, instead of waiting for the upload to finish and then having the single point of failure replicate back out. S3's API gives the frontend service the _option_ to redirect each incoming PUT, but it almost always dispatches uploads to the internal storage systems with a single PUT from the client. > But it saves Backblaze money to not have such high loads and high uptime demand on the original URL. The endpoint that handles `b2_get_upload_url` still needs to be highly available and serve lots of requests. Its load isn't 1:1 for each uploaded object and its failure wouldn't immediately stop all uploads, but… it still has to work. > The "need" was to save money. See example above. Amazon S3 would require 3 redirects from the original URL, while Backblaze B2 chose a different tradeoff that is zero redirects and 4 total requests where 3 out of 4 requests were NOT redirected. The 3 out of 4 requests go to the final location DIRECTLY. How much does this help? A search of GitHub shows that many users are calling `b2_get_upload_url` and then uploading a single file, a use case for which B2 requires twice as many HTTP requests as a typical (non-redirecting) S3 PUT: https://github.com/UKn0Me/b2-api/blob/master/examples/b2_upload_file.php#L11-L12 https://github.com/UKn0Me/b2-api/blob/master/examples/b2_upl... https://github.com/volnt/e2e-upload/blob/master/project/upload/upload_endpoints.py#L30-L36 https://github.com/volnt/e2e-upload/blob/master/project/uplo... https://github.com/djoon/webboard/blob/master/src/com/webboard/main/web/backblazeb2/b2UploadFile.java#L71-L74 https://github.com/djoon/webboard/blob/master/src/com/webboa... https://github.com/jaeming/benjidalton.com-rails/blob/master/app/services/image_upload_service.rb#L14-L21 https://github.com/jaeming/benjidalton.com-rails/blob/master... > But either way, it is the tradeoff we chose. Sure, that's fine, but I think this is disingenuous: > And it’s a big deal that B2 is free of the “load balancer problem.” It solves for a huge scaling issue. When we roll out new vaults in new data centers in new countries, the clients are contacting those vaults DIRECTLY (over whatever network path is shortest) and so there are fewer choke points in our architecture. When you connect to s3.amazonaws.com, you're talking to the S3 frontend service in the us-standard region. This is the service that knows how to list your buckets (by talking to the internal index services), knows how to encode and distribute your incoming objects (by talking to the internal storage services), knows how to request/repair/decrypt/assemble your outgoing objects, etc. It's not in the way, it's not a "choke point", it is the _only_ part of S3 that performs these frontend functions. When S3 adds a new region, that region gets a new endpoint: s3.us-east-2.amazonaws.com, s3.us-west-1.amazonaws.com, s3.eu-west-1.amazonaws.com, etc. all perform the same functions for their respective geographical areas. And if you request something from the wrong region: $ curl -s https://s3.us-west-1.amazonaws.com/elevation-tiles-prod | tidy -q -xml <?xml version="1.0" encoding="utf-8"?> <Error> <Code>PermanentRedirect</Code> <Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message> <Bucket>elevation-tiles-prod</Bucket> <Endpoint>s3.amazonaws.com</Endpoint> <RequestId>D18584530CD6A9E1</RequestId> <HostId> N7Oa6U6FouXBvz40Wa920r+D2qj5nWUZ8nxP14L+HhqfbEX5jeR+Yo80boODWqVxUFVeCeZVBqw=</HostId> </Error> …you're told which regional endpoint you should use instead.

teraflop 8y ago

> For B2, one call is made to the "dispatch server" to ask for where there is spare space, then there is no redirect. An API call that retrieves the address of another machine to contact is functionally the same thing as an HTTP redirect, just with different syntax, right? > The client must disconnect, and contact the final destination three times. This seems like the key idea that wasn't quite explained in the blog post: the client is expected to cache and reuse the same "vault" server for future requests, not just for multiple pieces of a single upload. If the client doesn't conform to that expectation, your approach would end up with exactly the same performance characteristics as S3. What I find interesting is that unless I'm misunderstanding the API documentation [1], file downloads apparently do go through a load balancer. That is, you make an HTTP request to a single endpoint that's the same for all files in the account, and it fetches the data from whichever "vault" server is actually storing it. So does that mean that Backblaze's rate of incoming uploads is much larger than the rate of downloads? Otherwise it seems like you could just reuse the same load-balancing infrastructure for both. [1]: https://www.backblaze.com/b2/docs/b2_download_file_by_name.html https://www.backblaze.com/b2/docs/b2_download_file_by_name.h...

brianwski 8y ago
> An API call that retrieves the address of another machine to contact is functionally the same thing as an HTTP redirect If you are only uploading exactly one file -> yes. But if you are uploading 1 million files, a redirect system would result in 1 million redirects. The B2 system would result in less load on the dispatch server. > the client is expected to cache and reuse the same "vault" server for future requests Yes, exactly! This is the very core part of the B2 architecture. In fact, it is a waste of time and performance to keep asking the dispatch server for a location for every file. Just assume the last vault you got continues to be "valid" until the vault kicks you off with a 503. In B2 world, the 503 is NOT a fatal error, it means "go back to the dispatching server and ask for a new location to upload to". > file downloads apparently do go through a load balancer Correct. Because we wanted the ability to serve up static web content such as this picture (this is served out of B2): https://f001.backblazeb2.com/file/bucket9/cute3.jpg https://f001.backblazeb2.com/file/bucket9/cute3.jpg and do it in a highly available and highly scalable way if the content goes viral. The way we do that is we have load balanced "download servers" that also act as a caching layer. The FIRST time somebody fetches a file from the vault, the caching layer has to ask the vault to reassemble the file from parts, and then the caching layer caches it on fast SSDs inside the download servers. The second time (in 24 hours) that a customer requests the file, it comes out of the cache and not out of the vault. Otherwise, if a video went viral the vault would get crushed trying to reassemble the video for all 10 million views. :-) The load balancers we used are described in a different blog post here: https://www.backblaze.com/blog/load-balancing-and-b2-cloud-storage/ https://www.backblaze.com/blog/load-balancing-and-b2-cloud-s... > does that mean that Backblaze's rate of incoming uploads is much larger than the rate of downloads? Yes. VERY MUCH yes. Backblaze has more than 10x the incoming bandwidth as outgoing bandwidth. Backblaze started as an online backup company with the "Backblaze Personal Backup" solution. Online Backup is highly inbound heavy.

3pt14159 8y ago

I fully admit that 301s / 308s have problems in practice that means they need to be treated very carefully, but in theory they could be utilized here and you wouldn't need to do 6 requests, only 4. Though I really don't get what all this jerking off is about. I use B2. It took me like 2 fucking minutes to get working. It's ridiculously simple. My concern is more that they'll be acquired by some shitty company and I'll have to migrate off of them. It's not how long it takes me to setup an API.

brianwski 8y ago
Disclaimer: I work for Backblaze. > It took me like 2 fucking minutes to get working. It's ridiculously simple. I'm so glad you said that, thank you! Our confidence is sometimes shaken when a customer says they would prefer it to be S3 compatible. It honestly isn't that hard, I always wonder what the disconnect in communication is. > My concern is more that they'll be acquired by some shitty company Backblaze is not for sale. And definitely not for sale to shitty companies. And even for non-shitty companies Backblaze is not for sale at prices any company would probably want to spend (or could justify to their board). I don't know if this is common knowledge but Backblaze is employee owned. We never took any significant VC funding, so the only people with voting rights on the board of directors are the original 5 Backblaze founders (including myself). We like what we are doing and we COMPLETELY control our own destiny and work environment. Literally nobody (except customers) can tell us what to do, how to price our products, or what features to build. Backblaze is a fun company to work at, and we all make a good living. So it would be inordinately expensive to buy us out just to put us out of business and dissolve our tight-knit group here and ruin all our fun. Our plan is to stay independent forever. More background: the original 5 Backblaze founders have worked together for 19 years and have worked together at three startups. The first two starts were acquired by shitty companies. I do not want that to happen ever again, as long as I live. You know what it is like spending YEARS of your life building something beautiful, then hand it over to a shitty company and stand helplessly by while the beautiful product is first twisted, mangled, then withers and dies? It sucks. It sucks dead bunnies through a bent straw. So Backblaze was born different. Backblaze was born LEAN. The original founders and a few demi-partners going 18 months without salary, and another year on minimum wage, so that we wouldn't lose control of it. Employee-owned. Employee-run. Not. For. Sale.

willglynn 8y ago

> With S3, every upload call must start by hitting the same URL that is then redirected. Well… hold on. The docs say that S3 _can_ redirect PUTs and clients need to handle it correctly. This means a service could redirect every single incoming PUT and be S3-compatible. For example, this is how Internet Archive's S3ish service works: https://archive.org/help/abouts3.txt https://archive.org/help/abouts3.txt In practice, S3 doesn't seem to redirect very much at all, and I've never actually seen it redirect small objects. My speculation is that S3's frontend service is responsible for erasure coding and replicating inbound PUTs. Remember that S3's default storage class needs to send bits to multiple machines in multiple datacenters; it makes sense to me that this happens while you're uploading, instead of waiting for the upload to finish and then having the single point of failure replicate back out. S3's API gives the frontend service the _option_ to redirect each incoming PUT, but it almost always dispatches uploads to the internal storage systems with a single PUT from the client. > But it saves Backblaze money to not have such high loads and high uptime demand on the original URL. The endpoint that handles `b2_get_upload_url` still needs to be highly available and serve lots of requests. Its load isn't 1:1 for each uploaded object and its failure wouldn't immediately stop all uploads, but… it still has to work. > The "need" was to save money. See example above. Amazon S3 would require 3 redirects from the original URL, while Backblaze B2 chose a different tradeoff that is zero redirects and 4 total requests where 3 out of 4 requests were NOT redirected. The 3 out of 4 requests go to the final location DIRECTLY. How much does this help? A search of GitHub shows that many users are calling `b2_get_upload_url` and then uploading a single file, a use case for which B2 requires twice as many HTTP requests as a typical (non-redirecting) S3 PUT: https://github.com/UKn0Me/b2-api/blob/master/examples/b2_upload_file.php#L11-L12 https://github.com/UKn0Me/b2-api/blob/master/examples/b2_upl... https://github.com/volnt/e2e-upload/blob/master/project/upload/upload_endpoints.py#L30-L36 https://github.com/volnt/e2e-upload/blob/master/project/uplo... https://github.com/djoon/webboard/blob/master/src/com/webboard/main/web/backblazeb2/b2UploadFile.java#L71-L74 https://github.com/djoon/webboard/blob/master/src/com/webboa... https://github.com/jaeming/benjidalton.com-rails/blob/master/app/services/image_upload_service.rb#L14-L21 https://github.com/jaeming/benjidalton.com-rails/blob/master... > But either way, it is the tradeoff we chose. Sure, that's fine, but I think this is disingenuous: > And it’s a big deal that B2 is free of the “load balancer problem.” It solves for a huge scaling issue. When we roll out new vaults in new data centers in new countries, the clients are contacting those vaults DIRECTLY (over whatever network path is shortest) and so there are fewer choke points in our architecture. When you connect to s3.amazonaws.com, you're talking to the S3 frontend service in the us-standard region. This is the service that knows how to list your buckets (by talking to the internal index services), knows how to encode and distribute your incoming objects (by talking to the internal storage services), knows how to request/repair/decrypt/assemble your outgoing objects, etc. It's not in the way, it's not a "choke point", it is the _only_ part of S3 that performs these frontend functions. When S3 adds a new region, that region gets a new endpoint: s3.us-east-2.amazonaws.com, s3.us-west-1.amazonaws.com, s3.eu-west-1.amazonaws.com, etc. all perform the same functions for their respective geographical areas. And if you request something from the wrong region: $ curl -s https://s3.us-west-1.amazonaws.com/elevation-tiles-prod | tidy -q -xml <?xml version="1.0" encoding="utf-8"?> <Error> <Code>PermanentRedirect</Code> <Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message> <Bucket>elevation-tiles-prod</Bucket> <Endpoint>s3.amazonaws.com</Endpoint> <RequestId>D18584530CD6A9E1</RequestId> <HostId> N7Oa6U6FouXBvz40Wa920r+D2qj5nWUZ8nxP14L+HhqfbEX5jeR+Yo80boODWqVxUFVeCeZVBqw=</HostId> </Error> …you're told which regional endpoint you should use instead.