16 ms·

Design Thinking: B2 APIs and the Hidden Costs of S3 Compatibility

67 points by Manozco 8y ago

metalrain 8y ago
It's great that cost of elasticity is not hidden. I'm glad that there are alternatives.
vpribish 8y ago
"Design Thinking" >>cringe<<
elFarto 8y ago
I wonder if you could get the same functionality of AWS, with the same implementation of B2, by having a single URL to POST files to, that simply sent a redirect to the correct location (apparently there's the 307 HTTP status code for exactly this). E.g: => POST https://upload.backblaze.com/bucket/file <= 307 redirect to https://pod-000-1007-13.backblaze.com/b2api/v1/b2_upload_file/... => POST https://pod-000-1007-13.backblaze.com/b2api/v1/b2_upload_file/...
- BillinghamJ 8y ago
  I'm not sure you could prevent the client from providing the body before you're able to stop it and tell it to redirect instead - so there's a high risk you end up with everything being uploaded twice.
  - willglynn 8y ago
    `Expect: 100-continue` lets the server control this. If it responds "100 Continue", you upload the body. If it responds with a redirect, you redirect without uploading the body. This is standard HTTP/1.1, and it's used by the S3 API for PUT requests.
- user5994461 8y ago
  Strictly speaking, it's possible but it's not reliable and it shouldn't be used. A redirect on a POST never results into another POST with the same content. - API typically don't follow redirects (without special flags). Too much risk to cause damages by doing repeated calls. - Browsers follow the redirect by a GET with no data. It's typical for a completed form to redirect to another page, do not re submit the form there. There are some settings and flags to alter the behavior into what you describe but it's not a good idea to go there. It will definitely not work out of the box with much of anything.
  - willglynn 8y ago
    There are different kinds of redirects. See the discussion of 303 See Other vs 307 Temporary Redirect in RFC 7231: https://tools.ietf.org/html/rfc7231#section-6.4.4 https://tools.ietf.org/html/rfc7231#section-6.4.4 https://tools.ietf.org/html/rfc7231#section-6.4.7 https://tools.ietf.org/html/rfc7231#section-6.4.7 303 means "GET this new URL" while 307 means "resend your request at this URL without changing the verb or body".
    - user5994461 8y ago
      And you can see that the RFC only talks in terms of should and may. It's beyond basic HTTP functionality, the RFC is basically moot at this stage and you're dealing in implementation specific details. You will have to work individually on every client you plan to support, both browsers and libraries. It can be done but it's not necessarily a good idea.
  - SahAssar 8y ago
    307 & 308 specifically say to follow the redirect with the same method and body.
    - user5994461 8y ago
      I invite you to try it in your favorite browser interactively, then in ajax , then curl, then python requests, then wget, then whichever software and libraries are available to you.
deedubaya 8y ago
That's all fine and good, I don't care if you're S3 compatible or not.... I do care if I have to write my own API client for your storage backend. Or if you have examples to go off of. Backblaze doesn't seem to offer either for non-C++/Swift languages. Complete non-starter. The, perhaps obvious, win of being S3 compatible is that you open the door to thousands of existing S3 clients already implemented in my different technologies, for free. And you get the developers who use them as customers.
- blahblahblogger 8y ago
  That's kind of what minio is: https://github.com/minio/minio https://github.com/minio/minio
  - deedubaya 8y ago
    Yup, and there a language specific sdk's that provide a consistent API w/adapters for different cloud providers which can be swapped out. i.e. active_storage, fog in ruby
- deleted 8y ago
  [deleted]
- brianwski 8y ago
  Disclaimer: I work at Backblaze. > Backblaze doesn't seem to offer either for non-C++/Swift languages. On each of the API web pages there are code examples for the following languages: 1) cUrl, 2) Java, 3) Python, 4) Swift, 5) Ruby, 6) C#, and 7) PHP. For example, go to https://www.backblaze.com/b2/docs/b2_authorize_account.html https://www.backblaze.com/b2/docs/b2_authorize_account.html and scroll ALL THE WAY TO THE BOTTOM of that web page, and you should see "Sample Code" section. Click on the blue buttons to see the different code examples. If your favorite language is missing, we can add it for you! One of our client engineers wrote most of the code examples in all 7 languages in less than 1 week. With these working code examples, the expectation is you should be able to get B2 working in literally less than 2 days in any application in any language.
misterbowfinger 8y ago
Honestly surprised that AWS, GCP, or Azure haven't acquired BackBlaze by now. Seems like an obvious move.
- BillinghamJ 8y ago
  Why would they? They already have all of this technology and it's not like BackBlaze have a massive number of customers they could acquire.
  - bcheung 8y ago
    Cost is a big one. Compared to Digital Ocean Spaces, OVH's Object Store, or Blackblaze, AWS pricing is an order of magnitude more expensive.
    - jorams 8y ago
      Backblaze's technology is not that revolutionary. AWS is an order of magnitude more expensive because they want to, not because they need to be.
- brianwski 8y ago
  Disclaimer: I am the author of the blog post and work at Backblaze. > surprised that AWS, GCP, or Azure haven't acquired Backblaze by now Sometimes press/customers/people ask us how many customers has Backblaze "converted over from" S3 to B2. To be honest, I think the answer is approximately zero. If a customer already has a Petabyte uploaded into S3, it is simply not practical or cost effective to download that from S3 and import it into B2. The S3 download costs are extremely expensive (9 cents per GByte to download out of S3, vs 1 cent per GByte to download out of B2). Most of Backblaze's B2 customers fall into one of two camps: 1) New customers starting out that have just started to look at cloud storage and need to decide where to put their data. They look at S3, compare with B2, and make a decision and Backblaze B2 gets some of those customers. Realistically we don't even get half of these new customers either because B2 is missing a feature the potential customer needs, or because the customer has not even heard of Backblaze. So Backblaze is probably not causing Amazon too much lost revenue yet. 2) Multi-cloud customers. Any one "cloud provider" can have outages, including Backblaze and including Amazon. So if you store an entire copy of your data in Amazon S3, and also store an entire copy in Backblaze B2, you will (by definition) have both more durable data and higher availability data than a single copy in either one alone. By definition this doesn't actually cost Amazon S3 any sales, because you STILL need a complete copy in S3!! For multi-cloud, Backblaze B2 doesn't harm Amazon one bit. > Seems like an obvious move (for Amazon to acquire Backblaze). Amazon has never offered to buy Backblaze, but also we are not for sale. Or at very least not at the prices Amazon would probably want to spend (or could justify to their board). I don't know if this is common knowledge but Backblaze is employee owned. We never took any significant VC funding, so the only people with voting rights on the board of directors are the original 5 Backblaze founders (including myself). We like what we are doing and we COMPLETELY control our own destiny and work environment. Literally nobody (except customers) can tell us what to do, how to price our products, or what features to build. Backblaze is a fun company to work at, and we all make a good living. So it would be inordinately expensive to buy us out just to put us out of business and dissolve our tight-knit group here and ruin all our fun. Our plan is to stay independent forever.
willglynn 8y ago
This article contains some misunderstandings about the S3 API. > The interface to upload data into Amazon S3 is actually a bit simpler than Backblaze B2’s API. But it comes at a literal cost. It requires Amazon to have a massive and expensive choke point in their network: load balancers. When a customer tries to upload to S3, she is given a single upload URL to use. For instance, http://s3.amazonaws.com/<bucketname> http://s3.amazonaws.com/<bucketname>. This is great for the customer as she can just start pushing data to the URL. But that requires Amazon to be able to take that data and then, in a second step behind the scenes, find available storage space and then push that data to that available location. The second step creates a choke point as it requires having high bandwidth load balancers. That, in turn, carries a significant customer implication; load balancers cost significant money. In fact, S3's REST API requires callers to follow HTTP redirects, and the PUT documentation expressly mentions the HTTP "Expect: 100-continue" mechanism precisely so that the S3 endpoint you reach in your initial PUT request does not have to handle the HTTP request body. https://docs.aws.amazon.com/AmazonS3/latest/dev/Redirects.html https://docs.aws.amazon.com/AmazonS3/latest/dev/Redirects.ht... https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPU... > The Dispatching Server (the API server answering the b2_get_upload_url call) tells the Client “there is space over on “Vault-8329.” This next step is our magic. Armed with the knowledge of the open vault, the Client ends its connection with the Dispatching Server and creates a brand new request DIRECTLY to Vault-8329 (calling b2_upload_file or b2_upload_part). No load balancers involved! Again, this could be done directly with HTTP. PUT to the first server, receive a redirect, PUT to vault-8329, receive "100 Continue", transmit file. There's no need to have a separate API call to get the "real" upload URL. > 3) Expensive, time consuming data copy needs (and “eventual consistency”). Amazon S3 requires the copying of massive amounts of data from one part of their network (the upload server) to wherever the data’s ultimate resting place will be. This is at the root of one of the biggest frustrations when dealing with S3: Amazon’s “eventual consistency.” Wait, I thought they were load balancers? Why does the load balancer need to copy any data once it's done uploading? As for eventual consistency, there is truth to this complaint -- but much less truth than in the distant past. Every S3 region except us-standard has always had read-after-write consistency for new objects since launch, and as of August 2015, us-standard does too: https://aws.amazon.com/about-aws/whats-new/2015/08/amazon-s3-introduces-new-usability-enhancements/ https://aws.amazon.com/about-aws/whats-new/2015/08/amazon-s3... If your PUT returns 200 OK, a subsequent GET will return the object, assuming you're using unique keys. This prevents the 2015-and-earlier problem where you'd create a new S3 object and enqueue a job to process it, then the job gets 404 Not Found while retrieving the new object. There are other cases where S3's eventual consistency can be an issue, but none of them have been dealbreakers for my applications. Having said that: S3's consistency model is a weaker than the model B2 provides, so this is not an argument against providing an S3-compatible interface.
bcheung 8y ago
Anyone know why Amazon didn't adopt existing standards like SCP / SFTP / WebDAV? I've always found the S3 APIs to be difficult to work with, especially for authorization and large uploads.
- blasdel 8y ago
  S3 buckets do not have filesystem semantics so most of that functionality would be inoperable, and also require heavyweight session-based proxies to support (which is something you can easily operate yourself)
- yzmtf2008 8y ago
  Because S3 is a K/V Store, not a file system.
  - bcheung 8y ago
    File systems can be used as key value as well. The Key is the path, and the value is the contents of the file. Most modern filesystems also have metadata. Granted it's not hierarchical, so listing a bucket lists everything in every folder, but I don't see why that's too big of a concern. Especially since there are pseudo directories in many of the S3 tools. Many people use S3 and have a folder-like hierarchical naming convention anyways. With WebDAV, the URL is the key, and the value is the contents of the upload / download.
- halbritt 8y ago
  SCP, SFTP, and WebDAV are all woefully inefficient for large data transfers as a result of bandwidth delay product.* They typically use a single TCP session and transmit files serially. WebDAV does support an mput, however. S3 does a much better job of utilizing available bandwidth. It supports multipart parallel uploads and resumes nicely. In my experience, it will usually fully saturate any available upload capacity. There are other protocols that may be more efficient for high bandwidth, high latency connections, i.e. WDT from facebook or UDT, but S3 does okay. *https://en.wikipedia.org/wiki/Bandwidth-delay_product https://en.wikipedia.org/wiki/Bandwidth-delay_product
- 8y ago
deepsun 8y ago
That "get_upload_url()" trick they invented was in AppEngine's BlobStore since 2008. Although Google deprecated it in favor of GCS.
hemancuso 8y ago
It’s a bit unclear to me what is so expensive about the load balancing nodes. Care that explain why it’s substantially more than a few round robin’d smart reverse proxies moving data to the correct storage node? With S3/Dynamo design the back end destination is largely known from the hash ring. Also- Wasabi has fantastic pricing and full s3 compatibility.
- zzzcpan 8y ago
  > It’s a bit unclear to me what is so expensive about the load balancing nodes. They also use Reed-Solomon and split data into multiple pieces to store on multiple servers. So they need all those "load balancing"-like nodes anyway and probably no new hardware or infrastructure is necessary to conform to S3 API.
  - brianwski 8y ago
    Disclaimer: I work at Backblaze. > They also use Reed-Solomon and split data into multiple pieces to store on multiple servers. So they need all those "load balancing"-like nodes anyway Yes. We definitely do "load balancing" or more accurately "disk space loading balancing" but we do it all in software. The net outcome is the same, but the cost is lower. > probably no new hardware or infrastructure is necessary to conform to S3 API No, it would require additional hardware we do not purchase at all right now. Backblaze's philosophy is to shave off cost at all layers if it doesn't actually contribute to uptime or durability. Put differently, if there is a lower cost way to achieve the same uptime or durability with some intelligent software or possibly an extra network round trip, we do it that way instead of purchasing extra hardware.
    - hemancuso 8y ago
      What special hardware vs a few cores with a reverse proxy? Surely a trivial cost.
      - brianwski 8y ago
        > Surely a trivial cost. So we both agree it is more than zero cost? Backblaze saves that cost passes on the savings to customers. I'm not sure what the exact costs would be because Backblaze did not implement it that way. > a few cores By "a few" do you mean 10, 100, 1000 or...? For how much bandwidth will your solution support? For example, can your few cores support 10 Gbits/sec? 100 GBits/sec? 1 TBit/sec? Backblaze is COMPLETELY FREE of worrying about these questions, because our solution does not require this additional step and this additional hardware, and therefore does not have this choke point.
jlmorton 8y ago
I'm really surprised B2 doesn't seem to charge for upload API requests. I have a project which uploads several billion small objects to Amazon S3. The vast, vast majority are written, stored with a 15 month TTL, and never touched again. Some small number are downloaded each month. To illustrate this, here's a recent S3 bill: $0.005 per 1,000 PUT, COPY, POST, or LIST requests 289,727,754 Requests $1,448.64 $0.004 per 10,000 GET and all other requests 62,305 Requests $0.02 $0.023 per GB - first 50 TB / month of storage used 18,990.009 GB-Mo $436.77 As you can see, most of our spend on S3 is from the PUT requests, not the storage, or download. Probably there are some things we could do to reduce the number of PUT requests. We don't really care that much, because the total cost is not that large, but there is at least some incentive to reduce the number of PUT calls. But if it was free? I would never change this system. Does Backblaze really want this sort of traffic profile?
- user5994461 8y ago
  >>> Probably there are some things we could do to reduce the number of PUT requests. You can zip very small files into bigger archives. It will save a lot of requests. Generally speaking, it's very inefficient to deal with numerous small files because of latency and overhead. It's not a specificity of S3.
  - brianwski 8y ago
    > Generally speaking, it's very inefficient to deal with numerous small files because of latency I want to second this. Backblaze's Personal Backup uses the same storage that B2 uses, and what we see is that tiny 10 byte files transmitted from "far away" (New Zealand to the United States) can be as slow as 15 kbits/sec transmission rate (yes, dial up modem speeds) because throughput is slaughtered by the latency to push each file. Due to internet philosophies such as "TCP Slow Start" we do not see full throughput rates until after 40 MBytes in a single HTTPS POST if a customer is as far away as New Zealand. One way we circumvent TCP Slow Start is to launch multiple threads. If TCP Slow Start results in only 1 Mbit/sec of throughput, we launch 20 threads to achieve 20 Mbits/sec of throughput. But you have to ask yourself what is going so horribly wrong on the internet if EVERYBODY has to write multiple threads on the same identical path through the internet in order to take advantage of the bandwidth available. Why not just allow 1 thread to get the full 20 Mbits/sec of throughput?

67 points by Manozco 8y ago

metalrain 8y ago

It's great that cost of elasticity is not hidden. I'm glad that there are alternatives.

vpribish 8y ago

"Design Thinking" >>cringe<<

elFarto 8y ago

I wonder if you could get the same functionality of AWS, with the same implementation of B2, by having a single URL to POST files to, that simply sent a redirect to the correct location (apparently there's the 307 HTTP status code for exactly this). E.g: => POST https://upload.backblaze.com/bucket/file <= 307 redirect to https://pod-000-1007-13.backblaze.com/b2api/v1/b2_upload_file/... => POST https://pod-000-1007-13.backblaze.com/b2api/v1/b2_upload_file/...

BillinghamJ 8y ago
I'm not sure you could prevent the client from providing the body before you're able to stop it and tell it to redirect instead - so there's a high risk you end up with everything being uploaded twice.
- willglynn 8y ago
  `Expect: 100-continue` lets the server control this. If it responds "100 Continue", you upload the body. If it responds with a redirect, you redirect without uploading the body. This is standard HTTP/1.1, and it's used by the S3 API for PUT requests.
user5994461 8y ago
Strictly speaking, it's possible but it's not reliable and it shouldn't be used. A redirect on a POST never results into another POST with the same content. - API typically don't follow redirects (without special flags). Too much risk to cause damages by doing repeated calls. - Browsers follow the redirect by a GET with no data. It's typical for a completed form to redirect to another page, do not re submit the form there. There are some settings and flags to alter the behavior into what you describe but it's not a good idea to go there. It will definitely not work out of the box with much of anything.
- willglynn 8y ago
  There are different kinds of redirects. See the discussion of 303 See Other vs 307 Temporary Redirect in RFC 7231: https://tools.ietf.org/html/rfc7231#section-6.4.4 https://tools.ietf.org/html/rfc7231#section-6.4.4 https://tools.ietf.org/html/rfc7231#section-6.4.7 https://tools.ietf.org/html/rfc7231#section-6.4.7 303 means "GET this new URL" while 307 means "resend your request at this URL without changing the verb or body".
  - user5994461 8y ago
    And you can see that the RFC only talks in terms of should and may. It's beyond basic HTTP functionality, the RFC is basically moot at this stage and you're dealing in implementation specific details. You will have to work individually on every client you plan to support, both browsers and libraries. It can be done but it's not necessarily a good idea.
- SahAssar 8y ago
  307 & 308 specifically say to follow the redirect with the same method and body.
  - user5994461 8y ago
    I invite you to try it in your favorite browser interactively, then in ajax , then curl, then python requests, then wget, then whichever software and libraries are available to you.

deedubaya 8y ago

That's all fine and good, I don't care if you're S3 compatible or not.... I do care if I have to write my own API client for your storage backend. Or if you have examples to go off of. Backblaze doesn't seem to offer either for non-C++/Swift languages. Complete non-starter. The, perhaps obvious, win of being S3 compatible is that you open the door to thousands of existing S3 clients already implemented in my different technologies, for free. And you get the developers who use them as customers.

blahblahblogger 8y ago
That's kind of what minio is: https://github.com/minio/minio https://github.com/minio/minio
- deedubaya 8y ago
  Yup, and there a language specific sdk's that provide a consistent API w/adapters for different cloud providers which can be swapped out. i.e. active_storage, fog in ruby
deleted 8y ago
[deleted]
brianwski 8y ago
Disclaimer: I work at Backblaze. > Backblaze doesn't seem to offer either for non-C++/Swift languages. On each of the API web pages there are code examples for the following languages: 1) cUrl, 2) Java, 3) Python, 4) Swift, 5) Ruby, 6) C#, and 7) PHP. For example, go to https://www.backblaze.com/b2/docs/b2_authorize_account.html https://www.backblaze.com/b2/docs/b2_authorize_account.html and scroll ALL THE WAY TO THE BOTTOM of that web page, and you should see "Sample Code" section. Click on the blue buttons to see the different code examples. If your favorite language is missing, we can add it for you! One of our client engineers wrote most of the code examples in all 7 languages in less than 1 week. With these working code examples, the expectation is you should be able to get B2 working in literally less than 2 days in any application in any language.

misterbowfinger 8y ago

Honestly surprised that AWS, GCP, or Azure haven't acquired BackBlaze by now. Seems like an obvious move.

BillinghamJ 8y ago
Why would they? They already have all of this technology and it's not like BackBlaze have a massive number of customers they could acquire.
- bcheung 8y ago
  Cost is a big one. Compared to Digital Ocean Spaces, OVH's Object Store, or Blackblaze, AWS pricing is an order of magnitude more expensive.
  - jorams 8y ago
    Backblaze's technology is not that revolutionary. AWS is an order of magnitude more expensive because they want to, not because they need to be.
brianwski 8y ago
Disclaimer: I am the author of the blog post and work at Backblaze. > surprised that AWS, GCP, or Azure haven't acquired Backblaze by now Sometimes press/customers/people ask us how many customers has Backblaze "converted over from" S3 to B2. To be honest, I think the answer is approximately zero. If a customer already has a Petabyte uploaded into S3, it is simply not practical or cost effective to download that from S3 and import it into B2. The S3 download costs are extremely expensive (9 cents per GByte to download out of S3, vs 1 cent per GByte to download out of B2). Most of Backblaze's B2 customers fall into one of two camps: 1) New customers starting out that have just started to look at cloud storage and need to decide where to put their data. They look at S3, compare with B2, and make a decision and Backblaze B2 gets some of those customers. Realistically we don't even get half of these new customers either because B2 is missing a feature the potential customer needs, or because the customer has not even heard of Backblaze. So Backblaze is probably not causing Amazon too much lost revenue yet. 2) Multi-cloud customers. Any one "cloud provider" can have outages, including Backblaze and including Amazon. So if you store an entire copy of your data in Amazon S3, and also store an entire copy in Backblaze B2, you will (by definition) have both more durable data and higher availability data than a single copy in either one alone. By definition this doesn't actually cost Amazon S3 any sales, because you STILL need a complete copy in S3!! For multi-cloud, Backblaze B2 doesn't harm Amazon one bit. > Seems like an obvious move (for Amazon to acquire Backblaze). Amazon has never offered to buy Backblaze, but also we are not for sale. Or at very least not at the prices Amazon would probably want to spend (or could justify to their board). I don't know if this is common knowledge but Backblaze is employee owned. We never took any significant VC funding, so the only people with voting rights on the board of directors are the original 5 Backblaze founders (including myself). We like what we are doing and we COMPLETELY control our own destiny and work environment. Literally nobody (except customers) can tell us what to do, how to price our products, or what features to build. Backblaze is a fun company to work at, and we all make a good living. So it would be inordinately expensive to buy us out just to put us out of business and dissolve our tight-knit group here and ruin all our fun. Our plan is to stay independent forever.

willglynn 8y ago

This article contains some misunderstandings about the S3 API. > The interface to upload data into Amazon S3 is actually a bit simpler than Backblaze B2’s API. But it comes at a literal cost. It requires Amazon to have a massive and expensive choke point in their network: load balancers. When a customer tries to upload to S3, she is given a single upload URL to use. For instance, http://s3.amazonaws.com/<bucketname> http://s3.amazonaws.com/<bucketname>. This is great for the customer as she can just start pushing data to the URL. But that requires Amazon to be able to take that data and then, in a second step behind the scenes, find available storage space and then push that data to that available location. The second step creates a choke point as it requires having high bandwidth load balancers. That, in turn, carries a significant customer implication; load balancers cost significant money. In fact, S3's REST API requires callers to follow HTTP redirects, and the PUT documentation expressly mentions the HTTP "Expect: 100-continue" mechanism precisely so that the S3 endpoint you reach in your initial PUT request does not have to handle the HTTP request body. https://docs.aws.amazon.com/AmazonS3/latest/dev/Redirects.html https://docs.aws.amazon.com/AmazonS3/latest/dev/Redirects.ht... https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPU... > The Dispatching Server (the API server answering the b2_get_upload_url call) tells the Client “there is space over on “Vault-8329.” This next step is our magic. Armed with the knowledge of the open vault, the Client ends its connection with the Dispatching Server and creates a brand new request DIRECTLY to Vault-8329 (calling b2_upload_file or b2_upload_part). No load balancers involved! Again, this could be done directly with HTTP. PUT to the first server, receive a redirect, PUT to vault-8329, receive "100 Continue", transmit file. There's no need to have a separate API call to get the "real" upload URL. > 3) Expensive, time consuming data copy needs (and “eventual consistency”). Amazon S3 requires the copying of massive amounts of data from one part of their network (the upload server) to wherever the data’s ultimate resting place will be. This is at the root of one of the biggest frustrations when dealing with S3: Amazon’s “eventual consistency.” Wait, I thought they were load balancers? Why does the load balancer need to copy any data once it's done uploading? As for eventual consistency, there is truth to this complaint -- but much less truth than in the distant past. Every S3 region except us-standard has always had read-after-write consistency for new objects since launch, and as of August 2015, us-standard does too: https://aws.amazon.com/about-aws/whats-new/2015/08/amazon-s3-introduces-new-usability-enhancements/ https://aws.amazon.com/about-aws/whats-new/2015/08/amazon-s3... If your PUT returns 200 OK, a subsequent GET will return the object, assuming you're using unique keys. This prevents the 2015-and-earlier problem where you'd create a new S3 object and enqueue a job to process it, then the job gets 404 Not Found while retrieving the new object. There are other cases where S3's eventual consistency can be an issue, but none of them have been dealbreakers for my applications. Having said that: S3's consistency model is a weaker than the model B2 provides, so this is not an argument against providing an S3-compatible interface.

bcheung 8y ago

Anyone know why Amazon didn't adopt existing standards like SCP / SFTP / WebDAV? I've always found the S3 APIs to be difficult to work with, especially for authorization and large uploads.

blasdel 8y ago
S3 buckets do not have filesystem semantics so most of that functionality would be inoperable, and also require heavyweight session-based proxies to support (which is something you can easily operate yourself)
yzmtf2008 8y ago
Because S3 is a K/V Store, not a file system.
- bcheung 8y ago
  File systems can be used as key value as well. The Key is the path, and the value is the contents of the file. Most modern filesystems also have metadata. Granted it's not hierarchical, so listing a bucket lists everything in every folder, but I don't see why that's too big of a concern. Especially since there are pseudo directories in many of the S3 tools. Many people use S3 and have a folder-like hierarchical naming convention anyways. With WebDAV, the URL is the key, and the value is the contents of the upload / download.
halbritt 8y ago
SCP, SFTP, and WebDAV are all woefully inefficient for large data transfers as a result of bandwidth delay product.* They typically use a single TCP session and transmit files serially. WebDAV does support an mput, however. S3 does a much better job of utilizing available bandwidth. It supports multipart parallel uploads and resumes nicely. In my experience, it will usually fully saturate any available upload capacity. There are other protocols that may be more efficient for high bandwidth, high latency connections, i.e. WDT from facebook or UDT, but S3 does okay. *https://en.wikipedia.org/wiki/Bandwidth-delay_product https://en.wikipedia.org/wiki/Bandwidth-delay_product
8y ago

deepsun 8y ago

That "get_upload_url()" trick they invented was in AppEngine's BlobStore since 2008. Although Google deprecated it in favor of GCS.

hemancuso 8y ago

It’s a bit unclear to me what is so expensive about the load balancing nodes. Care that explain why it’s substantially more than a few round robin’d smart reverse proxies moving data to the correct storage node? With S3/Dynamo design the back end destination is largely known from the hash ring. Also- Wasabi has fantastic pricing and full s3 compatibility.

zzzcpan 8y ago
> It’s a bit unclear to me what is so expensive about the load balancing nodes. They also use Reed-Solomon and split data into multiple pieces to store on multiple servers. So they need all those "load balancing"-like nodes anyway and probably no new hardware or infrastructure is necessary to conform to S3 API.
- brianwski 8y ago
  Disclaimer: I work at Backblaze. > They also use Reed-Solomon and split data into multiple pieces to store on multiple servers. So they need all those "load balancing"-like nodes anyway Yes. We definitely do "load balancing" or more accurately "disk space loading balancing" but we do it all in software. The net outcome is the same, but the cost is lower. > probably no new hardware or infrastructure is necessary to conform to S3 API No, it would require additional hardware we do not purchase at all right now. Backblaze's philosophy is to shave off cost at all layers if it doesn't actually contribute to uptime or durability. Put differently, if there is a lower cost way to achieve the same uptime or durability with some intelligent software or possibly an extra network round trip, we do it that way instead of purchasing extra hardware.
  - hemancuso 8y ago
    What special hardware vs a few cores with a reverse proxy? Surely a trivial cost.
    - brianwski 8y ago
      > Surely a trivial cost. So we both agree it is more than zero cost? Backblaze saves that cost passes on the savings to customers. I'm not sure what the exact costs would be because Backblaze did not implement it that way. > a few cores By "a few" do you mean 10, 100, 1000 or...? For how much bandwidth will your solution support? For example, can your few cores support 10 Gbits/sec? 100 GBits/sec? 1 TBit/sec? Backblaze is COMPLETELY FREE of worrying about these questions, because our solution does not require this additional step and this additional hardware, and therefore does not have this choke point.

jlmorton 8y ago

I'm really surprised B2 doesn't seem to charge for upload API requests. I have a project which uploads several billion small objects to Amazon S3. The vast, vast majority are written, stored with a 15 month TTL, and never touched again. Some small number are downloaded each month. To illustrate this, here's a recent S3 bill: $0.005 per 1,000 PUT, COPY, POST, or LIST requests 289,727,754 Requests $1,448.64 $0.004 per 10,000 GET and all other requests 62,305 Requests $0.02 $0.023 per GB - first 50 TB / month of storage used 18,990.009 GB-Mo $436.77 As you can see, most of our spend on S3 is from the PUT requests, not the storage, or download. Probably there are some things we could do to reduce the number of PUT requests. We don't really care that much, because the total cost is not that large, but there is at least some incentive to reduce the number of PUT calls. But if it was free? I would never change this system. Does Backblaze really want this sort of traffic profile?

user5994461 8y ago
>>> Probably there are some things we could do to reduce the number of PUT requests. You can zip very small files into bigger archives. It will save a lot of requests. Generally speaking, it's very inefficient to deal with numerous small files because of latency and overhead. It's not a specificity of S3.
- brianwski 8y ago
  > Generally speaking, it's very inefficient to deal with numerous small files because of latency I want to second this. Backblaze's Personal Backup uses the same storage that B2 uses, and what we see is that tiny 10 byte files transmitted from "far away" (New Zealand to the United States) can be as slow as 15 kbits/sec transmission rate (yes, dial up modem speeds) because throughput is slaughtered by the latency to push each file. Due to internet philosophies such as "TCP Slow Start" we do not see full throughput rates until after 40 MBytes in a single HTTPS POST if a customer is as far away as New Zealand. One way we circumvent TCP Slow Start is to launch multiple threads. If TCP Slow Start results in only 1 Mbit/sec of throughput, we launch 20 threads to achieve 20 Mbits/sec of throughput. But you have to ask yourself what is going so horribly wrong on the internet if EVERYBODY has to write multiple threads on the same identical path through the internet in order to take advantage of the bandwidth available. Why not just allow 1 thread to get the full 20 Mbits/sec of throughput?