3 ms·

I'm really surprised B2 doesn't seem to charge for upload API requests. I have a project which uploads several billion small objects to Amazon S3. The vast, v

I'm really surprised B2 doesn't seem to charge for upload API requests. I have a project which uploads several billion small objects to Amazon S3. The vast, vast majority are written, stored with a 15 month TTL, and never touched again. Some small number are downloaded each month. To illustrate this, here's a recent S3 bill: $0.005 per 1,000 PUT, COPY, POST, or LIST requests 289,727,754 Requests $1,448.64 $0.004 per 10,000 GET and all other requests 62,305 Requests $0.02 $0.023 per GB - first 50 TB / month of storage used 18,990.009 GB-Mo $436.77 As you can see, most of our spend on S3 is from the PUT requests, not the storage, or download. Probably there are some things we could do to reduce the number of PUT requests. We don't really care that much, because the total cost is not that large, but there is at least some incentive to reduce the number of PUT calls. But if it was free? I would never change this system. Does Backblaze really want this sort of traffic profile?

user5994461 8y ago
>>> Probably there are some things we could do to reduce the number of PUT requests. You can zip very small files into bigger archives. It will save a lot of requests. Generally speaking, it's very inefficient to deal with numerous small files because of latency and overhead. It's not a specificity of S3.
- brianwski 8y ago
  > Generally speaking, it's very inefficient to deal with numerous small files because of latency I want to second this. Backblaze's Personal Backup uses the same storage that B2 uses, and what we see is that tiny 10 byte files transmitted from "far away" (New Zealand to the United States) can be as slow as 15 kbits/sec transmission rate (yes, dial up modem speeds) because throughput is slaughtered by the latency to push each file. Due to internet philosophies such as "TCP Slow Start" we do not see full throughput rates until after 40 MBytes in a single HTTPS POST if a customer is as far away as New Zealand. One way we circumvent TCP Slow Start is to launch multiple threads. If TCP Slow Start results in only 1 Mbit/sec of throughput, we launch 20 threads to achieve 20 Mbits/sec of throughput. But you have to ask yourself what is going so horribly wrong on the internet if EVERYBODY has to write multiple threads on the same identical path through the internet in order to take advantage of the bandwidth available. Why not just allow 1 thread to get the full 20 Mbits/sec of throughput?
aclelland 8y ago
I run a service which writes around 50-80 million small documents into S3 per month and, like you, we spend more on PUTs than storage and retrieval. So reducing PUT costs is something we've looked into a few times. We do make heavy use of the S3 versioning API to let us roll back files, lifecycle rules for past revisions and have never been able to find any service which offers the same amount of functionality with reasonable pricing. One thing we have done which has saved us quite a lot is adding a buffer between our service and S3 which delays file writing by up to an hour in case an updated version of a file is uploaded. This lets us limit our PUT requests but still gives us hourly revisions of files if we need to roll back. Without this system we'd be looking at over a billion writes a month. Certainly not a strategy suitable for everyone though.
- brianwski 8y ago
  Disclaimer: I work at Backblaze. > We do make heavy use of the S3 versioning API to let us roll back files, lifecycle rules for past revisions and have never been able to find any service which offers the same amount of functionality with reasonable pricing. Backblaze B2 has both lifecycle rules ( https://www.backblaze.com/b2/docs/lifecycle_rules.html https://www.backblaze.com/b2/docs/lifecycle_rules.html ) and file versioning ( https://www.backblaze.com/b2/docs/file_versions.html https://www.backblaze.com/b2/docs/file_versions.html ). I would LOVE for you to evaluate it and tell us what we are missing to win your business! The Backblaze B2 system is still new and we are actively expanding the APIs all the time, so we're always collecting customer requirements looking for common issues we can tackle next.
- user5994461 8y ago
  Depends on the size of the file. If you're below 4 KB, the main competing system is databases, probably cassandra or DynamoDB for scaling/availability. You should also look into redis/memcache to provide the caching/buffering. Beyond that, S3 is pretty good and it surely makes sense to model the content as files rather than blob.
brianwski 8y ago
Disclaimer: I work at Backblaze. > Does Backblaze really want this sort of traffic profile? Oh heck yes, we would very much like your business! :-) Backblaze only charges you $0.005/GByte/Month so your $436.77 bill would go down to $94.95 and we would be happy to have it. That is profitable for us. (Since Backblaze doesn't have any deep pockets or VC funding, we have to stay profitable.)
- donavanm 8y ago
  So your pricing is aligned with the storage cost. But youre explicitely not pricing in write throughput/IO access, deletes, or similar caused by lifecycle events? How does that square with long term trends to greatly increase density while IO remains flat for the past decade?
  - deleted 8y ago
    [deleted]
  - mmt 8y ago
    > IO remains flat for the past decade Although I agree that storage density growth for HDDs has greatly outstripped any I/O growth, I don't agree that the latter has been flat (i.e. no growth). Are you sure you're not doing something like comparing 7200rpm drives from 10 years ago to 5900rpm (or slower) or variable-speed "green" or even SMR drives from today? That said, I think what many people forget is that for "cloud" storage, the I/O bottleneck is almost certainly going to be the network and not the backend storage, especially for mainly sequential access. If each of their "pods" holds 60 drives and each "vault" holds 20 pods (17 of which are non-parity data), that's over a thousand drives per vault. If each drive is 7200rpm non-SMR, it can saturate a 1Gb ethernet with sequential I/O, and random I/O would divide that by 10 or so. That's the equivalent of 100Gb/s, per vault. That kind of bandwidth is possible and even affordable to provision inside the datacenter without metering and charging for it and is likely to dwarf the size of the connection to the Internet.
  - brianwski 8y ago
    Oh, Backblaze B2 DOES charge for transactions, but it is not intended to make Backblaze a profit, only to recoup the cost of the transactions. You can see a complete list of what each and every API call costs here: https://www.backblaze.com/b2/b2-transactions-price.html https://www.backblaze.com/b2/b2-transactions-price.html

user5994461 8y ago

>>> Probably there are some things we could do to reduce the number of PUT requests. You can zip very small files into bigger archives. It will save a lot of requests. Generally speaking, it's very inefficient to deal with numerous small files because of latency and overhead. It's not a specificity of S3.

brianwski 8y ago
> Generally speaking, it's very inefficient to deal with numerous small files because of latency I want to second this. Backblaze's Personal Backup uses the same storage that B2 uses, and what we see is that tiny 10 byte files transmitted from "far away" (New Zealand to the United States) can be as slow as 15 kbits/sec transmission rate (yes, dial up modem speeds) because throughput is slaughtered by the latency to push each file. Due to internet philosophies such as "TCP Slow Start" we do not see full throughput rates until after 40 MBytes in a single HTTPS POST if a customer is as far away as New Zealand. One way we circumvent TCP Slow Start is to launch multiple threads. If TCP Slow Start results in only 1 Mbit/sec of throughput, we launch 20 threads to achieve 20 Mbits/sec of throughput. But you have to ask yourself what is going so horribly wrong on the internet if EVERYBODY has to write multiple threads on the same identical path through the internet in order to take advantage of the bandwidth available. Why not just allow 1 thread to get the full 20 Mbits/sec of throughput?

aclelland 8y ago

I run a service which writes around 50-80 million small documents into S3 per month and, like you, we spend more on PUTs than storage and retrieval. So reducing PUT costs is something we've looked into a few times. We do make heavy use of the S3 versioning API to let us roll back files, lifecycle rules for past revisions and have never been able to find any service which offers the same amount of functionality with reasonable pricing. One thing we have done which has saved us quite a lot is adding a buffer between our service and S3 which delays file writing by up to an hour in case an updated version of a file is uploaded. This lets us limit our PUT requests but still gives us hourly revisions of files if we need to roll back. Without this system we'd be looking at over a billion writes a month. Certainly not a strategy suitable for everyone though.

brianwski 8y ago
Disclaimer: I work at Backblaze. > We do make heavy use of the S3 versioning API to let us roll back files, lifecycle rules for past revisions and have never been able to find any service which offers the same amount of functionality with reasonable pricing. Backblaze B2 has both lifecycle rules ( https://www.backblaze.com/b2/docs/lifecycle_rules.html https://www.backblaze.com/b2/docs/lifecycle_rules.html ) and file versioning ( https://www.backblaze.com/b2/docs/file_versions.html https://www.backblaze.com/b2/docs/file_versions.html ). I would LOVE for you to evaluate it and tell us what we are missing to win your business! The Backblaze B2 system is still new and we are actively expanding the APIs all the time, so we're always collecting customer requirements looking for common issues we can tackle next.
user5994461 8y ago
Depends on the size of the file. If you're below 4 KB, the main competing system is databases, probably cassandra or DynamoDB for scaling/availability. You should also look into redis/memcache to provide the caching/buffering. Beyond that, S3 is pretty good and it surely makes sense to model the content as files rather than blob.

brianwski 8y ago

Disclaimer: I work at Backblaze. > Does Backblaze really want this sort of traffic profile? Oh heck yes, we would very much like your business! :-) Backblaze only charges you $0.005/GByte/Month so your $436.77 bill would go down to $94.95 and we would be happy to have it. That is profitable for us. (Since Backblaze doesn't have any deep pockets or VC funding, we have to stay profitable.)

donavanm 8y ago
So your pricing is aligned with the storage cost. But youre explicitely not pricing in write throughput/IO access, deletes, or similar caused by lifecycle events? How does that square with long term trends to greatly increase density while IO remains flat for the past decade?
- deleted 8y ago
  [deleted]
- mmt 8y ago
  > IO remains flat for the past decade Although I agree that storage density growth for HDDs has greatly outstripped any I/O growth, I don't agree that the latter has been flat (i.e. no growth). Are you sure you're not doing something like comparing 7200rpm drives from 10 years ago to 5900rpm (or slower) or variable-speed "green" or even SMR drives from today? That said, I think what many people forget is that for "cloud" storage, the I/O bottleneck is almost certainly going to be the network and not the backend storage, especially for mainly sequential access. If each of their "pods" holds 60 drives and each "vault" holds 20 pods (17 of which are non-parity data), that's over a thousand drives per vault. If each drive is 7200rpm non-SMR, it can saturate a 1Gb ethernet with sequential I/O, and random I/O would divide that by 10 or so. That's the equivalent of 100Gb/s, per vault. That kind of bandwidth is possible and even affordable to provision inside the datacenter without metering and charging for it and is likely to dwarf the size of the connection to the Internet.
- brianwski 8y ago
  Oh, Backblaze B2 DOES charge for transactions, but it is not intended to make Backblaze a profit, only to recoup the cost of the transactions. You can see a complete list of what each and every API call costs here: https://www.backblaze.com/b2/b2-transactions-price.html https://www.backblaze.com/b2/b2-transactions-price.html