5 ms·

It's not trivial, but it is fairly easy to beat S3 on storage costs alone, let alone the other factors. I personally wouldn't use Backblaze's storage pod, so m

by skuhn 12y ago

It's not trivial, but it is fairly easy to beat S3 on storage costs alone, let alone the other factors. I personally wouldn't use Backblaze's storage pod, so my numbers are based on equivalent components (and don't reflect what's possible if you negotiate prices and optimize for your particular needs -- Backblaze's costs are surely lower than this): 11 Supermicro 36 drive 4U server : $4000 each 1 48-port top-of-rack switch : $5000 each 374 4TB drive : $400 each 22 500GB drive : $200 each 2 CDU : $3000 each Rack + integration : $10000 each Total rack cost is $220,000 for 1.4PB or 700GB with basic 2-copy redundancy. Power draw is around 8kw, so 1 year of power is around $1100. Figure on a drive failure rate of 5%, memory 2%, PSU 3% and you might spend another $18,000 to stock spares. So besides your core datacenter costs, this solution will cost $239,100 to operate for a year. These costs only improve when you amortize the hardware over a 3 year period, which is standard. Now how much does S3 cost to store 700GB for a year? $248,530.94. Finally, consider bandwidth costs. I'll crib from a recent comment of mine [1]: 1 gbit/s in a datacenter: $30,000 / year 500 mbit/s in S3: $142,540 / year And that in a nutshell is why AWS doesn't make financial sense at any significant scale. S3 makes sense if you aren't really going to use all of that space: you can scale your spending up as your needs scale up. That isn't the kind of problem Backblaze or other storage businesses have, there isn't much chance of that storage going unused for any extended period of time. And if they architect their datacenter and service correctly, the down period between rolling in hardware and getting it utilized sufficiently to begin seeing a return on the investment should be minimal. [1] https://news.ycombinator.com/item?id=9139244 https://news.ycombinator.com/item?id=9139244

moe 12y ago
Your cost estimates are outlandish, making Amazon S3 look like a bargain. In reality 1.4PB raw will set you back well below $50k/pa, not $239k/pa. If you're lazy you could even rent 1.4PB raw from e.g. Hetzner and you'd still be paying under $90k/pa (that would be 23 of their SX290 servers).
- skuhn 12y ago
  You can do significantly better with the right vendors and some negotiation -- this is specifically a pricing ballpark, not rock bottom deals. It's not outlandish at all to say that in terms of raw storage you're going to be close to S3's pricing, but it also doesn't reflect well on S3 at all. They are buying at a scale far beyond what you can hope to achieve. This is money spent on physical equipment that you can choose to lease, that you can take depreciation on, that you can resell -- these are assets, not a line item on a monthly bill. S3 has no such benefits. And this doesn't even taking into account bandwidth, which is the real killer. The stored data has to come out sometime. 1.4PB of raw storage requires ~374 4TB hard drives. If you want to spend $50,000, you have at most $130 to spend on drives, and that's only if you get literally everything else for free including electricity. You can almost buy 4TB desktop class drives for that price from Newegg, but not quite. Don't buy drives from retail channels, and don't buy desktop drives at all (yes, I know Backblaze does both of these things). HGST 4TB MegaScale or UltraStar drives are a safe bet, depending on your performance requirements. They run between $250-300 in quantity, which means 1.4PB is $93,500 in disks alone.
  - moe 12y ago
    Stop switching around between hardware cost and yearly cost, jesus... (pa means per annum) 0.7PB on Amazon S3 costs $810k for 3 years (as per their cost calculator). Renting 1.4PB raw from Hetzner costs about $270k for 3 years. Running 1.4PB raw on your own hardware costs about $150k for 3 years (on Supermicro JBODs). It's not that hard really, and my point merely was that the difference is much bigger than you made it sound.
    - parasubvert 12y ago
      Apples, to apples please. 0.7PB on Amazon S3 reduced redundancy is $582,660 over three years with geo-redundancy and availability across an entire region, and lifecycle to push to Glacier which would drop the price to $289,872 annually given 50,000 monthly uploads and 20% monthly retrieval. No labour when it comes to dealing with equipment failures. I'd also say that only SMBs and web/internet companies run the equivalent SuperMicro JBODs, most large companies use enterprise storage vendors like EMC, NetApp, HP/3PAR which are premium priced.
deleted 12y ago
[deleted]
toomuchtodo 12y ago
Thanks for doing the math. I've repeatedly shown people (I do Infrastructure, sometimes AWS sometimes physical) that AWS is awesome for proof of concept. All opex, no commitments. As soon as you start getting to a certain scale though, time to move onto physical equipment when you've got predictable load patterns. If Stackoverflow can still do hardware colo'd, so can you.
- skuhn 12y ago
  I completely agree. I sometimes take flack for "hating" AWS (and I do hate some aspects of it), but I think it's a great platform for incubating your service or proving a concept. Every company I've been at in the last five years has used AWS to an extent, despite operating their own datacenters. It's great for random analytics jobs or other tasks that aren't steady work tied to your core business. It quickly starts to not make sense for services that have usage-driven growth patterns. AWS's billing model just doesn't work for this -- even though higher usage is discounted, it isn't enough to overcome the skew of the model. Companies that have deployed on AWS often realize this too late: their business starts to take off, and they get crushed under giant AWS bills that would be disproportionate even at 50% off list. It's much harder to move off once you've hit scale.
  - acdha 12y ago
    A lot of this also comes down to ops & the features you really need – e.g. your example above didn't include the costs of the geographic redundancy or strong data integrity features which are standard with S3. A lot of places simply don't have geographically redundant datacenters and either lack the support team to run something like a replicated cluster filesystem in heavy production usage or, more commonly, don't have enough usage to really push it into clear win. In the case above, the cost differential at 700TB is roughly the cost of a single sysadmin and so the running operation would probably be a wash versus S3 unless it had heavy data churn to maximize the S3 expenses or a good way to amortize the staffing needed for 24-hour-a-day support across other projects. Many places have budget processes which favor spending more on known upfront costs than getting permission to add full-time employees in the hopes of being able to beat those costs down the road. I'm hoping that something like OpenStack Swift matures to the point where some of the complexity at the software + ops level starts making the DIY option routine. You'll never get rid of the minimum sysadmin commitment but that becomes a lot easier if it can be 5% of the night-shift because the software doesn't require a lot of care and feeding.
jrs235 12y ago
I think you mean 700TB not 700GB
- skuhn 12y ago
  Right, just a unit problem -- the pricing is still correct. I seem to make this mistake a lot lately, since everything is billed per-gig or per-meg, but you only really care about per-TB units. Oops.
  - atYevP 12y ago
    Don't you mean...ops? Hah...sorry puns :-/
nullc 12y ago
>1.4PB or 700GB with basic 2-copy redundancy You mean 700TB.
vidarh 12y ago
I've done similar calculations for clients, and fully agree. Cutting the storage costs is easy, but bandwidth in particular is ridiculously priced at Amazon (for the cost of 1TB transfer from AWS, I can rent a server with many times as much transfer and still have money left over), to the point where even if a client for some reason insists on S3 storage for perceived safety, it can still pay to store in S3 but put boxes "in front" with capacity for a single copy of each object, and fall back to S3 in case of local drive failures. Unless the load is totally write dominated (so not good for backup services), cutting the AWS bandwidth bill can often pay for the extra servers and colo costs many times over.
parasubvert 12y ago
So, why do Instagram and Netflix continue to use S3 for petabytes of video and image data? In Netflix' case, this doesn't even count their vast replicated CDN infrastructure between your device and S3. The math is never this simple. Firstly I assume you mean 700 TB, not GB. You might as well look at reduced-redundancy S3 which brings the annual cost down to $194,220. With that you still have multiple WAN links, excellent throughput, multiple data centers, and no labour for dealing with failed drives. And if we're truly trying to do backup / archival, where download throughput doesn't matter, let's look at Glacier, even if we do 50000 monthly uploads and retrieve 20% of the 700TB monthly, we're down to $96,630. Most sizable businesses also aren't buying bargain basement drive arrays from SuperMicro, they're buying EMC, NetApp, 3PAR, or Hitachi. As with anything S3 is not perfect, but it really depends on your use case if your own data center or rented equipment is appropriate. For many large businesses, Amazon makes more sense.
- skuhn 12y ago
  > In Netflix' case, this doesn't even count their vast replicated CDN infrastructure between your device and S3. There is part of your answer. They only pay S3 outbound on cache misses, not on cache hits. That will lower their S3 bandwidth bill to 1/50th or 1/100th of the full cost. Also, Netflix doesn't pay list prices for AWS, and they made a call that they wanted to be fully cloud based early in the streaming service's life. Maybe they would do it differently today, but that ship has obviously sailed for them. On the other hand, lots of large AWS hosted services do discover that they need to move off in order to get their costs right; I have quite a bit of first hand experience there. > Glacier Costs and time to retrieve data from Glacier are quite different, and I would only use it for data that I basically never expect to access. A service like Backblaze couldn't use Glacier in reality, despite being a service for backups. > Bargain basement drive arrays On the contrary, I think that building it yourself with basic hardware building blocks is much more common at large scale Internet companies. A less technical or more appliance driven company might choose NetApp or EMC, but I don't think it's the right move for somewhere with some in-house expertise. It's the right move if you want more of a black box that you hope will work without much internal effort (hint: it won't work out that way).

by skuhn 12y ago

moe 12y ago

Your cost estimates are outlandish, making Amazon S3 look like a bargain. In reality 1.4PB raw will set you back well below $50k/pa, not $239k/pa. If you're lazy you could even rent 1.4PB raw from e.g. Hetzner and you'd still be paying under $90k/pa (that would be 23 of their SX290 servers).

skuhn 12y ago
You can do significantly better with the right vendors and some negotiation -- this is specifically a pricing ballpark, not rock bottom deals. It's not outlandish at all to say that in terms of raw storage you're going to be close to S3's pricing, but it also doesn't reflect well on S3 at all. They are buying at a scale far beyond what you can hope to achieve. This is money spent on physical equipment that you can choose to lease, that you can take depreciation on, that you can resell -- these are assets, not a line item on a monthly bill. S3 has no such benefits. And this doesn't even taking into account bandwidth, which is the real killer. The stored data has to come out sometime. 1.4PB of raw storage requires ~374 4TB hard drives. If you want to spend $50,000, you have at most $130 to spend on drives, and that's only if you get literally everything else for free including electricity. You can almost buy 4TB desktop class drives for that price from Newegg, but not quite. Don't buy drives from retail channels, and don't buy desktop drives at all (yes, I know Backblaze does both of these things). HGST 4TB MegaScale or UltraStar drives are a safe bet, depending on your performance requirements. They run between $250-300 in quantity, which means 1.4PB is $93,500 in disks alone.
- moe 12y ago
  Stop switching around between hardware cost and yearly cost, jesus... (pa means per annum) 0.7PB on Amazon S3 costs $810k for 3 years (as per their cost calculator). Renting 1.4PB raw from Hetzner costs about $270k for 3 years. Running 1.4PB raw on your own hardware costs about $150k for 3 years (on Supermicro JBODs). It's not that hard really, and my point merely was that the difference is much bigger than you made it sound.
  - parasubvert 12y ago
    Apples, to apples please. 0.7PB on Amazon S3 reduced redundancy is $582,660 over three years with geo-redundancy and availability across an entire region, and lifecycle to push to Glacier which would drop the price to $289,872 annually given 50,000 monthly uploads and 20% monthly retrieval. No labour when it comes to dealing with equipment failures. I'd also say that only SMBs and web/internet companies run the equivalent SuperMicro JBODs, most large companies use enterprise storage vendors like EMC, NetApp, HP/3PAR which are premium priced.

deleted 12y ago

[deleted]

toomuchtodo 12y ago

Thanks for doing the math. I've repeatedly shown people (I do Infrastructure, sometimes AWS sometimes physical) that AWS is awesome for proof of concept. All opex, no commitments. As soon as you start getting to a certain scale though, time to move onto physical equipment when you've got predictable load patterns. If Stackoverflow can still do hardware colo'd, so can you.

skuhn 12y ago
I completely agree. I sometimes take flack for "hating" AWS (and I do hate some aspects of it), but I think it's a great platform for incubating your service or proving a concept. Every company I've been at in the last five years has used AWS to an extent, despite operating their own datacenters. It's great for random analytics jobs or other tasks that aren't steady work tied to your core business. It quickly starts to not make sense for services that have usage-driven growth patterns. AWS's billing model just doesn't work for this -- even though higher usage is discounted, it isn't enough to overcome the skew of the model. Companies that have deployed on AWS often realize this too late: their business starts to take off, and they get crushed under giant AWS bills that would be disproportionate even at 50% off list. It's much harder to move off once you've hit scale.
- acdha 12y ago
  A lot of this also comes down to ops & the features you really need – e.g. your example above didn't include the costs of the geographic redundancy or strong data integrity features which are standard with S3. A lot of places simply don't have geographically redundant datacenters and either lack the support team to run something like a replicated cluster filesystem in heavy production usage or, more commonly, don't have enough usage to really push it into clear win. In the case above, the cost differential at 700TB is roughly the cost of a single sysadmin and so the running operation would probably be a wash versus S3 unless it had heavy data churn to maximize the S3 expenses or a good way to amortize the staffing needed for 24-hour-a-day support across other projects. Many places have budget processes which favor spending more on known upfront costs than getting permission to add full-time employees in the hopes of being able to beat those costs down the road. I'm hoping that something like OpenStack Swift matures to the point where some of the complexity at the software + ops level starts making the DIY option routine. You'll never get rid of the minimum sysadmin commitment but that becomes a lot easier if it can be 5% of the night-shift because the software doesn't require a lot of care and feeding.

jrs235 12y ago

I think you mean 700TB not 700GB

skuhn 12y ago
Right, just a unit problem -- the pricing is still correct. I seem to make this mistake a lot lately, since everything is billed per-gig or per-meg, but you only really care about per-TB units. Oops.
- atYevP 12y ago
  Don't you mean...ops? Hah...sorry puns :-/

nullc 12y ago

>1.4PB or 700GB with basic 2-copy redundancy You mean 700TB.

vidarh 12y ago

I've done similar calculations for clients, and fully agree. Cutting the storage costs is easy, but bandwidth in particular is ridiculously priced at Amazon (for the cost of 1TB transfer from AWS, I can rent a server with many times as much transfer and still have money left over), to the point where even if a client for some reason insists on S3 storage for perceived safety, it can still pay to store in S3 but put boxes "in front" with capacity for a single copy of each object, and fall back to S3 in case of local drive failures. Unless the load is totally write dominated (so not good for backup services), cutting the AWS bandwidth bill can often pay for the extra servers and colo costs many times over.

parasubvert 12y ago

So, why do Instagram and Netflix continue to use S3 for petabytes of video and image data? In Netflix' case, this doesn't even count their vast replicated CDN infrastructure between your device and S3. The math is never this simple. Firstly I assume you mean 700 TB, not GB. You might as well look at reduced-redundancy S3 which brings the annual cost down to $194,220. With that you still have multiple WAN links, excellent throughput, multiple data centers, and no labour for dealing with failed drives. And if we're truly trying to do backup / archival, where download throughput doesn't matter, let's look at Glacier, even if we do 50000 monthly uploads and retrieve 20% of the 700TB monthly, we're down to $96,630. Most sizable businesses also aren't buying bargain basement drive arrays from SuperMicro, they're buying EMC, NetApp, 3PAR, or Hitachi. As with anything S3 is not perfect, but it really depends on your use case if your own data center or rented equipment is appropriate. For many large businesses, Amazon makes more sense.

skuhn 12y ago
> In Netflix' case, this doesn't even count their vast replicated CDN infrastructure between your device and S3. There is part of your answer. They only pay S3 outbound on cache misses, not on cache hits. That will lower their S3 bandwidth bill to 1/50th or 1/100th of the full cost. Also, Netflix doesn't pay list prices for AWS, and they made a call that they wanted to be fully cloud based early in the streaming service's life. Maybe they would do it differently today, but that ship has obviously sailed for them. On the other hand, lots of large AWS hosted services do discover that they need to move off in order to get their costs right; I have quite a bit of first hand experience there. > Glacier Costs and time to retrieve data from Glacier are quite different, and I would only use it for data that I basically never expect to access. A service like Backblaze couldn't use Glacier in reality, despite being a service for backups. > Bargain basement drive arrays On the contrary, I think that building it yourself with basic hardware building blocks is much more common at large scale Internet companies. A less technical or more appliance driven company might choose NetApp or EMC, but I don't think it's the right move for somewhere with some in-house expertise. It's the right move if you want more of a black box that you hope will work without much internal effort (hint: it won't work out that way).