5 ms·

What if you want to move off S3? Let's do the math. * To store 10+ PB of data. * You need 15 PB of storage (running at 66% capacity) * You need 30 PB of raw

What if you want to move off S3? Let's do the math. * To store 10+ PB of data. * You need 15 PB of storage (running at 66% capacity) * You need 30 PB of raw disks (twice for redundancy). You're looking at buying thousands of large disks, in the order of a million dollar upfront. Do you have that sort of money available right now? Maybe you do. Then, are you ready to receive and handle entire pallets of hardware? That will need to go somewhere with power and networking. They won't show up for another 3-6 months because that's the lead time to receive an order like that. If you talk to Dell/HP/other, they can advise you and sell you large storage appliances. Problem is, the larger appliances will only host 1 or 2 PB. That's nowhere near enough. There is a sweet spot in moving off the cloud, if you can fit your entire infrastructure into one rack. You're not in that sweet spot. You're going to be filling multiple racks, which is a pretty serious issue in terms of logistics (space, power, upfront costs, networking). Then you're going to have to handle "sharding" on top of the storage because there's no filesystem that can easily address 4 racks of disks. (Ceph/Lustre is another year long project for half a person). The conclusion of this story. S3 is pretty good. Your time would be better spend optimizing the software. What is expensive? The storage or the bandwidth or both? * If it's the bandwidth. You need to improve your CDN and caching layer. * If it's the storage. You should work on better compression for the images and videos. And check whether you can adjust retention.

louwrentius 5y ago
Very good advice!
crescentfresh 5y ago
> * To store 10+TB of data. > * You need 15 TB of storage (running at 66% capacity) > * You need 30 TB of raw disks (twice for redundancy). Did you mean PB?
- user5994461 5y ago
  Corrected.
gamegoblin 5y ago
FWIW you can get great redundancy with far less than 2x storage factor. e.g. Facebook uses a 10:14 erasure coding scheme[1] so they can lose up to 4 disks without losing data, and that only incurs a 1.4x storage factor. If one's data is cold enough, one can go wider than this, e.g. 50:55 or something has a 1.1x factor. Not that this fundamentally changes your analysis and other totally valid points, but the 2x bit can probably be reduced a lot. [1] https://engineering.fb.com/2015/05/04/core-data/under-the-hood-facebook-s-cold-storage-system/ https://engineering.fb.com/2015/05/04/core-data/under-the-ho...
- dsyrk 5y ago
  https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive Basically they use par2 multi-file archives for cold storage with each archive file segment scattered across different physical locations. Always fun to see the kids rediscovering tricks from the old days.
pier25 5y ago
> If it's the bandwidth. You need to improve your CDN and caching layer. What would you recommend for this? (considering data is stored in S3)
- shyn3 5y ago
  Verizon and Redis has worked well for me.
latch 5y ago
> Let's do the math. Offers no math. At retail, 625 16TB drives is $400000. This is about 2x the MONTHLY retail s3 pricing. Further, as we all know, AWS bandwidth pricing is absolutely bonkers (1). I think your conclusion that S3 is "pretty good" needs a lot more math to support. (1) https://twitter.com/eastdakota/status/1371252709836263425 https://twitter.com/eastdakota/status/1371252709836263425
- _nickwhite 5y ago
  The math should also include the price of the staff who babysit 625 spinning metal disks, who likely drive to a data center multiple times a week to swap failed drives. I shudder to think if this job fell in my lap!
  - latch 5y ago
    Sure, but you actually have to go through the steps. The very first step, back of the napkin, indicates savings. More than enough to warrant a more detailed evaluation. Then you can start considering the more complicated factors (redundancy, staffing, power, ...). My response was in the context of someone who didn't do any of that. Also, my tongue-in-cheek response to you is: the price will be offset by the SRE engineers who were babysitting your AWS setup that you'll no longer need. (More seriously, I don't think finding quality sysadmins who enjoy this stuff is particularly harder than finding quality roles for any other tech positions).
    - GauntletWizard 5y ago
      Been there, done that, at all levels. I would much rather be working on a 10PB set of hardware racks, including all the drive replacements. When you factor in the costs of compute hardware (to make that useful), networking equipment, etc, it's trebled again the cost, and then again for the power, cooling, and cage space to run it all. The actual break-even point of running your own hardware is more like 2 years. But it's not about price: It's about control, and it's about the expertise you gain from running all of that. If you have 10PB of data, you should have someone in-house who knows how to work with 10PB of data at a low level, and the best way to get that is to employ people at all levels to make that work. You gain significant advantage from having the direct performance data and the expertise of having techs whose 9-5 is replacing disks.
Nacraile 5y ago
> If you talk to Dell/HP/other, they can advise you and sell you large storage appliances. Problem is, the larger appliances will only host 1 or 2 PB. That's nowhere near enough. This is just incorrect. If you talk to HPE, they should be quite happy to sell you the my employer's software (Qumulo) alongside their hardware. 10+ PB is definitely supported. (The HPE part is not required) If you talk to Dell EMC, they will quite happily sell you their competing product, which is also quite capable of scaling beyond 1-2PB.
- user5994461 5y ago
  I've checked what I could find on Qumulo. It is software that you run on top of regular servers, to form a storage cluster. It seems to me you're only confirming my previous point, that you need to invest in complicated/expensive software to make the raw storage usable. >>> Then you're going to have to handle "sharding" on top of the storage because there's no filesystem that can easily address 4 racks of disks. (Ceph/Lustre is another year long project for half a person). There's no listed price on the website, you will need to call sales. Wouldn't be surprised if it started at 6 figures a year for a few servers. It looks like it may not run on just any server, but may need certified server hardware from HP or Qumulo.
- hedora 5y ago
  Most (all?) enterprise vendors will go well beyond 1-2PB. Four years ago, one of the all flash vendors routinely advertised “well under a dollar a gigabyte”. Their prices have dropped dramatically since then, but the out of date numbers translate to “well under a million per PB”. That’s at the high end of performance with posix (nfs) or crash coherent (block) semantics. (Some also do S3, if that’s preferable for some reason) With a 5 year depreciation cycle, those old machines were at << $16K / month per PB. Today’s all flash systems fit multiple PB per rack, and need less than one full time admin. Hope that helps.
- qwertykb 5y ago
  Always fun stumbling across another Qumulon on here :)
quantumofalpha 5y ago
AWS is ridiculously expensively at their scale, both for storage and egress. But the choice is not only between that and building a staffed on-premise storage facility. You can compromise at a middle ground - rent a bunch of VPS/managed servers and let the hosting companies deal with all the nastiness of managing physical hardware and CAPEX. Cost around $1.6-2/TB/month (e.g. Hetzner's SX) for raw non-redundant storage, an order of magnitude better than AWS. Comes with far more reasonably priced bandwidth too. Build some error correction on top using one of the many open-source distributed filesystems out there or perhaps an in-house software solution (reed-solomon isn't exactly rocket science). And for some 30+% overhead, depending on workload (you can have very low overload if you have few reads or very relaxed latency requirements), you should have a decently fault tolerant distributed storage at a fraction of AWS costs.
- mark_l_watson 5y ago
  I agree that considering Hetzner is a good idea. I have used them often, never any problems, and very low pricing.
montroser 5y ago
> You need to improve your CDN and caching layer. Depends on usage patterns, but if this is 10PB of users'personal photos and videos, then you're not going to get much value from caching because the hit rate will be so low.

louwrentius 5y ago

Very good advice!

crescentfresh 5y ago

> * To store 10+TB of data. > * You need 15 TB of storage (running at 66% capacity) > * You need 30 TB of raw disks (twice for redundancy). Did you mean PB?

user5994461 5y ago
Corrected.

gamegoblin 5y ago

FWIW you can get great redundancy with far less than 2x storage factor. e.g. Facebook uses a 10:14 erasure coding scheme[1] so they can lose up to 4 disks without losing data, and that only incurs a 1.4x storage factor. If one's data is cold enough, one can go wider than this, e.g. 50:55 or something has a 1.1x factor. Not that this fundamentally changes your analysis and other totally valid points, but the 2x bit can probably be reduced a lot. [1] https://engineering.fb.com/2015/05/04/core-data/under-the-hood-facebook-s-cold-storage-system/ https://engineering.fb.com/2015/05/04/core-data/under-the-ho...

dsyrk 5y ago
https://en.wikipedia.org/wiki/Parchive https://en.wikipedia.org/wiki/Parchive Basically they use par2 multi-file archives for cold storage with each archive file segment scattered across different physical locations. Always fun to see the kids rediscovering tricks from the old days.

pier25 5y ago

> If it's the bandwidth. You need to improve your CDN and caching layer. What would you recommend for this? (considering data is stored in S3)

shyn3 5y ago
Verizon and Redis has worked well for me.

latch 5y ago

> Let's do the math. Offers no math. At retail, 625 16TB drives is $400000. This is about 2x the MONTHLY retail s3 pricing. Further, as we all know, AWS bandwidth pricing is absolutely bonkers (1). I think your conclusion that S3 is "pretty good" needs a lot more math to support. (1) https://twitter.com/eastdakota/status/1371252709836263425 https://twitter.com/eastdakota/status/1371252709836263425

_nickwhite 5y ago
The math should also include the price of the staff who babysit 625 spinning metal disks, who likely drive to a data center multiple times a week to swap failed drives. I shudder to think if this job fell in my lap!
- latch 5y ago
  Sure, but you actually have to go through the steps. The very first step, back of the napkin, indicates savings. More than enough to warrant a more detailed evaluation. Then you can start considering the more complicated factors (redundancy, staffing, power, ...). My response was in the context of someone who didn't do any of that. Also, my tongue-in-cheek response to you is: the price will be offset by the SRE engineers who were babysitting your AWS setup that you'll no longer need. (More seriously, I don't think finding quality sysadmins who enjoy this stuff is particularly harder than finding quality roles for any other tech positions).
  - GauntletWizard 5y ago
    Been there, done that, at all levels. I would much rather be working on a 10PB set of hardware racks, including all the drive replacements. When you factor in the costs of compute hardware (to make that useful), networking equipment, etc, it's trebled again the cost, and then again for the power, cooling, and cage space to run it all. The actual break-even point of running your own hardware is more like 2 years. But it's not about price: It's about control, and it's about the expertise you gain from running all of that. If you have 10PB of data, you should have someone in-house who knows how to work with 10PB of data at a low level, and the best way to get that is to employ people at all levels to make that work. You gain significant advantage from having the direct performance data and the expertise of having techs whose 9-5 is replacing disks.

Nacraile 5y ago

> If you talk to Dell/HP/other, they can advise you and sell you large storage appliances. Problem is, the larger appliances will only host 1 or 2 PB. That's nowhere near enough. This is just incorrect. If you talk to HPE, they should be quite happy to sell you the my employer's software (Qumulo) alongside their hardware. 10+ PB is definitely supported. (The HPE part is not required) If you talk to Dell EMC, they will quite happily sell you their competing product, which is also quite capable of scaling beyond 1-2PB.

user5994461 5y ago
I've checked what I could find on Qumulo. It is software that you run on top of regular servers, to form a storage cluster. It seems to me you're only confirming my previous point, that you need to invest in complicated/expensive software to make the raw storage usable. >>> Then you're going to have to handle "sharding" on top of the storage because there's no filesystem that can easily address 4 racks of disks. (Ceph/Lustre is another year long project for half a person). There's no listed price on the website, you will need to call sales. Wouldn't be surprised if it started at 6 figures a year for a few servers. It looks like it may not run on just any server, but may need certified server hardware from HP or Qumulo.
hedora 5y ago
Most (all?) enterprise vendors will go well beyond 1-2PB. Four years ago, one of the all flash vendors routinely advertised “well under a dollar a gigabyte”. Their prices have dropped dramatically since then, but the out of date numbers translate to “well under a million per PB”. That’s at the high end of performance with posix (nfs) or crash coherent (block) semantics. (Some also do S3, if that’s preferable for some reason) With a 5 year depreciation cycle, those old machines were at << $16K / month per PB. Today’s all flash systems fit multiple PB per rack, and need less than one full time admin. Hope that helps.
qwertykb 5y ago
Always fun stumbling across another Qumulon on here :)

quantumofalpha 5y ago

AWS is ridiculously expensively at their scale, both for storage and egress. But the choice is not only between that and building a staffed on-premise storage facility. You can compromise at a middle ground - rent a bunch of VPS/managed servers and let the hosting companies deal with all the nastiness of managing physical hardware and CAPEX. Cost around $1.6-2/TB/month (e.g. Hetzner's SX) for raw non-redundant storage, an order of magnitude better than AWS. Comes with far more reasonably priced bandwidth too. Build some error correction on top using one of the many open-source distributed filesystems out there or perhaps an in-house software solution (reed-solomon isn't exactly rocket science). And for some 30+% overhead, depending on workload (you can have very low overload if you have few reads or very relaxed latency requirements), you should have a decently fault tolerant distributed storage at a fraction of AWS costs.

mark_l_watson 5y ago
I agree that considering Hetzner is a good idea. I have used them often, never any problems, and very low pricing.

montroser 5y ago

> You need to improve your CDN and caching layer. Depends on usage patterns, but if this is 10PB of users'personal photos and videos, then you're not going to get much value from caching because the hit rate will be so low.