4 ms·

Wasabi has them. Scaling to 230pb would be more than their 1PB calculation. Taking their 5 yr total number, multiply to 230pb, divide by 5, then divide by 12 ha

by jabart 3y ago

Wasabi has them. Scaling to 230pb would be more than their 1PB calculation. Taking their 5 yr total number, multiply to 230pb, divide by 5, then divide by 12 has a monthly total of 5,021,666. It's also 131 server racks according to wasabi. At 131 racks of storage you likely now need racks dedicated to networking since a top of rack switch wouldn't be enough. With space for the racks it's about 2,227sqft of space. https://wasabi.com/blog/on-premises-vs-cloud-storage/ https://wasabi.com/blog/on-premises-vs-cloud-storage/

hinkley 3y ago
140 racks is a quite a lot of tonnage of AC as well. And probably hot and cold rows to make it function.
- jabart 3y ago
  Hot/cold was factored in not the AC or power needs. I don't know many places you could just casually ask for 100+ racks. Maybe our midwest datacenters are just smaller than what is on the coast.
  - hinkley 3y ago
    The last place I worked that had that kind of capacity, I had access to the lobby of that building and that was about it. The server room I could access had space for <wiggles fingers> maybe 15-20 racks and in fact had maybe 7, which is probably why it was always cold as fuck in there and I didn’t like being in there much. Then there was a contract which had a server room twice the size and mostly contained an AS/3<mumble> and a couple racks of Intel hardware. You could probably get a ping pong table into that one. Giant rooms that are mostly white and too cold are a little freaky, and reminiscent of 2001. There have been a number of stories of people not being able to utilize their floor space for more servers because they ran out of space on the roof for AC units. At one point I recall Google was working on power dissipation because they had hit the max electrical code, and so no one would agree to run more power into their buildings. More servers meant more compute per watt and per ton of chillers.
    - jabart 3y ago
      AWS has a talk about things like this. They were, no idea if they still do, setup an AZ based on MWH figure for most efficiency because any higher the power company doesn't like it. Our local DC the water cooling and refrigerant cooling is larger than the 5 generators they have. I've crammed a lot of servers in my rack and I'm out of power plugs but still under 2kwh of what we are on contract for. Dense servers have really messed with datacenters.
  - littlestymaar 3y ago
    > I don't know many places you could just casually ask for 100+ racks. That's obviously not something you can do on a whim, but at the same time we're talking about a seven figures investment plus the time to build the in-house team to manage this, so it's not going to happen overnight anyway.
littlestymaar 3y ago
This is a marketing pitch for a cloud storage company and it shows: > - System hardware: $500,000 For one PB? Gold plated HDDs and cases I guess… > Assume 15% annual system maintenance over five years Do not run your hard drive cluster on the same floor as vibration-inducing machines, folks. /s According to Backblaze, even a 8 year-old hard drive is still twice more reliable than this, and 15% failure rate is what to expect over 6 years of consecutive use! > Assume the space needed to store 1PB is between 25%-50% of a standard rack (42U) You can fit almost 10PB in a rack though, so that's another figure that they've inflated. > Assume personnel costs to manage a 1PB system are 0.5 IT/storage admin FTE This one is probably true (and it's even an underestimation) if you have only 1PB of data, but it's not going to scale linearly: you're never going to need 130 FTE for 260PB. Overall, if a supplier (who's going to use the same kind of off-the-shelf tech as you would) claims that his price is 4 times cheaper that what it would cost you to run it by yourself, he's probably just lying to sell you stuff, just saying…
- jabart 3y ago
  Can you fit 10pb in a rack, sure if you want it all flash. You also need double that for mirroring, or quadruple that for raid 10. Double that if you want a mirrored copy of that array too. If you don't have high performance needs you can get by with slower hard drives. You also have the 25gb,40gb, or 100gb top of rack switch. Well you want two of those for redundancy. It starts to quickly add up. Annual system maintenance could be lower but is still a cost. Number 1 reason I'm in a datacenter is to swap a dead drive. Oct 2020, StorONE announced a 1PB All-Flash array for $499k. Other examples which are close to their pricing. https://www.reddit.com/r/storage/comments/vdr6ql/pricing_examples_for_1pb_cheap_enterprise_storage/ https://www.reddit.com/r/storage/comments/vdr6ql/pricing_exa...
  - littlestymaar 3y ago
    > Number 1 reason I'm in a datacenter is to swap a dead drive. No doubt about that, when you have tens of thousands of drive you end up replacing some of them all the time, but not 15% of them every year… > Oct 2020, StorONE announced a 1PB All-Flash array for $499k. With Optane and all, right? The kind of things that's a massive overkill if your workload allows you to use Glacier… > Other examples which are close to their pricing. In that list, the only offers that are close to this pricing (but still almost 20% cheaper) also includes several years of support (which you probably don't want if you're Canva's scale).

by jabart 3y ago

hinkley 3y ago

140 racks is a quite a lot of tonnage of AC as well. And probably hot and cold rows to make it function.

jabart 3y ago
Hot/cold was factored in not the AC or power needs. I don't know many places you could just casually ask for 100+ racks. Maybe our midwest datacenters are just smaller than what is on the coast.
- hinkley 3y ago
  The last place I worked that had that kind of capacity, I had access to the lobby of that building and that was about it. The server room I could access had space for <wiggles fingers> maybe 15-20 racks and in fact had maybe 7, which is probably why it was always cold as fuck in there and I didn’t like being in there much. Then there was a contract which had a server room twice the size and mostly contained an AS/3<mumble> and a couple racks of Intel hardware. You could probably get a ping pong table into that one. Giant rooms that are mostly white and too cold are a little freaky, and reminiscent of 2001. There have been a number of stories of people not being able to utilize their floor space for more servers because they ran out of space on the roof for AC units. At one point I recall Google was working on power dissipation because they had hit the max electrical code, and so no one would agree to run more power into their buildings. More servers meant more compute per watt and per ton of chillers.
  - jabart 3y ago
    AWS has a talk about things like this. They were, no idea if they still do, setup an AZ based on MWH figure for most efficiency because any higher the power company doesn't like it. Our local DC the water cooling and refrigerant cooling is larger than the 5 generators they have. I've crammed a lot of servers in my rack and I'm out of power plugs but still under 2kwh of what we are on contract for. Dense servers have really messed with datacenters.
- littlestymaar 3y ago
  > I don't know many places you could just casually ask for 100+ racks. That's obviously not something you can do on a whim, but at the same time we're talking about a seven figures investment plus the time to build the in-house team to manage this, so it's not going to happen overnight anyway.

littlestymaar 3y ago

This is a marketing pitch for a cloud storage company and it shows: > - System hardware: $500,000 For one PB? Gold plated HDDs and cases I guess… > Assume 15% annual system maintenance over five years Do not run your hard drive cluster on the same floor as vibration-inducing machines, folks. /s According to Backblaze, even a 8 year-old hard drive is still twice more reliable than this, and 15% failure rate is what to expect over 6 years of consecutive use! > Assume the space needed to store 1PB is between 25%-50% of a standard rack (42U) You can fit almost 10PB in a rack though, so that's another figure that they've inflated. > Assume personnel costs to manage a 1PB system are 0.5 IT/storage admin FTE This one is probably true (and it's even an underestimation) if you have only 1PB of data, but it's not going to scale linearly: you're never going to need 130 FTE for 260PB. Overall, if a supplier (who's going to use the same kind of off-the-shelf tech as you would) claims that his price is 4 times cheaper that what it would cost you to run it by yourself, he's probably just lying to sell you stuff, just saying…

jabart 3y ago
Can you fit 10pb in a rack, sure if you want it all flash. You also need double that for mirroring, or quadruple that for raid 10. Double that if you want a mirrored copy of that array too. If you don't have high performance needs you can get by with slower hard drives. You also have the 25gb,40gb, or 100gb top of rack switch. Well you want two of those for redundancy. It starts to quickly add up. Annual system maintenance could be lower but is still a cost. Number 1 reason I'm in a datacenter is to swap a dead drive. Oct 2020, StorONE announced a 1PB All-Flash array for $499k. Other examples which are close to their pricing. https://www.reddit.com/r/storage/comments/vdr6ql/pricing_examples_for_1pb_cheap_enterprise_storage/ https://www.reddit.com/r/storage/comments/vdr6ql/pricing_exa...
- littlestymaar 3y ago
  > Number 1 reason I'm in a datacenter is to swap a dead drive. No doubt about that, when you have tens of thousands of drive you end up replacing some of them all the time, but not 15% of them every year… > Oct 2020, StorONE announced a 1PB All-Flash array for $499k. With Optane and all, right? The kind of things that's a massive overkill if your workload allows you to use Glacier… > Other examples which are close to their pricing. In that list, the only offers that are close to this pricing (but still almost 20% cheaper) also includes several years of support (which you probably don't want if you're Canva's scale).