3 ms·

About $1m to build 21 PB of live storage (not tape), and $50k/year to power it (not factoring maintenance, rack space cost, and cooling) A service offering liv

About $1m to build 21 PB of live storage (not tape), and $50k/year to power it (not factoring maintenance, rack space cost, and cooling) A service offering live storage at this (Petabyte) scale [1] - costs 22,020,096 GB * 3¢/GB * 12mo = $7,927,234.56 a year. You could build six fully-mirrored copies for about that much, though management of 360 4U machines (30x 48U full-height racks worth) is a business on its own (see BackBlaze photos), and you need to deal with drive replacement after a point. [1] http://www.rsync.net/products/petabyte.html http://www.rsync.net/products/petabyte.html (If you actually sign up for this, be sure to send me that $24,000 referral bonus o_o) Math: -- Machine Cost -- $300 * 45 * 8TB drives = $13,500 (drive cost) + $3,387.28 (pod cost) == $16,887.28 (unit cost) / 360TB = $46.91 per TB $46.91/TB * (21 PB * 1024 == 21,504 TB) == $1,008,752.64 / 21 PB (for no redundancy) -- Power Cost -- Let's say the motherboard/controllers draw around 100W continuous and each drive pulls 11W (from my post elsewhere in the comments). This is 2,688 drives, which fit into 60x 45-drive 4U BackBlaze storage pods. 2,688 * 11W = 29,568 W (for the drives) + 6,000 W (for the computers) = 35.568 kW to run a non-redundant 21 PB array. You'll need at least two network switches and moderate cooling infrastructure as well. The heat from using 35.5 kWh continuously takes nontrivial effort to disperse, but let's assume that's somehow free for the sake of simplicity here. 35.568 kW * 24 * 365 = 311.57568 MWh (in a year) I'm sure at this point you might be able to work out a bulk deal of some kind, but at $0.16/kWh you're talking a pretty low cost of $49,852.11 in power every year. ---- Building arrays of nonredundant live drives would be significantly cheaper over the long term than Amazon Glacier (which is almost certainly cold storage, not live) at this scale.

cshimmin 12y ago
Right but Glacier _will_ provide redundancy. If you introduce even the most basic redundancy to your math (multiply everything by 2), it pretty nearly closes the gap with Glacier's costs. You also forgot to add in the costs of paying at least one human (probably a few) to set all this up and maintain it forever. The real question is, why do they need this archive to be spinning? What's wrong with tape?
ghaff 12y ago
>Amazon Glacier (which is tape-based!) Citation? Not saying you're wrong but I've seen supposedly "in the know" commentary in favor of both the tape-based and some sort of cold disk-based storage. I've been curious and also a bit surprised that the true answer has never really hit publicly although I'm sure it's well-known within some communities.
- lunixbochs 12y ago
  Oh, you're right. I haven't thought about this for a while, and I remember my previous assumption about tape storage being challenged. I'll replace that part with "cold storage". After refreshing my memory: Amazon outright they're using denied tape backup, some third-parties pointed out cold spinning disks wouldn't be very economical, and a few people theorized it's actually BDXL.
  - ghaff 12y ago
    Cold storage is probably safe. I've seen the assertion that it's tape from someone who I wouldn't have thought would get it wrong but I'd now bet against it whether or not Amazon's denial seals the deal. Robin Harris has argued in favor of BDXL but he's speculated about a number of things. http://storagemojo.com/2014/04/25/amazons-glacier-secret-bdxl/ http://storagemojo.com/2014/04/25/amazons-glacier-secret-bdx... My money would be on the simplest: cold disk. Some support here: http://mjtsai.com/blog/2014/04/27/how-is-amazon-glacier-implemented/ http://mjtsai.com/blog/2014/04/27/how-is-amazon-glacier-impl... but it's all speculative and anecdotal. I don't have a problem with the thought that Amazon is getting out ahead of the drive economics, especially given that I'm not sure how widely used Glacier is today. I do find it interesting that this has been apparently kept under wraps from anyone who would share more widely.
    - res0nat0r 12y ago
      Someone that worked set amazon at the time Glacier came out a while back commented that it is some type of extremely low power hard drive that backs this system. Which makes sense due to the 1-4 hour delay in retrieval time. The racks are idle and only come online one every 4 hours or so and batch retrieve/store their data then power down. This small power usage is probably what makes them profit on this very slim margin.
deleted 12y ago
[deleted]
jacobsenscott 12y ago
For a more accurate comparison to amazon multiply by at least 3 for redundancy, don't forget to factor in the cost of the real estate, replacement equipment, employees to run it all, etc. You would be hard pressed to beat amazon prices, which are falling over time.
- lunixbochs 12y ago
  You're absolutely right. I was aiming for a baseline/ballpark here. At this scale, redundancy via 3x mirror is pretty expensive. You could likely get away with something like chunks of 12 drives in raidz3, especially if you're planning on having at least a few facilities like this. $1m setup + land + a few salaries upkeep sounds like the kind of thing you could convince a few governments to do. However, Glacier won't do at all for the live Internet Archive, so it's a question of whether they want this to have live replication or not (if the whole backup is offline, who does your integrity checks?) I'm actually curious how much it would money/storage space it would require to print the entire Internet Archive on whatever the current equivalent of microfiche is (anything cheap and more permanent than magnetic or optical storage).

cshimmin 12y ago

Right but Glacier _will_ provide redundancy. If you introduce even the most basic redundancy to your math (multiply everything by 2), it pretty nearly closes the gap with Glacier's costs. You also forgot to add in the costs of paying at least one human (probably a few) to set all this up and maintain it forever. The real question is, why do they need this archive to be spinning? What's wrong with tape?

ghaff 12y ago

>Amazon Glacier (which is tape-based!) Citation? Not saying you're wrong but I've seen supposedly "in the know" commentary in favor of both the tape-based and some sort of cold disk-based storage. I've been curious and also a bit surprised that the true answer has never really hit publicly although I'm sure it's well-known within some communities.

lunixbochs 12y ago
Oh, you're right. I haven't thought about this for a while, and I remember my previous assumption about tape storage being challenged. I'll replace that part with "cold storage". After refreshing my memory: Amazon outright they're using denied tape backup, some third-parties pointed out cold spinning disks wouldn't be very economical, and a few people theorized it's actually BDXL.
- ghaff 12y ago
  Cold storage is probably safe. I've seen the assertion that it's tape from someone who I wouldn't have thought would get it wrong but I'd now bet against it whether or not Amazon's denial seals the deal. Robin Harris has argued in favor of BDXL but he's speculated about a number of things. http://storagemojo.com/2014/04/25/amazons-glacier-secret-bdxl/ http://storagemojo.com/2014/04/25/amazons-glacier-secret-bdx... My money would be on the simplest: cold disk. Some support here: http://mjtsai.com/blog/2014/04/27/how-is-amazon-glacier-implemented/ http://mjtsai.com/blog/2014/04/27/how-is-amazon-glacier-impl... but it's all speculative and anecdotal. I don't have a problem with the thought that Amazon is getting out ahead of the drive economics, especially given that I'm not sure how widely used Glacier is today. I do find it interesting that this has been apparently kept under wraps from anyone who would share more widely.
  - res0nat0r 12y ago
    Someone that worked set amazon at the time Glacier came out a while back commented that it is some type of extremely low power hard drive that backs this system. Which makes sense due to the 1-4 hour delay in retrieval time. The racks are idle and only come online one every 4 hours or so and batch retrieve/store their data then power down. This small power usage is probably what makes them profit on this very slim margin.

deleted 12y ago

[deleted]

jacobsenscott 12y ago

For a more accurate comparison to amazon multiply by at least 3 for redundancy, don't forget to factor in the cost of the real estate, replacement equipment, employees to run it all, etc. You would be hard pressed to beat amazon prices, which are falling over time.

lunixbochs 12y ago
You're absolutely right. I was aiming for a baseline/ballpark here. At this scale, redundancy via 3x mirror is pretty expensive. You could likely get away with something like chunks of 12 drives in raidz3, especially if you're planning on having at least a few facilities like this. $1m setup + land + a few salaries upkeep sounds like the kind of thing you could convince a few governments to do. However, Glacier won't do at all for the live Internet Archive, so it's a question of whether they want this to have live replication or not (if the whole backup is offline, who does your integrity checks?) I'm actually curious how much it would money/storage space it would require to print the entire Internet Archive on whatever the current equivalent of microfiche is (anything cheap and more permanent than magnetic or optical storage).