3 ms·

I know they don't use SSD due to cost and it's probably excessive performance for their needs. But I wonder how their costs compare when you factor in the densi

I know they don't use SSD due to cost and it's probably excessive performance for their needs. But I wonder how their costs compare when you factor in the density and energy efficiencies. You could fit a lot of silicon in the space a storage post has for 45 3.5" drives. I found their 2014 blog post about a 500PB new datacenter, and their recent 2018 post is about crossing 500PB. So does storage per sqr/foot ever matter or is it just better to get another DC?

pjc50 9y ago
SSD is not a great long-term archival medium as it's not passively stable in the way that magnetised media is, and it's still much more expensive.
chx 9y ago
A spinning disk dissipates about 6W on average, for 100 000 disks that's 600kw. With an average of 0.12 USD paid for a kwh times 24 hours, their energy bill is less than 2000 USD a day. Their HDD cost based on the "81.76 — The number of hard drives that were installed each day in 2017." and we know they mostly now do 8TB at roughly 0.02 USD per GB according to https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/ https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/ which is 13000 USD a day. That means even if we had SSDs at zero watts, they would need to be (13+2/13) ~ 15% more expensive than HDDs in order to break even but if you look at https://pcpartpicker.com/trends/price/internal-hard-drive/ https://pcpartpicker.com/trends/price/internal-hard-drive/ you will see it's still about 8 times as expensive. Obviously there's some savings due to floor space but the savings are going to be really low there. And this is just the economics side.
AdamJacobMuller 9y ago
In addition to the other points, there are other issues besides just cramming a raw number of bytes into a raw amount of space. You need to balance disk space with IO capacity with compute with memory. Simply replacing all of the 3.5" drives with an equivalent volume of SSDs probably leaves you well under-balanced on the memory/compute side and way overbalanced on the IO side. In addition, you need to develop hardware to interface with way more drives per chassis (massive-scale expanders). There'd be a lot of engineering costs associated with getting that much denser in terms of the number of drives per U (figure 6 or 8 times as many per U). Of course, this could pay off if there was a cost benefit to denser servers, but, datacenter space at their scale is primarily priced at a function of how much (and what quality) power you need. Being denser isn't ultimately going save them much if any money.

pjc50 9y ago

SSD is not a great long-term archival medium as it's not passively stable in the way that magnetised media is, and it's still much more expensive.

chx 9y ago

A spinning disk dissipates about 6W on average, for 100 000 disks that's 600kw. With an average of 0.12 USD paid for a kwh times 24 hours, their energy bill is less than 2000 USD a day. Their HDD cost based on the "81.76 — The number of hard drives that were installed each day in 2017." and we know they mostly now do 8TB at roughly 0.02 USD per GB according to https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/ https://www.backblaze.com/blog/hard-drive-cost-per-gigabyte/ which is 13000 USD a day. That means even if we had SSDs at zero watts, they would need to be (13+2/13) ~ 15% more expensive than HDDs in order to break even but if you look at https://pcpartpicker.com/trends/price/internal-hard-drive/ https://pcpartpicker.com/trends/price/internal-hard-drive/ you will see it's still about 8 times as expensive. Obviously there's some savings due to floor space but the savings are going to be really low there. And this is just the economics side.

AdamJacobMuller 9y ago

In addition to the other points, there are other issues besides just cramming a raw number of bytes into a raw amount of space. You need to balance disk space with IO capacity with compute with memory. Simply replacing all of the 3.5" drives with an equivalent volume of SSDs probably leaves you well under-balanced on the memory/compute side and way overbalanced on the IO side. In addition, you need to develop hardware to interface with way more drives per chassis (massive-scale expanders). There'd be a lot of engineering costs associated with getting that much denser in terms of the number of drives per U (figure 6 or 8 times as many per U). Of course, this could pay off if there was a cost benefit to denser servers, but, datacenter space at their scale is primarily priced at a function of how much (and what quality) power you need. Being denser isn't ultimately going save them much if any money.