3 ms·

"For example, one of our test production runs hit a complete standstill when we realized that the data center personnel simply could not move the racks. Since t

"For example, one of our test production runs hit a complete standstill when we realized that the data center personnel simply could not move the racks. Since these racks were a modification of the OpenVault system, we used the same rack castors that allowed us to easily roll the racks into place. But the inclusion of 480 4 TB drives drove the weight to over 1,100 kg, effectively crushing the rubber wheels. This was the first time we'd ever deployed a rack that heavy, and it wasn't something we'd originally baked into the development plan!" That's just fun. In some way I'm amazed that nobody of the smart people at FB tough of the weight of this stuff. But then again, I probably made sillier mistakes myself on a much smaller scale.

mbubb 11y ago
Fascinating stuff. My guess is that they were definitely aware of the weight tolerance of the floor and the tiles but looks like the weak spot was the canister wheels on the bottom of the rack. I wonder how they eventually got it out. They probably used one of those rack jacks to get a stronger dolly underneath... Or removed the HDDs and moved it. This made me wonder about something else - the power density. They say the layout: "can support up to one exabyte (1,000 PB) per data hall. Since storage density generally increases as technology advances, this was the baseline from which we started. In other words, there's plenty of room to grow." I wonder how they account for increased energy consumption as they put in larger powersupplies to deal with the increase in power to run and cooling for the larger HDDs. I imagine in this layout everything is in proportion. CPU / RAM / Disk Space. It must be hard to accurately target growth with in the constraints of a limited power footprint.
- pcl 11y ago
  It sounds like they actually reduced the overall power requirements significantly, by only delivering power to a subset of the drives at any one time.
- andyidsinga 11y ago
  afaik - as density increases per disk, ex going from 1TB to 4TB, power efficiency per TB should be better too as long as they are also not also increasing the total number of spindles by adding new storage nodes. One thing I'm not sure of is if _total load_ for the hall increases as increased accesses may occur because of having more density. The hard disk power management part is really fascinating in terms of keeping the storage cluster software alive and well to be able to respond to requests while keep the hard disks off. A low power SSD cache tier on the front may be part of the solution.

mbubb 11y ago

Fascinating stuff. My guess is that they were definitely aware of the weight tolerance of the floor and the tiles but looks like the weak spot was the canister wheels on the bottom of the rack. I wonder how they eventually got it out. They probably used one of those rack jacks to get a stronger dolly underneath... Or removed the HDDs and moved it. This made me wonder about something else - the power density. They say the layout: "can support up to one exabyte (1,000 PB) per data hall. Since storage density generally increases as technology advances, this was the baseline from which we started. In other words, there's plenty of room to grow." I wonder how they account for increased energy consumption as they put in larger powersupplies to deal with the increase in power to run and cooling for the larger HDDs. I imagine in this layout everything is in proportion. CPU / RAM / Disk Space. It must be hard to accurately target growth with in the constraints of a limited power footprint.

pcl 11y ago
It sounds like they actually reduced the overall power requirements significantly, by only delivering power to a subset of the drives at any one time.
andyidsinga 11y ago
afaik - as density increases per disk, ex going from 1TB to 4TB, power efficiency per TB should be better too as long as they are also not also increasing the total number of spindles by adding new storage nodes. One thing I'm not sure of is if _total load_ for the hall increases as increased accesses may occur because of having more density. The hard disk power management part is really fascinating in terms of keeping the storage cluster software alive and well to be able to respond to requests while keep the hard disks off. A low power SSD cache tier on the front may be part of the solution.