6 ms·

Why does Facebook's Open Rack use a nonstandard rack size? That seems like an obvious barrier for adoption of hardware that was designed to be a commodity.

DiabloD3 11y ago
Racks were designed to fit telecom hardware originally. The Open Rack size is designed around common computer hardware sizes. It isn't a barrier for adoption because swapping racks out of a datacenter is easy, and they fit on standard datacenter floor tiles. What is a barrier is that damned 48V. Disclaimer: I run a hosting company.
- rdtsc 11y ago
  > What is a barrier is that damned 48V. Interesting what is the issue with 48V, I saw equipment for it seemed to be overpriced. Remember pricing out some stuff and as soon as 48V option for power came into play then price rose quite a bit. Or is it that voltage is not high enough to be efficient for a large data center?
  - DiabloD3 11y ago
    The voltage is high enough. The way Facebook's solution is, is you have triplets of racks: left, right racks hold computers, middle rack holds network, power distribution, and UPS. The computers have extremely simplistic power supplies that basically can't fail, and just DC->DC transform from 48v to 12, 5, and 3.3; and the large scale power supplies that convert three phase 240v (or whatever you're supplying it with) from the datacenter to 48V are much higher efficiency than the ones that would have been in the server (which you would have fed them, usually, something like single phase 208v). Redundancy is supplied by just hooking multiple transformers in the middle rack to the + and - terminals on each PSU, instead of a convoluted multi-module redundant PSU (which always uses a single backplane, and backplanes in redundant PSUs fail surprisingly frequently). The total round trip efficiency of this system is about as high as you can realistically get. 80Plus Titanium is 90-95% efficient (depending on load), but has efficiency losses in rack level distribution which 48V tries to correct. However, 48V DC can be very dangerous to work with, and a lot of tech workers refuse to work with it. Now, if you believe it is dangerous or not (I've seen arguments stating that it is no more dangerous than single phase 208v) is immaterial, this is the opinion of a lot of workers. The cost of 48V gear is expensive if you're not in a datacenter already setup to handle it. Facebook obviously doesn't have this problem because they build entire datacenters from scratch. I personally don't believe in it because it doesn't buy me anything that single phase 208v doesn't give me, I do not pay enough in electricity to have the overhead of dealing with it.
    - patrickg_zill 11y ago
      It is not the voltage it is the high number of amps that DC systems have. While you could get fried from a single rack server's failing power supply, it is likely that it will die of some other cause before zapping you if you take simple precautions. DC on the other hand, puts "the forces of nature" up close and personal to the back of the rack. A wrong move will vaporize the thickest of metal screwdrivers and can easily do irreparable damage to a human. Something as simple as not removing your wedding ring can result in an inadvertent bridge between + and - ; after which, bad things happen.
      - DiabloD3 11y ago
        Correct, 48VDC exchanges volts for amps. Basic electrical theory states volts * amps = watts. We measure computer power usage in watts for a reason. 48VDC isn't sufficient to vaporize a metal screwdriver at the amperages used in most 48V datacenters, however, I know people who work with higher voltages than that, and they own expensive ceramic tools just for safety reasons. If I personally ran 48VDC, and my workers requested ceramic tools, I would not hesitate to expense those. And yes, if you're working around DC >12v, you should be seriously using every method available to make sure you don't become a conductor, including only using one hand at a time to touch power relays (to avoid having your heart stopped).
    - raverbashing 11y ago
      48V danger is debatable, but it might be a good idea to start adding RCD protection to the power sources (yes, this will bring your rack down (or just one output), but better than having people getting stuck to wires or tools causing short circuits)
      - bogomipz 11y ago
        RCD? Yeah, its usually the AMPs that kills you not the voltage.
      - jermy 11y ago
        Is this possible? Fault currents are much more obvious at a higher voltage than lower, and I thought most RCDs would not be made to be sensitive enough to catch what might be a dangerous DC leakage.
- jzwinck 11y ago
  Swapping racks may not be possible if you don't own the racks. Lots of datacenters are built with their own racks. You might be able to ask for extra deep racks but that's it.
  - DiabloD3 11y ago
    It isn't about rack depth, but rack width. They are deep, but not unusually so. Datacenters are not "built" with racks. They are not permanently affixed to the floor. Most datacenters do not keep empty racks on the datacenter floor, and keep the floor open. If you are renting by the rack, no, most datacenters won't swap racks for you. You'd need to be renting entire cages for them to consider it. Open Rack isn't really useful for small scale providers, it's more use to hyperscale companies like Facebook, Google, Amazon, and Microsoft. I don't think anyone we'd consider "medium scale" has adopted it (if I'm wrong, I'd love to see a story hit the front page about it). Only one of those companies have adopted it, the others have considered it and, although they manufacture custom hardware and thus could take advantage of it easily, they have not done it.
    - StreamBright 11y ago
      I think Amazon does not want to use OpenRack, not sure if they consider it. I think they are in the position to negotiate good price from normal x86 server vendors or make them produce a modified version of a normal server (without the things Amazon does not need). I am curious what is up with them nowadays.
- bogomipz 11y ago
  Can you explain how and why 48V is a barrier to begin with?
bluedino 11y ago
The Open Rack’s equipment bay has been widened from 19 inches to 21 inches, but it can still accommodate standard 19-inch equipment. A wider 21-inch bay, however, enables some “interesting configurations”, like installing three motherboards or five 3.5-inch disk drives side-by-side in one chassis. The outer width of the rack has remained a standard 24 inches to accommodate standard floor tiles.

DiabloD3 11y ago

Racks were designed to fit telecom hardware originally. The Open Rack size is designed around common computer hardware sizes. It isn't a barrier for adoption because swapping racks out of a datacenter is easy, and they fit on standard datacenter floor tiles. What is a barrier is that damned 48V. Disclaimer: I run a hosting company.

rdtsc 11y ago
> What is a barrier is that damned 48V. Interesting what is the issue with 48V, I saw equipment for it seemed to be overpriced. Remember pricing out some stuff and as soon as 48V option for power came into play then price rose quite a bit. Or is it that voltage is not high enough to be efficient for a large data center?
- DiabloD3 11y ago
  The voltage is high enough. The way Facebook's solution is, is you have triplets of racks: left, right racks hold computers, middle rack holds network, power distribution, and UPS. The computers have extremely simplistic power supplies that basically can't fail, and just DC->DC transform from 48v to 12, 5, and 3.3; and the large scale power supplies that convert three phase 240v (or whatever you're supplying it with) from the datacenter to 48V are much higher efficiency than the ones that would have been in the server (which you would have fed them, usually, something like single phase 208v). Redundancy is supplied by just hooking multiple transformers in the middle rack to the + and - terminals on each PSU, instead of a convoluted multi-module redundant PSU (which always uses a single backplane, and backplanes in redundant PSUs fail surprisingly frequently). The total round trip efficiency of this system is about as high as you can realistically get. 80Plus Titanium is 90-95% efficient (depending on load), but has efficiency losses in rack level distribution which 48V tries to correct. However, 48V DC can be very dangerous to work with, and a lot of tech workers refuse to work with it. Now, if you believe it is dangerous or not (I've seen arguments stating that it is no more dangerous than single phase 208v) is immaterial, this is the opinion of a lot of workers. The cost of 48V gear is expensive if you're not in a datacenter already setup to handle it. Facebook obviously doesn't have this problem because they build entire datacenters from scratch. I personally don't believe in it because it doesn't buy me anything that single phase 208v doesn't give me, I do not pay enough in electricity to have the overhead of dealing with it.
  - patrickg_zill 11y ago
    It is not the voltage it is the high number of amps that DC systems have. While you could get fried from a single rack server's failing power supply, it is likely that it will die of some other cause before zapping you if you take simple precautions. DC on the other hand, puts "the forces of nature" up close and personal to the back of the rack. A wrong move will vaporize the thickest of metal screwdrivers and can easily do irreparable damage to a human. Something as simple as not removing your wedding ring can result in an inadvertent bridge between + and - ; after which, bad things happen.
    - DiabloD3 11y ago
      Correct, 48VDC exchanges volts for amps. Basic electrical theory states volts * amps = watts. We measure computer power usage in watts for a reason. 48VDC isn't sufficient to vaporize a metal screwdriver at the amperages used in most 48V datacenters, however, I know people who work with higher voltages than that, and they own expensive ceramic tools just for safety reasons. If I personally ran 48VDC, and my workers requested ceramic tools, I would not hesitate to expense those. And yes, if you're working around DC >12v, you should be seriously using every method available to make sure you don't become a conductor, including only using one hand at a time to touch power relays (to avoid having your heart stopped).
  - raverbashing 11y ago
    48V danger is debatable, but it might be a good idea to start adding RCD protection to the power sources (yes, this will bring your rack down (or just one output), but better than having people getting stuck to wires or tools causing short circuits)
    - bogomipz 11y ago
      RCD? Yeah, its usually the AMPs that kills you not the voltage.
    - jermy 11y ago
      Is this possible? Fault currents are much more obvious at a higher voltage than lower, and I thought most RCDs would not be made to be sensitive enough to catch what might be a dangerous DC leakage.
jzwinck 11y ago
Swapping racks may not be possible if you don't own the racks. Lots of datacenters are built with their own racks. You might be able to ask for extra deep racks but that's it.
- DiabloD3 11y ago
  It isn't about rack depth, but rack width. They are deep, but not unusually so. Datacenters are not "built" with racks. They are not permanently affixed to the floor. Most datacenters do not keep empty racks on the datacenter floor, and keep the floor open. If you are renting by the rack, no, most datacenters won't swap racks for you. You'd need to be renting entire cages for them to consider it. Open Rack isn't really useful for small scale providers, it's more use to hyperscale companies like Facebook, Google, Amazon, and Microsoft. I don't think anyone we'd consider "medium scale" has adopted it (if I'm wrong, I'd love to see a story hit the front page about it). Only one of those companies have adopted it, the others have considered it and, although they manufacture custom hardware and thus could take advantage of it easily, they have not done it.
  - StreamBright 11y ago
    I think Amazon does not want to use OpenRack, not sure if they consider it. I think they are in the position to negotiate good price from normal x86 server vendors or make them produce a modified version of a normal server (without the things Amazon does not need). I am curious what is up with them nowadays.
bogomipz 11y ago
Can you explain how and why 48V is a barrier to begin with?

bluedino 11y ago

The Open Rack’s equipment bay has been widened from 19 inches to 21 inches, but it can still accommodate standard 19-inch equipment. A wider 21-inch bay, however, enables some “interesting configurations”, like installing three motherboards or five 3.5-inch disk drives side-by-side in one chassis. The outer width of the rack has remained a standard 24 inches to accommodate standard floor tiles.