6 ms·

This strikes me as a gigantic coordinated form of commoditize your complement.

by mlthoughts2018 8y ago

noir_lord 8y ago
Makes sense but has nice second order effects. If we could standardise more we can optimise for power efficiency, things like more DC in DCs
- baybal2 8y ago
  FIY, the original telecom rack equipment been standardised on 48V since time immemorial. So, at 48V you can simply use off the shelf telecom power supplies
  - StudentStuff 8y ago
    I've been trying to find cheap 48v power supplies for 1A and less telcom gear, but they seem to be pretty spendy. Cheaper to buy old gear that includes the PSU than order the PSU itself.
- shereadsthenews 8y ago
  Actually it seems to me that standardization hampers the last bit of efficiency. Just one example that grinds my gears is the LEDs on the reference designs. Nobody needs these but there they are, drawing current 24x7 in a dark warehouse in the middle of nowhere.
  - remarkEon 8y ago
    Can’t they just be turned off after power-on, or flipped on as needed?
kayhi 8y ago
how so? like commoditizing Oracle databases?
walrus01 8y ago
DC power wise, the original reason why they did this is that it's grossly inefficient to have a 42RU worth of 1U servers, each with its own individual 110-240VAC to 12/5VDC power supply in it. Or the equivalent of that with four-servers-in-2RU type setups, but also with AC to DC power supplies. They centralized the AC to DC conversion in a single unit in the rack, feeding either 277VAC or 480VAC to each rack, and ran 12VDC to each server. The new system wisely moves from 12VDC to 48VDC (same as most telecom equipment), and probably has basic DC-DC converters in each server unit for 48 to 12VDC conversion. They've also gone with custom motherboards that entirely eliminate the 5VDC rails which are distributed by 'normal' ATX server power supplies.
- baybal2 8y ago
  There are 90%+ efficient reasonably cheap flex psus, but the mainstream is around 80%. So, you would not be saving much if you only have access to 110 or 220 mains. The economy changes though if you can get access to industrial 3 phase linkup at 400V, and can avoid additional low step down transformers.
  - bigbangchina 8y ago
    You can use microwave oven transformer to step to 1000VDC then simply use step down converter in each rack. I am doing this for my home lab with 100 servers.
  - vanderZwan 8y ago
    > So, you would not be saving much if you only have access to 110 or 220 mains. "Much" is a relative term here, no? Like, doesn't that kind of depend at the scale at which Facebook operates?
  - detaro 8y ago
    I read somewhere that part of it was also doing the short-term battery backup on a rack-level, without needing conversion back to mains AC and then again back to DC in the server for it.
- altmind 8y ago
meruru 8y ago
>commoditize your complement You talk like that's a well-known concept, but as far as I know it has been coined only one year ago in this essay: https://www.gwern.net/Complement https://www.gwern.net/Complement
- 0815test 8y ago
  It was described in detail much earlier than that, in ESR's The Magic Cauldron, part of The Cathedral and the Bazaar: "...When the development of the open-source X window system was funded by DEC in the 1980s, their explicit goal was to 'reset the competition'. At the time there were several competing alternative graphics environments for Unix in play, notably including Sun Microsystems's NeWS system. DEC strategists believed (probably correctly) that if Sun were able to establish a proprietary graphics standard it would get a lock on the booming Unix-workstation market. By funding X and lending it engineers, and by allying with many smaller vendors to establish X as a de-facto standard, DEC was able to neutralize advantages held by Sun and other competitors with more in-house expertise in graphics. This moved the focus of competition in the workstation market towards hardware, where DEC was historically strong. ..." from http://www.catb.org/~esr/writings/cathedral-bazaar/magic-cauldron/ar01s11.html http://www.catb.org/~esr/writings/cathedral-bazaar/magic-cau...
- codetrotter 8y ago
  The essay you linked attributes the concept to Joel Spolsky, saying that he wrote about it in 2002 and quoting from that. So the concept was coined quite long ago.
  - meruru 8y ago
    The idea is old, but I didn't think people were referring to it by name before. Searching for "commoditize your complement" I get first recent results discussing the essay I linked, though digging deeper there are quite a few linking to the Spolsky post as well.

This strikes me as a gigantic coordinated form of commoditize your complement.

Makes sense but has nice second order effects. If we could standardise more we can optimise for power efficiency, things like more DC in DCs

baybal2 8y ago
FIY, the original telecom rack equipment been standardised on 48V since time immemorial. So, at 48V you can simply use off the shelf telecom power supplies
- StudentStuff 8y ago
  I've been trying to find cheap 48v power supplies for 1A and less telcom gear, but they seem to be pretty spendy. Cheaper to buy old gear that includes the PSU than order the PSU itself.
shereadsthenews 8y ago
Actually it seems to me that standardization hampers the last bit of efficiency. Just one example that grinds my gears is the LEDs on the reference designs. Nobody needs these but there they are, drawing current 24x7 in a dark warehouse in the middle of nowhere.
- remarkEon 8y ago
  Can’t they just be turned off after power-on, or flipped on as needed?

kayhi 8y ago

how so? like commoditizing Oracle databases?

walrus01 8y ago

DC power wise, the original reason why they did this is that it's grossly inefficient to have a 42RU worth of 1U servers, each with its own individual 110-240VAC to 12/5VDC power supply in it. Or the equivalent of that with four-servers-in-2RU type setups, but also with AC to DC power supplies. They centralized the AC to DC conversion in a single unit in the rack, feeding either 277VAC or 480VAC to each rack, and ran 12VDC to each server. The new system wisely moves from 12VDC to 48VDC (same as most telecom equipment), and probably has basic DC-DC converters in each server unit for 48 to 12VDC conversion. They've also gone with custom motherboards that entirely eliminate the 5VDC rails which are distributed by 'normal' ATX server power supplies.

baybal2 8y ago
There are 90%+ efficient reasonably cheap flex psus, but the mainstream is around 80%. So, you would not be saving much if you only have access to 110 or 220 mains. The economy changes though if you can get access to industrial 3 phase linkup at 400V, and can avoid additional low step down transformers.
- bigbangchina 8y ago
  You can use microwave oven transformer to step to 1000VDC then simply use step down converter in each rack. I am doing this for my home lab with 100 servers.
- vanderZwan 8y ago
  > So, you would not be saving much if you only have access to 110 or 220 mains. "Much" is a relative term here, no? Like, doesn't that kind of depend at the scale at which Facebook operates?
- detaro 8y ago
  I read somewhere that part of it was also doing the short-term battery backup on a rack-level, without needing conversion back to mains AC and then again back to DC in the server for it.
altmind 8y ago

meruru 8y ago

>commoditize your complement You talk like that's a well-known concept, but as far as I know it has been coined only one year ago in this essay: https://www.gwern.net/Complement https://www.gwern.net/Complement

0815test 8y ago
It was described in detail much earlier than that, in ESR's The Magic Cauldron, part of The Cathedral and the Bazaar: "...When the development of the open-source X window system was funded by DEC in the 1980s, their explicit goal was to 'reset the competition'. At the time there were several competing alternative graphics environments for Unix in play, notably including Sun Microsystems's NeWS system. DEC strategists believed (probably correctly) that if Sun were able to establish a proprietary graphics standard it would get a lock on the booming Unix-workstation market. By funding X and lending it engineers, and by allying with many smaller vendors to establish X as a de-facto standard, DEC was able to neutralize advantages held by Sun and other competitors with more in-house expertise in graphics. This moved the focus of competition in the workstation market towards hardware, where DEC was historically strong. ..." from http://www.catb.org/~esr/writings/cathedral-bazaar/magic-cauldron/ar01s11.html http://www.catb.org/~esr/writings/cathedral-bazaar/magic-cau...
codetrotter 8y ago
The essay you linked attributes the concept to Joel Spolsky, saying that he wrote about it in 2002 and quoting from that. So the concept was coined quite long ago.
- meruru 8y ago
  The idea is old, but I didn't think people were referring to it by name before. Searching for "commoditize your complement" I get first recent results discussing the essay I linked, though digging deeper there are quite a few linking to the Spolsky post as well.