3 ms·

I'm not someone who works on this part of the product, but we talk a little about this stuff here: * https://oxide.computer/blog/how-oxide-cuts-data-center-pow

I'm not someone who works on this part of the product, but we talk a little about this stuff here: * https://oxide.computer/blog/how-oxide-cuts-data-center-power-consumption-in-half https://oxide.computer/blog/how-oxide-cuts-data-center-power... * https://docs.oxide.computer/guides/introduction https://docs.oxide.computer/guides/introduction I feel like we had a good Oxide and Friends on this too... https://oxide-and-friends.transistor.fm/episodes/bringing-up-cosmo https://oxide-and-friends.transistor.fm/episodes/bringing-up... has some info about our power setup. Anyway, I barely know anything about this topic, but I think this answers your immediate question: we convert AC -> DC once at the rack level, and then use a bus bar to distribute that to each sled. Each sled also has a converter to convert that 54V down to 12V for its own bus within each sled.

jeffbee 1y ago
Probably the sweet spot, since you can get to market fast with known 12V designs, but still enjoy the possibility of later announcing you've made the sled even more efficient by getting rid of the intermediate voltage!
- syntheticgate 1y ago
  At a certain point in EE power design you don't really want to go from 54V -> point of load for every rail (1.8V, 1.1V, 0.9V, SVI3 rails etc), so sticking with an intermediate voltage makes sense often even when viewing this from an efficiency perspective. Voltages such as 54V require different creepage and clearance requirements, so saddling every point of load regulator (of which we have many many!) with those requirements is often detrimental to an already complex board layout. Picking something like 12V or 24V as an intermediate voltage helps balance those requirements with the amount of copper you need for power delivery since the parts use low voltages but are extremely power hungry so your current at the point of load rail is a lot. This also means that your point of load regulators have to be distributed around the board near their loads otherwise the copper losses and noise would become problematic.
  - jeffbee 1y ago
    It's certainly the current mainstream style to have an intermediate voltage rail of 12V or more. But this OCP talk from a few years ago was interesting, showing a prototype direct 48V-1V conversion with high efficiency. https://www.youtube.com/watch?v=JQHiKIfrwI0 https://www.youtube.com/watch?v=JQHiKIfrwI0
    - syntheticgate 1y ago
      Yes it certainly can be done, but there's a cost and design complexity with doing that too. I did a quick count of gimlet (our server sled's) power rails and got to over 26 different power domains, and I probably missed a few in my quick scan! It's unclear to me if the efficiency gains from re-doing these with something more exotic to go from 54V would make enough of a difference to justify doing so, and we'd still end up with some stuff like the SVI2/3 controllers needing an intermediate rail (or have to go design those ourselves too) and some analog rails needing LDOs for noise rejection reasons etc. As mentioned before, creepage and clearance at higher voltages cascades into layout complexity and pain if you have to run it everywhere on the board: but for the same reasons we're talking about this, we can't very well do a 54V:1V conversion in the back of the sled and run it all the way to the front- losses, noise etc. As with all things engineering is a series of tradeoffs and right now, an IBC from 54V to 12V has been a reasonable design point for us.
  - hinkley 1y ago
    The thermal dissipation is a function of the current squared. The heat in the conductor is a function of the size of the conductor and the surface area for heat dissipation. So these high current common rail systems you can sometimes see in youtube videos for power distribution, have great honking bars of copper in them. And in most of the videos I've seen, the video is about someone screwing one of these up, damaging the bar, and now the electrician has to wait for a new one to arrive, because they are shipped from far away and they are expensive per pound, and the dumb things weigh many pounds. So you don't actually want the power to be at 12V for very long in a power dense rack. Their spec sheet says that each rack can pull 15KW. And that's wired for 208 or 3-phase power. That's 10 hair dryers of power per rack, so yeah maybe you shouldn't step it down until the last responsible moment. Do any parts of the rack run at the full 54V? That would make for some very nice cooling fans.

jeffbee 1y ago

Probably the sweet spot, since you can get to market fast with known 12V designs, but still enjoy the possibility of later announcing you've made the sled even more efficient by getting rid of the intermediate voltage!

syntheticgate 1y ago
At a certain point in EE power design you don't really want to go from 54V -> point of load for every rail (1.8V, 1.1V, 0.9V, SVI3 rails etc), so sticking with an intermediate voltage makes sense often even when viewing this from an efficiency perspective. Voltages such as 54V require different creepage and clearance requirements, so saddling every point of load regulator (of which we have many many!) with those requirements is often detrimental to an already complex board layout. Picking something like 12V or 24V as an intermediate voltage helps balance those requirements with the amount of copper you need for power delivery since the parts use low voltages but are extremely power hungry so your current at the point of load rail is a lot. This also means that your point of load regulators have to be distributed around the board near their loads otherwise the copper losses and noise would become problematic.
- jeffbee 1y ago
  It's certainly the current mainstream style to have an intermediate voltage rail of 12V or more. But this OCP talk from a few years ago was interesting, showing a prototype direct 48V-1V conversion with high efficiency. https://www.youtube.com/watch?v=JQHiKIfrwI0 https://www.youtube.com/watch?v=JQHiKIfrwI0
  - syntheticgate 1y ago
    Yes it certainly can be done, but there's a cost and design complexity with doing that too. I did a quick count of gimlet (our server sled's) power rails and got to over 26 different power domains, and I probably missed a few in my quick scan! It's unclear to me if the efficiency gains from re-doing these with something more exotic to go from 54V would make enough of a difference to justify doing so, and we'd still end up with some stuff like the SVI2/3 controllers needing an intermediate rail (or have to go design those ourselves too) and some analog rails needing LDOs for noise rejection reasons etc. As mentioned before, creepage and clearance at higher voltages cascades into layout complexity and pain if you have to run it everywhere on the board: but for the same reasons we're talking about this, we can't very well do a 54V:1V conversion in the back of the sled and run it all the way to the front- losses, noise etc. As with all things engineering is a series of tradeoffs and right now, an IBC from 54V to 12V has been a reasonable design point for us.
- hinkley 1y ago
  The thermal dissipation is a function of the current squared. The heat in the conductor is a function of the size of the conductor and the surface area for heat dissipation. So these high current common rail systems you can sometimes see in youtube videos for power distribution, have great honking bars of copper in them. And in most of the videos I've seen, the video is about someone screwing one of these up, damaging the bar, and now the electrician has to wait for a new one to arrive, because they are shipped from far away and they are expensive per pound, and the dumb things weigh many pounds. So you don't actually want the power to be at 12V for very long in a power dense rack. Their spec sheet says that each rack can pull 15KW. And that's wired for 208 or 3-phase power. That's 10 hair dryers of power per rack, so yeah maybe you shouldn't step it down until the last responsible moment. Do any parts of the rack run at the full 54V? That would make for some very nice cooling fans.