6 ms·

We didn't use anything from OCP. When we first started the company, we thought we might use the enclosure (and considered ourselves "OCP inspired"), but there e

We didn't use anything from OCP. When we first started the company, we thought we might use the enclosure (and considered ourselves "OCP inspired"), but there ended up being little value in doing so (and there was a clear cost). And on the stuff that we really cared about (e.g., getting rid of the traditional BMC), we were completely at odds with OCP (where ASPEED BMCs abound!). So in the end, even the mechanical and power didn't come from OCP. We clearly build on other components (we didn't build own rectifiers!), but we absolutely built the machine from first principles.

chubot 1y ago
Ah OK, thanks for clarifying! I'm glad to see these kinds of machines being built, especially with so much open source software
jeffbee 1y ago
How does the Oxide 48V architecture compare to what is known about the Google/OCP architecture? Does Oxide use single stage conversion, intermediate 12V buses, or ??
- steveklabnik 1y ago
  I'm not someone who works on this part of the product, but we talk a little about this stuff here: * https://oxide.computer/blog/how-oxide-cuts-data-center-power-consumption-in-half https://oxide.computer/blog/how-oxide-cuts-data-center-power... * https://docs.oxide.computer/guides/introduction https://docs.oxide.computer/guides/introduction I feel like we had a good Oxide and Friends on this too... https://oxide-and-friends.transistor.fm/episodes/bringing-up-cosmo https://oxide-and-friends.transistor.fm/episodes/bringing-up... has some info about our power setup. Anyway, I barely know anything about this topic, but I think this answers your immediate question: we convert AC -> DC once at the rack level, and then use a bus bar to distribute that to each sled. Each sled also has a converter to convert that 54V down to 12V for its own bus within each sled.
  - jeffbee 1y ago
    Probably the sweet spot, since you can get to market fast with known 12V designs, but still enjoy the possibility of later announcing you've made the sled even more efficient by getting rid of the intermediate voltage!
    - syntheticgate 1y ago
      At a certain point in EE power design you don't really want to go from 54V -> point of load for every rail (1.8V, 1.1V, 0.9V, SVI3 rails etc), so sticking with an intermediate voltage makes sense often even when viewing this from an efficiency perspective. Voltages such as 54V require different creepage and clearance requirements, so saddling every point of load regulator (of which we have many many!) with those requirements is often detrimental to an already complex board layout. Picking something like 12V or 24V as an intermediate voltage helps balance those requirements with the amount of copper you need for power delivery since the parts use low voltages but are extremely power hungry so your current at the point of load rail is a lot. This also means that your point of load regulators have to be distributed around the board near their loads otherwise the copper losses and noise would become problematic.
      - jeffbee 1y ago
        It's certainly the current mainstream style to have an intermediate voltage rail of 12V or more. But this OCP talk from a few years ago was interesting, showing a prototype direct 48V-1V conversion with high efficiency. https://www.youtube.com/watch?v=JQHiKIfrwI0 https://www.youtube.com/watch?v=JQHiKIfrwI0
        syntheticgate 1y ago
        Yes it certainly can be done, but there's a cost and design complexity with doing that too. I did a quick count of gimlet (our server sled's) power rails and got to over 26 different power domains, and I probably missed a few in my quick scan! It's unclear to me if the efficiency gains from re-doing these with something more exotic to go from 54V would make enough of a difference to justify doing so, and we'd still end up with some stuff like the SVI2/3 controllers needing an intermediate rail (or have to go design those ourselves too) and some analog rails needing LDOs for noise rejection reasons etc. As mentioned before, creepage and clearance at higher voltages cascades into layout complexity and pain if you have to run it everywhere on the board: but for the same reasons we're talking about this, we can't very well do a 54V:1V conversion in the back of the sled and run it all the way to the front- losses, noise etc. As with all things engineering is a series of tradeoffs and right now, an IBC from 54V to 12V has been a reasonable design point for us.
hinkley 1y ago
My read on the situation is that you copied OCP only inasmuch as OCP made some observations about physics and you made the same observations. The most obvious change is that you guys use half-width, full length enclosures. But I'm realizing now that I haven't looked at the OCP specs in a long, long time. I recall the common DC rail from some early Facebook papers on this topic, and I thought they were pretty similar to how you plug into power. I just looked at the OCP power connector, and I would have lost any bet anyone was willing to make me about what they looked like. That's not at all where I thought we were. I think I understand now why you guys went to such pains getting the keyed connectors to work exactly right. Their power connectors look like something from a scifi movie, and not in a good way.

chubot 1y ago

Ah OK, thanks for clarifying! I'm glad to see these kinds of machines being built, especially with so much open source software

jeffbee 1y ago

How does the Oxide 48V architecture compare to what is known about the Google/OCP architecture? Does Oxide use single stage conversion, intermediate 12V buses, or ??

steveklabnik 1y ago
I'm not someone who works on this part of the product, but we talk a little about this stuff here: * https://oxide.computer/blog/how-oxide-cuts-data-center-power-consumption-in-half https://oxide.computer/blog/how-oxide-cuts-data-center-power... * https://docs.oxide.computer/guides/introduction https://docs.oxide.computer/guides/introduction I feel like we had a good Oxide and Friends on this too... https://oxide-and-friends.transistor.fm/episodes/bringing-up-cosmo https://oxide-and-friends.transistor.fm/episodes/bringing-up... has some info about our power setup. Anyway, I barely know anything about this topic, but I think this answers your immediate question: we convert AC -> DC once at the rack level, and then use a bus bar to distribute that to each sled. Each sled also has a converter to convert that 54V down to 12V for its own bus within each sled.
- jeffbee 1y ago
  Probably the sweet spot, since you can get to market fast with known 12V designs, but still enjoy the possibility of later announcing you've made the sled even more efficient by getting rid of the intermediate voltage!
  - syntheticgate 1y ago
    At a certain point in EE power design you don't really want to go from 54V -> point of load for every rail (1.8V, 1.1V, 0.9V, SVI3 rails etc), so sticking with an intermediate voltage makes sense often even when viewing this from an efficiency perspective. Voltages such as 54V require different creepage and clearance requirements, so saddling every point of load regulator (of which we have many many!) with those requirements is often detrimental to an already complex board layout. Picking something like 12V or 24V as an intermediate voltage helps balance those requirements with the amount of copper you need for power delivery since the parts use low voltages but are extremely power hungry so your current at the point of load rail is a lot. This also means that your point of load regulators have to be distributed around the board near their loads otherwise the copper losses and noise would become problematic.
    - jeffbee 1y ago
      It's certainly the current mainstream style to have an intermediate voltage rail of 12V or more. But this OCP talk from a few years ago was interesting, showing a prototype direct 48V-1V conversion with high efficiency. https://www.youtube.com/watch?v=JQHiKIfrwI0 https://www.youtube.com/watch?v=JQHiKIfrwI0
      - syntheticgate 1y ago
        Yes it certainly can be done, but there's a cost and design complexity with doing that too. I did a quick count of gimlet (our server sled's) power rails and got to over 26 different power domains, and I probably missed a few in my quick scan! It's unclear to me if the efficiency gains from re-doing these with something more exotic to go from 54V would make enough of a difference to justify doing so, and we'd still end up with some stuff like the SVI2/3 controllers needing an intermediate rail (or have to go design those ourselves too) and some analog rails needing LDOs for noise rejection reasons etc. As mentioned before, creepage and clearance at higher voltages cascades into layout complexity and pain if you have to run it everywhere on the board: but for the same reasons we're talking about this, we can't very well do a 54V:1V conversion in the back of the sled and run it all the way to the front- losses, noise etc. As with all things engineering is a series of tradeoffs and right now, an IBC from 54V to 12V has been a reasonable design point for us.

hinkley 1y ago

My read on the situation is that you copied OCP only inasmuch as OCP made some observations about physics and you made the same observations. The most obvious change is that you guys use half-width, full length enclosures. But I'm realizing now that I haven't looked at the OCP specs in a long, long time. I recall the common DC rail from some early Facebook papers on this topic, and I thought they were pretty similar to how you plug into power. I just looked at the OCP power connector, and I would have lost any bet anyone was willing to make me about what they looked like. That's not at all where I thought we were. I think I understand now why you guys went to such pains getting the keyed connectors to work exactly right. Their power connectors look like something from a scifi movie, and not in a good way.