4 ms·

I would say it's very clear and logical ... but is it really "from first principles"? I thought that Oxide was based on OpenCompute, which is basically the rac

by chubot 1y ago

I would say it's very clear and logical ... but is it really "from first principles"? I thought that Oxide was based on OpenCompute, which is basically the rack designs that Facebook open sourced, after hiring some Google employees to build their custom data centers. This project started in 2011: https://en.wikipedia.org/wiki/Open_Compute_Project https://en.wikipedia.org/wiki/Open_Compute_Project https://github.com/opencomputeproject https://github.com/opencomputeproject Google definitely rebuilt the stack from first principles -- the data center was basically a huge embedded system, from power to racks to CPUs/memory/disk/network to kernel to user space to cluster software. (And no, they did not use Kubernetes.) I have no idea how active the OpenCompute project is -- is Facebook still the main contributor, and are they still releasing their new rack designs? And I also wonder how much Oxide has diverged from it? At least on the hardware side. On the software side, I guess the Illumos-derived parts and Rust parts are completely different. Well, when they say they did their own: - board designs - microcontroller OS - platform enablement software - host hypervisor - switch - integrated storage service - control plane Then yeah it seems like maybe only board designs and the switch COULD have either come from or been influenced by OpenCompute, but maybe those didn't either. (I have no idea tbh) Maybe they only got the mechanical and power stuff from OpenCompute? i.e. the parts that change more slowly

steveklabnik 1y ago
We initially talked about OpenCompute, yeah, but as far as I know we ended up moving away from that years ago.
bcantrill 1y ago
We didn't use anything from OCP. When we first started the company, we thought we might use the enclosure (and considered ourselves "OCP inspired"), but there ended up being little value in doing so (and there was a clear cost). And on the stuff that we really cared about (e.g., getting rid of the traditional BMC), we were completely at odds with OCP (where ASPEED BMCs abound!). So in the end, even the mechanical and power didn't come from OCP. We clearly build on other components (we didn't build own rectifiers!), but we absolutely built the machine from first principles.
- chubot 1y ago
  Ah OK, thanks for clarifying! I'm glad to see these kinds of machines being built, especially with so much open source software
- jeffbee 1y ago
  How does the Oxide 48V architecture compare to what is known about the Google/OCP architecture? Does Oxide use single stage conversion, intermediate 12V buses, or ??
  - steveklabnik 1y ago
    I'm not someone who works on this part of the product, but we talk a little about this stuff here: * https://oxide.computer/blog/how-oxide-cuts-data-center-power-consumption-in-half https://oxide.computer/blog/how-oxide-cuts-data-center-power... * https://docs.oxide.computer/guides/introduction https://docs.oxide.computer/guides/introduction I feel like we had a good Oxide and Friends on this too... https://oxide-and-friends.transistor.fm/episodes/bringing-up-cosmo https://oxide-and-friends.transistor.fm/episodes/bringing-up... has some info about our power setup. Anyway, I barely know anything about this topic, but I think this answers your immediate question: we convert AC -> DC once at the rack level, and then use a bus bar to distribute that to each sled. Each sled also has a converter to convert that 54V down to 12V for its own bus within each sled.
    - jeffbee 1y ago
      Probably the sweet spot, since you can get to market fast with known 12V designs, but still enjoy the possibility of later announcing you've made the sled even more efficient by getting rid of the intermediate voltage!
      - syntheticgate 1y ago
        At a certain point in EE power design you don't really want to go from 54V -> point of load for every rail (1.8V, 1.1V, 0.9V, SVI3 rails etc), so sticking with an intermediate voltage makes sense often even when viewing this from an efficiency perspective. Voltages such as 54V require different creepage and clearance requirements, so saddling every point of load regulator (of which we have many many!) with those requirements is often detrimental to an already complex board layout. Picking something like 12V or 24V as an intermediate voltage helps balance those requirements with the amount of copper you need for power delivery since the parts use low voltages but are extremely power hungry so your current at the point of load rail is a lot. This also means that your point of load regulators have to be distributed around the board near their loads otherwise the copper losses and noise would become problematic.

by chubot 1y ago

steveklabnik 1y ago

We initially talked about OpenCompute, yeah, but as far as I know we ended up moving away from that years ago.

bcantrill 1y ago

We didn't use anything from OCP. When we first started the company, we thought we might use the enclosure (and considered ourselves "OCP inspired"), but there ended up being little value in doing so (and there was a clear cost). And on the stuff that we really cared about (e.g., getting rid of the traditional BMC), we were completely at odds with OCP (where ASPEED BMCs abound!). So in the end, even the mechanical and power didn't come from OCP. We clearly build on other components (we didn't build own rectifiers!), but we absolutely built the machine from first principles.

chubot 1y ago
Ah OK, thanks for clarifying! I'm glad to see these kinds of machines being built, especially with so much open source software
jeffbee 1y ago
How does the Oxide 48V architecture compare to what is known about the Google/OCP architecture? Does Oxide use single stage conversion, intermediate 12V buses, or ??
- steveklabnik 1y ago
  I'm not someone who works on this part of the product, but we talk a little about this stuff here: * https://oxide.computer/blog/how-oxide-cuts-data-center-power-consumption-in-half https://oxide.computer/blog/how-oxide-cuts-data-center-power... * https://docs.oxide.computer/guides/introduction https://docs.oxide.computer/guides/introduction I feel like we had a good Oxide and Friends on this too... https://oxide-and-friends.transistor.fm/episodes/bringing-up-cosmo https://oxide-and-friends.transistor.fm/episodes/bringing-up... has some info about our power setup. Anyway, I barely know anything about this topic, but I think this answers your immediate question: we convert AC -> DC once at the rack level, and then use a bus bar to distribute that to each sled. Each sled also has a converter to convert that 54V down to 12V for its own bus within each sled.
  - jeffbee 1y ago
    Probably the sweet spot, since you can get to market fast with known 12V designs, but still enjoy the possibility of later announcing you've made the sled even more efficient by getting rid of the intermediate voltage!
    - syntheticgate 1y ago
      At a certain point in EE power design you don't really want to go from 54V -> point of load for every rail (1.8V, 1.1V, 0.9V, SVI3 rails etc), so sticking with an intermediate voltage makes sense often even when viewing this from an efficiency perspective. Voltages such as 54V require different creepage and clearance requirements, so saddling every point of load regulator (of which we have many many!) with those requirements is often detrimental to an already complex board layout. Picking something like 12V or 24V as an intermediate voltage helps balance those requirements with the amount of copper you need for power delivery since the parts use low voltages but are extremely power hungry so your current at the point of load rail is a lot. This also means that your point of load regulators have to be distributed around the board near their loads otherwise the copper losses and noise would become problematic.