5 ms·

> When we started Oxide, the DC bus bar stood as one of the most glaring differences between the rack-scale machines at the hyperscalers and the rack-and-stack

> When we started Oxide, the DC bus bar stood as one of the most glaring differences between the rack-scale machines at the hyperscalers and the rack-and-stack servers that the rest of the market was stuck with. That a relatively simple piece of copper was unavailable to commercial buyers It seems that 0xide was founded in 2019 and Open Compute Project had been specifying dc bus bars for 6 years at that point. People could purchase racks if they wanted, but it seems like, by large, people didn't care enough to go whole hog in on it. Wonder if the economics have changed or if it's still just neat but won't move the needle.

walrus01 2y ago
Things like -48VDC bus bars in the 'telco' world significantly predate the OCP, all the way back to like 1952 in the Bell system. In general, the telco world concept hasn't changed much. You have AC grid power coming from your local utility into some BIG ASS RECTIFIERS which create -48VDC (and are responsible for charging your BIG ASS BATTERY BANK to float voltage), then various DC fuses/breakers going to distribution of -48VDC bus bars powering the equipment in a CO. Re: Open Compute, the general concept of what they did was go to a bunch of 1U/2U server power supply manufacturers and get them to make a series of 48VDC-to-12VDC power supplies (which can be 92%+ efficient), and cut out the need for legacy 5VDC feed from power supply into ATX-derived-design x86-64 motherboards.
- m463 2y ago
  I remember seeing an old telephone switching system from the 20's and I think it was 48vdc. Uncertain though.
  - ttyprintk 2y ago
    Yeah, would have been 48 vdc for line operations, 60 and up AC for the ring.
zamalek 2y ago
It's normally incredibly difficult for employees to disrupt at massive companies that would be the type which runs a data center. Disruption usually enters the corp in a sales deck, much like the one Oxide would have. It's stupid, but that's why we all have jobs.
- hnthrowaway0328 2y ago
  I think engineers should be more forceful to lead their own visions instead being led by accountants and lawyers. After engineers have the power of implementation and de-implementstion. They need to step into dirty politics and bend other people's views. It's either theirs or ours. Win-win is a fallacy.
  - philipov 2y ago
    Let me know how that works out for you!
  - andrewjf 2y ago
    Being able to navigate this is what differentiates a very senior IC (principal, distinguished, etc) and random employees.
    - orochimaaru 2y ago
      Yes. I think as an engineer at this level you need to also have the patience to deal with the bean counters. But as I’ve grown in my career I’ve actually found that line of thinking refreshing. Can you quantify benefit? If it requires too many assumptions it’s probably not worth it. But then again there’s always the Vp or the svp who wants to “showcase his towers’ innovative spirit” and then there goes money that could be used for better things. The innovative spirit of the day is random Llm apps.
  - hinkley 2y ago
    Once the accountants are convinced the entire company is about them, there’s not much the engineers can do. They just starve you out by refusing to buy anything. It’s a big reason why open source is as successful as it is. It’s free so they can’t stop you with the checkbook.
bigfatkitten 2y ago
OCP hardware is only really accessible to hyperscalers. You can't go out and just buy a rack or two, the Taiwanese OEMs don't do direct deals that small. Even if they did, no integration is done for you. You would have to integrate the compute hardware from one company, the network fabric from another company, and then the OS and everything else from yet another. That's a lot of risk, a lot of engineering resources, a lot of procurement overhead, and a lot of different vendors pointing fingers at each other when something doesn't work. If you're Amazon or Google, you can do this stuff yourself. If you're a normal company, you probably won't have the inhouse expertise. On the other hand, Oxide sells a turnkey IaaS platform that you can just roll off the pallet, plug in and start using immediately. You only need to pay one company, and you have one company to yell at if something goes wrong. You can buy a rack of 1-2U machines from Dell, HPE or Cisco with VMware or some other HCI platform, but you don't get that power efficiency or the really nice control plane Oxide have on their platform.
- leoc 2y ago
  But isn’t it a little surprising (I’m not an expert) that Dell or Supermicro or somefirm like that hadn’t already started offering an approachable access to either OCP gear or a proprietary knockoff of it? Presumably that may still happen if Oxide is seen to have proven the market.
  - unsnap_biceps 2y ago
    Supermicro does sell OCP racks. https://www.supermicro.com/solutions/Solution-Brief-Supermicro-OCP.pdf https://www.supermicro.com/solutions/Solution-Brief-Supermic... I recall them offering older versions of the specs but can't easily find a reference, so I might be wrong about how accessible they were.
  - kjellsbells 2y ago
    Azure tried this, not with their hyperscaler stuff, but with Azure Operator Nexus. Basically an "opinionated" combination of Dell, Arista, and Pure storage with a special Azure AKS running on top and a metric ton of management and orchestration smarts. The target customer base was telcos who needed local capabilities in their data centers and who might otherwise have gone to OCP. As far as I can surmise, it's dead, but not EOLed. Microsoft nuked the operator business unit earlier in the year, and judging by recent job postings from contract shops, AT&T might be the only customer.
indrora 2y ago
You simply can't buy OCP hardware is part of the issue, not new anyway. What you're going to find is "OCP Inspired" hardware that has some overlap with the full OCP specification but is almost always meant to run on 240VAC on 19in racks because nobody wants to invest the money in something that can't be bought from CDW.
- p_l 2y ago
  I remember the one time I had OCP hardware in data center, and how it was essentially rumoured it's better to not ask too much how it got there - not the level of "fell of a truck", but some possibility it was ex-(big tech) equipment acquired through favours, or some really insistent negotiating with Quanta till "to be sold to (big tech)" racks ended up with us
TZubiri 2y ago
One is the specs and the other is an actual implementation, what am I missing?
Sylamore 2y ago
HP BladeSystem p-series chassis were all DC bus bar powered back in the mid 2000s. You had a power enclosure which provided DC output to one or more chassis in a rack over the bus bar. We were glad to be rid of those blades but it wasn't because of their power configuration.

walrus01 2y ago

Things like -48VDC bus bars in the 'telco' world significantly predate the OCP, all the way back to like 1952 in the Bell system. In general, the telco world concept hasn't changed much. You have AC grid power coming from your local utility into some BIG ASS RECTIFIERS which create -48VDC (and are responsible for charging your BIG ASS BATTERY BANK to float voltage), then various DC fuses/breakers going to distribution of -48VDC bus bars powering the equipment in a CO. Re: Open Compute, the general concept of what they did was go to a bunch of 1U/2U server power supply manufacturers and get them to make a series of 48VDC-to-12VDC power supplies (which can be 92%+ efficient), and cut out the need for legacy 5VDC feed from power supply into ATX-derived-design x86-64 motherboards.

m463 2y ago
I remember seeing an old telephone switching system from the 20's and I think it was 48vdc. Uncertain though.
- ttyprintk 2y ago
  Yeah, would have been 48 vdc for line operations, 60 and up AC for the ring.

zamalek 2y ago

It's normally incredibly difficult for employees to disrupt at massive companies that would be the type which runs a data center. Disruption usually enters the corp in a sales deck, much like the one Oxide would have. It's stupid, but that's why we all have jobs.

hnthrowaway0328 2y ago
I think engineers should be more forceful to lead their own visions instead being led by accountants and lawyers. After engineers have the power of implementation and de-implementstion. They need to step into dirty politics and bend other people's views. It's either theirs or ours. Win-win is a fallacy.
- philipov 2y ago
  Let me know how that works out for you!
- andrewjf 2y ago
  Being able to navigate this is what differentiates a very senior IC (principal, distinguished, etc) and random employees.
  - orochimaaru 2y ago
    Yes. I think as an engineer at this level you need to also have the patience to deal with the bean counters. But as I’ve grown in my career I’ve actually found that line of thinking refreshing. Can you quantify benefit? If it requires too many assumptions it’s probably not worth it. But then again there’s always the Vp or the svp who wants to “showcase his towers’ innovative spirit” and then there goes money that could be used for better things. The innovative spirit of the day is random Llm apps.
- hinkley 2y ago
  Once the accountants are convinced the entire company is about them, there’s not much the engineers can do. They just starve you out by refusing to buy anything. It’s a big reason why open source is as successful as it is. It’s free so they can’t stop you with the checkbook.

bigfatkitten 2y ago

OCP hardware is only really accessible to hyperscalers. You can't go out and just buy a rack or two, the Taiwanese OEMs don't do direct deals that small. Even if they did, no integration is done for you. You would have to integrate the compute hardware from one company, the network fabric from another company, and then the OS and everything else from yet another. That's a lot of risk, a lot of engineering resources, a lot of procurement overhead, and a lot of different vendors pointing fingers at each other when something doesn't work. If you're Amazon or Google, you can do this stuff yourself. If you're a normal company, you probably won't have the inhouse expertise. On the other hand, Oxide sells a turnkey IaaS platform that you can just roll off the pallet, plug in and start using immediately. You only need to pay one company, and you have one company to yell at if something goes wrong. You can buy a rack of 1-2U machines from Dell, HPE or Cisco with VMware or some other HCI platform, but you don't get that power efficiency or the really nice control plane Oxide have on their platform.

leoc 2y ago
But isn’t it a little surprising (I’m not an expert) that Dell or Supermicro or somefirm like that hadn’t already started offering an approachable access to either OCP gear or a proprietary knockoff of it? Presumably that may still happen if Oxide is seen to have proven the market.
- unsnap_biceps 2y ago
  Supermicro does sell OCP racks. https://www.supermicro.com/solutions/Solution-Brief-Supermicro-OCP.pdf https://www.supermicro.com/solutions/Solution-Brief-Supermic... I recall them offering older versions of the specs but can't easily find a reference, so I might be wrong about how accessible they were.
- kjellsbells 2y ago
  Azure tried this, not with their hyperscaler stuff, but with Azure Operator Nexus. Basically an "opinionated" combination of Dell, Arista, and Pure storage with a special Azure AKS running on top and a metric ton of management and orchestration smarts. The target customer base was telcos who needed local capabilities in their data centers and who might otherwise have gone to OCP. As far as I can surmise, it's dead, but not EOLed. Microsoft nuked the operator business unit earlier in the year, and judging by recent job postings from contract shops, AT&T might be the only customer.

indrora 2y ago

You simply can't buy OCP hardware is part of the issue, not new anyway. What you're going to find is "OCP Inspired" hardware that has some overlap with the full OCP specification but is almost always meant to run on 240VAC on 19in racks because nobody wants to invest the money in something that can't be bought from CDW.

p_l 2y ago
I remember the one time I had OCP hardware in data center, and how it was essentially rumoured it's better to not ask too much how it got there - not the level of "fell of a truck", but some possibility it was ex-(big tech) equipment acquired through favours, or some really insistent negotiating with Quanta till "to be sold to (big tech)" racks ended up with us

TZubiri 2y ago

One is the specs and the other is an actual implementation, what am I missing?

Sylamore 2y ago

HP BladeSystem p-series chassis were all DC bus bar powered back in the mid 2000s. You had a power enclosure which provided DC output to one or more chassis in a rack over the bus bar. We were glad to be rid of those blades but it wasn't because of their power configuration.