4 ms·

> As someone who's only dealt with commodity server hardware, these specs make me salivate Commodity servers have the same specs. Here’s a reseller of SuperMi

> As someone who's only dealt with commodity server hardware, these specs make me salivate Commodity servers have the same specs. Here’s a reseller of SuperMicro servers, where you can buy similar compute on the cheap. https://www.siliconmechanics.com/systems/servers/rackform https://www.siliconmechanics.com/systems/servers/rackform

EvanAnderson 3y ago
I haven't worked with SuperMicro (aside from having it inside "appliance" devices that I've worked adjacent to), but I assume my experience with Dell and HP commodity servers are similar. The thick layer of hardware contrivances necessary to maintain IBM PC compatibility is unnecessary for the task of bulk hosting of x86/x64 VMs. There's a lot of hardware and software that just doesn't need to be there. Bare metal out-of-band management ends up being bolted-on to these "legacy" contrivances (scraping video memory for remote consoles, faking being USB peripherals). A serial console or SSH connection to a service processor would be vastly superior. I can't begin to count how many times an iDRAC "lied" to me about issues with a machine, or how many times the solution was "upgrade the iDRAC firmware and reboot it". I have been mostly unimpressed with the quality of firmware for motherboards, baseboard management controllers, RAID adapters, NICs, HBAs, power supplies, backplanes, front panels, etc. Every new model of system or component ends up being an exercise in fear / anticipation of problems. The integrator has very little power over the firmware quality and I can be assured that if I do have a firmware-induced issue I'm many, many steps away from actually communicating with somebody who can help. Granted, maybe if I was buying at the scale of Oxide's prospective Customers I'd have some pull with the integrators, but I'm skeptical of that, even. Oxide is actually building computers. Putting commodity motherboards into boxes with other commodity components won't ever have the level of attention to detail and integration that Oxide can provide.
sjdmdlakziggy 3y ago
They are not even close to the same specs.
- upon_drumhead 3y ago
  https://www.siliconmechanics.com/system/rackform-a235.v8.1/21444 https://www.siliconmechanics.com/system/rackform-a235.v8.1/2... This is pretty darn close to the "Gimlet" Compute Sleds One 225W TDP 64-core AMD Milan CPU 1 TiB of DRAM across 16 DIMMs 12 front-facing hot-swappable PCIe Gen 4 U.2 storage devices 2 internal M.2 devices 2 ports of 100 GbE networking Point for point, the same hardware as specified.
intelVISA 3y ago
As much as I wish Oxide the best, most of us here would probably prefer our own libre, hand-crafted OS atop Coreboot. (Or maybe I speak for myself only...)
Aurornis 3y ago
> Commodity servers have the same specs. Probably not a coincidence. It would be interesting to know which ODM they partnered with for the hardware. I've done some work with SuperMicro in the past. Some of their boards come with extensive headers and customization options right out of the box. They're also happy to work on board level customizations with the right contracts in place.
- bcantrill 3y ago
  We didn't work with an ODM: the ODMs were unwilling to contemplate some of the most basic things we needed (e.g., replacing the BMC with a much lighter weight service processor, having a true root-of-trust, etc.) -- let alone the more things we wanted to do (e.g., our own switch). The compute sled and the switch are both of our own design and look nothing like what you'll find from an ODM; if you're curious in the details, we have discussed them quite a bit in our Oxide and Friends podcast.[0][1][2] [0] https://oxide-and-friends.transistor.fm/episodes/tales-from-the-bringup-lab-2021-12-06 https://oxide-and-friends.transistor.fm/episodes/tales-from-... [1] https://oxide-and-friends.transistor.fm/episodes/the-sidecar-switch-2021-11-29 https://oxide-and-friends.transistor.fm/episodes/the-sidecar... [2] https://oxide-and-friends.transistor.fm/episodes/bringup-lab-chronicles-a-measurement-two-years-in-the-making https://oxide-and-friends.transistor.fm/episodes/bringup-lab...
  - mst 3y ago
    Please consider generating some sort of automated transcript from these. I'd hope your target audience would understand the limitations of such a thing, and I'm probably not the only person who'd rather read than listen even with the obvious caveats. (these days automated transcripts seem to be no harder to mentally fix up the errors in as I read them than "somebody typing fast on a software keyboard and suffering the inevitable tyop and autocorrupt related issues" is, though of course others' mileage may vary)
    - actionfromafar 3y ago
      They could probably have someone proofread the transcripts, there aren't that many episodes.

EvanAnderson 3y ago

I haven't worked with SuperMicro (aside from having it inside "appliance" devices that I've worked adjacent to), but I assume my experience with Dell and HP commodity servers are similar. The thick layer of hardware contrivances necessary to maintain IBM PC compatibility is unnecessary for the task of bulk hosting of x86/x64 VMs. There's a lot of hardware and software that just doesn't need to be there. Bare metal out-of-band management ends up being bolted-on to these "legacy" contrivances (scraping video memory for remote consoles, faking being USB peripherals). A serial console or SSH connection to a service processor would be vastly superior. I can't begin to count how many times an iDRAC "lied" to me about issues with a machine, or how many times the solution was "upgrade the iDRAC firmware and reboot it". I have been mostly unimpressed with the quality of firmware for motherboards, baseboard management controllers, RAID adapters, NICs, HBAs, power supplies, backplanes, front panels, etc. Every new model of system or component ends up being an exercise in fear / anticipation of problems. The integrator has very little power over the firmware quality and I can be assured that if I do have a firmware-induced issue I'm many, many steps away from actually communicating with somebody who can help. Granted, maybe if I was buying at the scale of Oxide's prospective Customers I'd have some pull with the integrators, but I'm skeptical of that, even. Oxide is actually building computers. Putting commodity motherboards into boxes with other commodity components won't ever have the level of attention to detail and integration that Oxide can provide.

sjdmdlakziggy 3y ago

They are not even close to the same specs.

upon_drumhead 3y ago
https://www.siliconmechanics.com/system/rackform-a235.v8.1/21444 https://www.siliconmechanics.com/system/rackform-a235.v8.1/2... This is pretty darn close to the "Gimlet" Compute Sleds One 225W TDP 64-core AMD Milan CPU 1 TiB of DRAM across 16 DIMMs 12 front-facing hot-swappable PCIe Gen 4 U.2 storage devices 2 internal M.2 devices 2 ports of 100 GbE networking Point for point, the same hardware as specified.

intelVISA 3y ago

As much as I wish Oxide the best, most of us here would probably prefer our own libre, hand-crafted OS atop Coreboot. (Or maybe I speak for myself only...)

Aurornis 3y ago

> Commodity servers have the same specs. Probably not a coincidence. It would be interesting to know which ODM they partnered with for the hardware. I've done some work with SuperMicro in the past. Some of their boards come with extensive headers and customization options right out of the box. They're also happy to work on board level customizations with the right contracts in place.

bcantrill 3y ago
We didn't work with an ODM: the ODMs were unwilling to contemplate some of the most basic things we needed (e.g., replacing the BMC with a much lighter weight service processor, having a true root-of-trust, etc.) -- let alone the more things we wanted to do (e.g., our own switch). The compute sled and the switch are both of our own design and look nothing like what you'll find from an ODM; if you're curious in the details, we have discussed them quite a bit in our Oxide and Friends podcast.[0][1][2] [0] https://oxide-and-friends.transistor.fm/episodes/tales-from-the-bringup-lab-2021-12-06 https://oxide-and-friends.transistor.fm/episodes/tales-from-... [1] https://oxide-and-friends.transistor.fm/episodes/the-sidecar-switch-2021-11-29 https://oxide-and-friends.transistor.fm/episodes/the-sidecar... [2] https://oxide-and-friends.transistor.fm/episodes/bringup-lab-chronicles-a-measurement-two-years-in-the-making https://oxide-and-friends.transistor.fm/episodes/bringup-lab...
- mst 3y ago
  Please consider generating some sort of automated transcript from these. I'd hope your target audience would understand the limitations of such a thing, and I'm probably not the only person who'd rather read than listen even with the obvious caveats. (these days automated transcripts seem to be no harder to mentally fix up the errors in as I read them than "somebody typing fast on a software keyboard and suffering the inevitable tyop and autocorrupt related issues" is, though of course others' mileage may vary)
  - actionfromafar 3y ago
    They could probably have someone proofread the transcripts, there aren't that many episodes.