5 ms·

Because Supermicro ships boards with things BMCs (ASPEED 2500s and the like) that have firmware they don't even control or support. That's a problem if you have

Because Supermicro ships boards with things BMCs (ASPEED 2500s and the like) that have firmware they don't even control or support. That's a problem if you have thousands of servers deployed and there's a security vulnerability or defect, you go to the vendor for support, and they just shrug. Or problems with DRAM training. Or any other weird BIOS/firmware quirk that Supermicro won't help you with. This stuff is less of a problem if you're just using a few dozen of these things at most. But at scale commodity hardware becomes a nightmare. It's why most hyperscalers find it more cost-effective to just build their own (Amazon, Google, Meta, others)

throwawaymaths 4y ago
Awesome, thanks this was helpful! I think also for some reason my brain forgot the definition of hyperscaler despite having used it (correctly) 6 or so years ago, I don't really work with servers anymore
zozbot234 4y ago
> Because Supermicro ships boards with things BMCs (ASPEED 2500s and the like) that have firmware they don't even control or support. That's not a Supermicro-specific problem, though. In fact, the Oxide Computers folks are predicting their whole business approach on being able to "control and support" most every piece of firmware that they ship to users. It's not easy, and it's far from standard in the industry.
- rcxdude 4y ago
  Yeah, Oxide's basic pitch is 'hyperscaling toys for the rest of us', where they essentially commoditize the kind of hardware and software setup which previously only the hyperscalers had (having built it themselves and only for themselves).
rocket_surgeron 4y ago
Hyperscalers either stick that same ASPEED BMC in their systems or roll their own solution based off USB-over-ethernet and/or some kind of janky I2C hack that is quite frankly an incomprehensibly terrifying safety nightmare of its own. And by "hyperscalar" I mean "the ODM who actually designed the system". Both Microsoft and Meta née Facebook hyperscaled their data centers using the AST1050. All of the various Open Cloudserver specs call for BMCs: https://www.opencompute.org/wiki/Server/SpecsAndDesigns-old https://www.opencompute.org/wiki/Server/SpecsAndDesigns-old Things haven't changed that much from these specifications.
dilyevsky 4y ago
I think supermicro supports openbmc on their top shelf line now. Not sure if Amazon builds most of their boxes in-house at least with Google it was not because of security but because they don't actually need most of the garbage that ships on the generic oem board.
- geerlingguy 4y ago
  Indeed; one of their 'Edge' models with the Altra Max was the one I tested with ServeTheHome[1], and it is running OpenBMC. [1] https://youtu.be/UT5UbSJOyog https://youtu.be/UT5UbSJOyog
  - dilyevsky 4y ago
    Ah cool, previously only seen it on their “hyperscaler” intel boxes. You mentioned you benched the m1 mini in that video, do you have the numbers published anywhere?
- cjg_ 4y ago
  Amazon creates their own security chips since several years with the Nitro system: https://aws.amazon.com/ec2/nitro/ https://aws.amazon.com/ec2/nitro/
  - rbanffy 4y ago
    The reasoning they don't need to build anything in their servers that's not needed still stands. It doesn't make sense if you need just a few units of a given type, but hyperscalers, who deploy computers by the acre, any unused SAS connector is dead weight.

throwawaymaths 4y ago

Awesome, thanks this was helpful! I think also for some reason my brain forgot the definition of hyperscaler despite having used it (correctly) 6 or so years ago, I don't really work with servers anymore

zozbot234 4y ago

> Because Supermicro ships boards with things BMCs (ASPEED 2500s and the like) that have firmware they don't even control or support. That's not a Supermicro-specific problem, though. In fact, the Oxide Computers folks are predicting their whole business approach on being able to "control and support" most every piece of firmware that they ship to users. It's not easy, and it's far from standard in the industry.

rcxdude 4y ago
Yeah, Oxide's basic pitch is 'hyperscaling toys for the rest of us', where they essentially commoditize the kind of hardware and software setup which previously only the hyperscalers had (having built it themselves and only for themselves).

rocket_surgeron 4y ago

Hyperscalers either stick that same ASPEED BMC in their systems or roll their own solution based off USB-over-ethernet and/or some kind of janky I2C hack that is quite frankly an incomprehensibly terrifying safety nightmare of its own. And by "hyperscalar" I mean "the ODM who actually designed the system". Both Microsoft and Meta née Facebook hyperscaled their data centers using the AST1050. All of the various Open Cloudserver specs call for BMCs: https://www.opencompute.org/wiki/Server/SpecsAndDesigns-old https://www.opencompute.org/wiki/Server/SpecsAndDesigns-old Things haven't changed that much from these specifications.

dilyevsky 4y ago

I think supermicro supports openbmc on their top shelf line now. Not sure if Amazon builds most of their boxes in-house at least with Google it was not because of security but because they don't actually need most of the garbage that ships on the generic oem board.

geerlingguy 4y ago
Indeed; one of their 'Edge' models with the Altra Max was the one I tested with ServeTheHome[1], and it is running OpenBMC. [1] https://youtu.be/UT5UbSJOyog https://youtu.be/UT5UbSJOyog
- dilyevsky 4y ago
  Ah cool, previously only seen it on their “hyperscaler” intel boxes. You mentioned you benched the m1 mini in that video, do you have the numbers published anywhere?
cjg_ 4y ago
Amazon creates their own security chips since several years with the Nitro system: https://aws.amazon.com/ec2/nitro/ https://aws.amazon.com/ec2/nitro/
- rbanffy 4y ago
  The reasoning they don't need to build anything in their servers that's not needed still stands. It doesn't make sense if you need just a few units of a given type, but hyperscalers, who deploy computers by the acre, any unused SAS connector is dead weight.