5 ms·

BMC's in general leave me uneasy. I like the idea, it is a small computer that is used to monitor and control your big computer. But hate the implementation. W

by somat 3y ago

BMC's in general leave me uneasy. I like the idea, it is a small computer that is used to monitor and control your big computer. But hate the implementation. Why are they all super secret special firmware blobs? Why can't I just install my linux of choice and run the manufacturers software? This would still suck but not as bad as the full stack nonsense they foist on you at this point.

donalhunt 3y ago
Was responsible for trying to improve the management and operation of a large fleet of BMCs for a while. Plenty of bugs and pace of releases is slow. :( Definitely an area where a more open ecosystem would improve the pace of innovation.
- hinkley 3y ago
  We need a Linux for BMCs. Oxide is working on one, but I'd like to see a contender fielded from the seL4 community, along with some other folks. For example, why doesn't Wind River have one already?
  - NexRebular 3y ago
    Why linux? Why not *BSD?
    - gaius_baltar 3y ago
      > Why linux? Why not *BSD? GPL can force manufacturers to cooperate with users. Of course, they can still use closed source binary modules and userland programs ...
      - NexRebular 3y ago
        GPL can also turn manufacturers away. I would rather have variation in the possible BMC operating systems instead of sticking linux everywhere and contributing to a monoculture.
        ilyt 3y ago
        ...which you wouldn't get with BSD as they'd just close it down and ship you binary, as they would have no obligation or reason to. The BSD "freedom" is not for the user of the software, it's for corporation to take.
        NexRebular 3y ago
        And how would this affect the original open BSD-based BMC firmware which would be available? If the corporations want to maintain their own fork, it's on them. The original BSD release will stay open and free for everyone to utilize.
        IntelMiner 3y ago
        How can you trust that a closed source OpenBSD fork is secure when there's no way to audit the quality (or lack thereof) in the firmware the vendor gives you? If it's GPL you can at least interrogate the code release and make an informed decision
        NexRebular 3y ago
        The same way I can trust all the binary blobs vendors stick into GPL-licensed systems. How do I make an informed decision about those?
        wmf 3y ago
        Software diversity is a luxury customers won't pay for.
      - taneq 3y ago
        That sure sounds appealing to manufacturers.
        wmf 3y ago
        Once it reaches critical mass it is appealing because the value you receive from the community is far greater than what you have to give back.
      - zrm 3y ago
        If the point is code diversity, what stops you producing a fork of BSD for BMCs under the GPL?
  - wmf 3y ago
    OpenBMC is the leader at this point.
- chasil 3y ago
  Sometimes the bugs end up at a security conference. https://airbus-seclab.github.io/ilo/BHUSA2021-Slides-hpe_ilo_5_security_go_home_cryptoprocessor_youre_drunk-gazet_perigaud_czarny.pdf https://airbus-seclab.github.io/ilo/BHUSA2021-Slides-hpe_ilo...
theideaofcoffee 3y ago
They’re special firmware blobs because generally the OEMs aren’t building their BMCs from scratch as they might with their main boards and other components. They’re generally getting the bmc SoC from the likes of Aspeed and others who are the ones keeping them closed up. I’ve tried to get the magic binaries and source for various projects but have given up because there are so many layers of NDAs and sales gatekeepers. I’m not entirely sure who makes the dell bmcs but I know supermicro bundles Aspeed (at least they did with older generations of their main boards.) I agree with you that you should be able to run whatever since in the end it’s just another computer, but the manufacturers believe otherwise since there’s “valuable IP” or whatever nonsense (insert rollseyes emoji here). There are open specs like redfish but still doesn’t get to the heart of the matter.
- jebr224 3y ago
  It is complicated. AMI sells a bmc software stack, https://www.ami.com/megarac/ https://www.ami.com/megarac/ Intel and small manufacturer were unhappy, about always paying the ami tax. So intel created openbmc, as a hedge against ami's monopoly for small manufacturers. I have heard Openbmc has user from facebook, google, ibm, bytdance, and ali. Dell owns their own stack in idrac, I have heard most of their systems are nuvoton based. I am suspect dell pays some big bucks to keep their systems at feature parity with the other options, and they view it as a an investment. There are also silicon devices on the motherboard, that have drivers that are not able to be shared. So it not surprising that companies don't share source in a way that would be useful. If you wanted a system that as a bmc that could be tested try the asrock-e3c246d4c, it looks like there are hobbyist, that have it running coreboot, and openbmc. (impressively) https://9esec.io/blog/coreboot-on-the-asrock-e3c246d4c/ https://9esec.io/blog/coreboot-on-the-asrock-e3c246d4c/
- mkj 3y ago
  Aspeed aren't the ones keeping them closed - OpenBMC running on Aspeed chips is an option chosen by some system vendors. It's a vendor choice whether to go with an open platform or a closed BMC from AMI etc. https://github.com/AspeedTech-BMC/openbmc https://github.com/AspeedTech-BMC/openbmc (or see the upstream OpenBMC tree) HPE are adding some OpenBMC support https://www.hpe.com/us/en/compute/openbmc-proliant-servers.html https://www.hpe.com/us/en/compute/openbmc-proliant-servers.h... This work from Supermicro looked promising https://lore.kernel.org/openbmc/CACPK8XdE0sRmt4x54YJVJO2wDT5cm2AkRfGBoyn+h9P668K5mg@mail.gmail.com/T/ https://lore.kernel.org/openbmc/CACPK8XdE0sRmt4x54YJVJO2wDT5...
WaitWaitWha 3y ago
Was responsible to poke at BMC security in data centers. BMCs lack the security fundamentals and often behave like cheap IoT knock-offs. They often use outdated kernels, libraries, and security mechanism.
- kabdib 3y ago
  Ran a fleet of servers with terrible BMCs. We kept those on a well-sealed-off private network. Woe betide you if you run into one of the BMC implementations that shares a host network interface; no separate cable! These things are terrible from a security standpoint.
  - deleted 3y ago
    [deleted]
  - septune 3y ago
    i hope you disable OS passtrough because it could be gore
- jdwithit 3y ago
  Yeah they are utter garbage. For years you had to use Java 6 with absolutely every modern security measure turned off in both the JVM runtime itself and your browser to access Dell DRACs. Accept expired certs, run unsigned code, I'm sure this is all fine ... I mostly work in the cloud now but when I last had to manage a bunch of physical machines we had a physically separate network accessed via its own VPN to get onto the BMCs. Because yeah, the security situation was a joke.
  - hsbauauvhabzb 3y ago
    I found that if you leave your bmc unplugged on a super micro, it’ll conveniently bridge it to whatever other Ethernet is plugged in, meaning an outage of your management network may roll over to another network unintentionally. Id put money on there being preauth vulnerabilities in those things, judging by the engineering quality.
Y_Y 3y ago
It may no be "enterprise" enough for a given employer, by it's not hard to replicate most of this functionality with cheap (and open) hardware. For example I had a case that called for BMC that I resolved with a spare raspi3b, a very cheap capture device and a "smart plug". Total cost of materials was about 30 euro and (for me) it wasn't any harder to operate that an idrac.
- Tijdreiziger 3y ago
  https://pikvm.org/ https://pikvm.org/ https://tinypilotkvm.com/ https://tinypilotkvm.com/
deleted 3y ago
[deleted]
ilyt 3y ago
It would also be sooo much easier to automate everything if it just ran slightly custom Debian install instead of... whatever the fuck abomination manufacturer made.
yjftsjthsd-h 3y ago
Actually, why aren't they literally normal little computers? Like, fully open, bring your own OS computers. All it needs is some peripherals - Ethernet, its own host USB, gadget mode USB to present mouse/kb/storage to the main computer, video capture card, some GPIO to control power - but there's nothing all that special there; then you just install Debian or w/e and control the perfectly standard interfaces at will.
- wmf 3y ago
  The interface between the BMC and motherboard is unique to each motherboard, especially for "added value" features that some servers have. DC-SCM is working on standardizing this but I don't know how interoperable it will be.
- croutonwagon 3y ago
  Personally I have some pikvms at home. And while not full parity to idrac enterprise, and it takes some extra effort to get atx, I have STRONGLY been considering dropping them in our DC to at least replace the dell ipkvm that frankly is a security nightmare.
scifi 3y ago
To add to this, manufacturers (HPE) are now requiring an expensive annual fee in order for customers to use the hardware.
pabs3 3y ago
I hear that many BMCs internally just run Linux.

by somat 3y ago

donalhunt 3y ago

Was responsible for trying to improve the management and operation of a large fleet of BMCs for a while. Plenty of bugs and pace of releases is slow. :( Definitely an area where a more open ecosystem would improve the pace of innovation.

hinkley 3y ago
We need a Linux for BMCs. Oxide is working on one, but I'd like to see a contender fielded from the seL4 community, along with some other folks. For example, why doesn't Wind River have one already?
- NexRebular 3y ago
  Why linux? Why not *BSD?
  - gaius_baltar 3y ago
    > Why linux? Why not *BSD? GPL can force manufacturers to cooperate with users. Of course, they can still use closed source binary modules and userland programs ...
    - NexRebular 3y ago
      GPL can also turn manufacturers away. I would rather have variation in the possible BMC operating systems instead of sticking linux everywhere and contributing to a monoculture.
      - ilyt 3y ago
        ...which you wouldn't get with BSD as they'd just close it down and ship you binary, as they would have no obligation or reason to. The BSD "freedom" is not for the user of the software, it's for corporation to take.
        NexRebular 3y ago
        And how would this affect the original open BSD-based BMC firmware which would be available? If the corporations want to maintain their own fork, it's on them. The original BSD release will stay open and free for everyone to utilize.
        IntelMiner 3y ago
        How can you trust that a closed source OpenBSD fork is secure when there's no way to audit the quality (or lack thereof) in the firmware the vendor gives you? If it's GPL you can at least interrogate the code release and make an informed decision
        NexRebular 3y ago
        The same way I can trust all the binary blobs vendors stick into GPL-licensed systems. How do I make an informed decision about those?
      - wmf 3y ago
        Software diversity is a luxury customers won't pay for.
    - taneq 3y ago
      That sure sounds appealing to manufacturers.
      - wmf 3y ago
        Once it reaches critical mass it is appealing because the value you receive from the community is far greater than what you have to give back.
    - zrm 3y ago
      If the point is code diversity, what stops you producing a fork of BSD for BMCs under the GPL?
- wmf 3y ago
  OpenBMC is the leader at this point.
chasil 3y ago
Sometimes the bugs end up at a security conference. https://airbus-seclab.github.io/ilo/BHUSA2021-Slides-hpe_ilo_5_security_go_home_cryptoprocessor_youre_drunk-gazet_perigaud_czarny.pdf https://airbus-seclab.github.io/ilo/BHUSA2021-Slides-hpe_ilo...

theideaofcoffee 3y ago

They’re special firmware blobs because generally the OEMs aren’t building their BMCs from scratch as they might with their main boards and other components. They’re generally getting the bmc SoC from the likes of Aspeed and others who are the ones keeping them closed up. I’ve tried to get the magic binaries and source for various projects but have given up because there are so many layers of NDAs and sales gatekeepers. I’m not entirely sure who makes the dell bmcs but I know supermicro bundles Aspeed (at least they did with older generations of their main boards.) I agree with you that you should be able to run whatever since in the end it’s just another computer, but the manufacturers believe otherwise since there’s “valuable IP” or whatever nonsense (insert rollseyes emoji here). There are open specs like redfish but still doesn’t get to the heart of the matter.

jebr224 3y ago
It is complicated. AMI sells a bmc software stack, https://www.ami.com/megarac/ https://www.ami.com/megarac/ Intel and small manufacturer were unhappy, about always paying the ami tax. So intel created openbmc, as a hedge against ami's monopoly for small manufacturers. I have heard Openbmc has user from facebook, google, ibm, bytdance, and ali. Dell owns their own stack in idrac, I have heard most of their systems are nuvoton based. I am suspect dell pays some big bucks to keep their systems at feature parity with the other options, and they view it as a an investment. There are also silicon devices on the motherboard, that have drivers that are not able to be shared. So it not surprising that companies don't share source in a way that would be useful. If you wanted a system that as a bmc that could be tested try the asrock-e3c246d4c, it looks like there are hobbyist, that have it running coreboot, and openbmc. (impressively) https://9esec.io/blog/coreboot-on-the-asrock-e3c246d4c/ https://9esec.io/blog/coreboot-on-the-asrock-e3c246d4c/
mkj 3y ago
Aspeed aren't the ones keeping them closed - OpenBMC running on Aspeed chips is an option chosen by some system vendors. It's a vendor choice whether to go with an open platform or a closed BMC from AMI etc. https://github.com/AspeedTech-BMC/openbmc https://github.com/AspeedTech-BMC/openbmc (or see the upstream OpenBMC tree) HPE are adding some OpenBMC support https://www.hpe.com/us/en/compute/openbmc-proliant-servers.html https://www.hpe.com/us/en/compute/openbmc-proliant-servers.h... This work from Supermicro looked promising https://lore.kernel.org/openbmc/CACPK8XdE0sRmt4x54YJVJO2wDT5cm2AkRfGBoyn+h9P668K5mg@mail.gmail.com/T/ https://lore.kernel.org/openbmc/CACPK8XdE0sRmt4x54YJVJO2wDT5...

WaitWaitWha 3y ago

Was responsible to poke at BMC security in data centers. BMCs lack the security fundamentals and often behave like cheap IoT knock-offs. They often use outdated kernels, libraries, and security mechanism.

kabdib 3y ago
Ran a fleet of servers with terrible BMCs. We kept those on a well-sealed-off private network. Woe betide you if you run into one of the BMC implementations that shares a host network interface; no separate cable! These things are terrible from a security standpoint.
- deleted 3y ago
  [deleted]
- septune 3y ago
  i hope you disable OS passtrough because it could be gore
jdwithit 3y ago
Yeah they are utter garbage. For years you had to use Java 6 with absolutely every modern security measure turned off in both the JVM runtime itself and your browser to access Dell DRACs. Accept expired certs, run unsigned code, I'm sure this is all fine ... I mostly work in the cloud now but when I last had to manage a bunch of physical machines we had a physically separate network accessed via its own VPN to get onto the BMCs. Because yeah, the security situation was a joke.
- hsbauauvhabzb 3y ago
  I found that if you leave your bmc unplugged on a super micro, it’ll conveniently bridge it to whatever other Ethernet is plugged in, meaning an outage of your management network may roll over to another network unintentionally. Id put money on there being preauth vulnerabilities in those things, judging by the engineering quality.

Y_Y 3y ago

It may no be "enterprise" enough for a given employer, by it's not hard to replicate most of this functionality with cheap (and open) hardware. For example I had a case that called for BMC that I resolved with a spare raspi3b, a very cheap capture device and a "smart plug". Total cost of materials was about 30 euro and (for me) it wasn't any harder to operate that an idrac.

Tijdreiziger 3y ago
https://pikvm.org/ https://pikvm.org/ https://tinypilotkvm.com/ https://tinypilotkvm.com/

deleted 3y ago

[deleted]

ilyt 3y ago

It would also be sooo much easier to automate everything if it just ran slightly custom Debian install instead of... whatever the fuck abomination manufacturer made.

yjftsjthsd-h 3y ago

Actually, why aren't they literally normal little computers? Like, fully open, bring your own OS computers. All it needs is some peripherals - Ethernet, its own host USB, gadget mode USB to present mouse/kb/storage to the main computer, video capture card, some GPIO to control power - but there's nothing all that special there; then you just install Debian or w/e and control the perfectly standard interfaces at will.

wmf 3y ago
The interface between the BMC and motherboard is unique to each motherboard, especially for "added value" features that some servers have. DC-SCM is working on standardizing this but I don't know how interoperable it will be.
croutonwagon 3y ago
Personally I have some pikvms at home. And while not full parity to idrac enterprise, and it takes some extra effort to get atx, I have STRONGLY been considering dropping them in our DC to at least replace the dell ipkvm that frankly is a security nightmare.

scifi 3y ago

To add to this, manufacturers (HPE) are now requiring an expensive annual fee in order for customers to use the hardware.

pabs3 3y ago

I hear that many BMCs internally just run Linux.