6 ms·

In Which I Savagely Impugn the Honor of IPMI and its Friends

63 points by dfc 14y ago

modeless 14y ago
Are these IPMI things ever installed on desktops and laptops, or are they strictly for server hardware?
- vault_ 14y ago
  You may be able to find it on high-end workstations sold by server companies (read, a 4U server turned on its side, with added feet), but by and large it's a server-only thing. The reason being it's only really necessary to have the level of remote access offered by these systems if it's difficult to access the hardware, or you have a very large amount of the hardware (in most cases where it's used, both of these are true).
  - SEJeff 14y ago
    You can also buy discrete bmc pci cards from a few different vendors. I believe super micro sells them, but I've always loathed any environment that uses super micro servers as they tend to be low to mediocre quality compared to the "big 3" tier 1 server vendors.
- lmz 14y ago
  On desktops and laptops Intel has AMT, which is somewhat similar.
thrownaway2424 14y ago
Pfft, this guy only mentioned the obvious horrible security implications. Aside from that glaring flaw, there is also the small matter that IPMI software is all implemented by raging idiots who got fired from BIOS vendors, and then got fired again from the people who write mobile radio firmware, and then got fired again from the company that writes the software in your electric toothbrush. They never work in actual outages, cause more host outages than they solve, and are in no way a substitute for having humans on site. I had a rack full of garbage from Penguin Computing that would hang on reboot with the message that the BMC was not responding. Thanks, that's helpful. And dozens of horror stories from every vendor ranging from SuperMicro to HP and IBM. I encourage you to guess whether Facebook, Google, and Amazon use IMPI to manage anything. Have a look at the OpenCompute designs for a hint.
- subway 14y ago
  http://opencompute.org/wp/wp-content/uploads/2012/05/OpenComputeProjectOpenHardwareMachinesManagement.pdf http://opencompute.org/wp/wp-content/uploads/2012/05/OpenCom... These designs?
  - thrownaway2424 14y ago
    Doh. Just goes to show, you can open up your process, but you can't make people have judgement and taste.
    - dfc 14y ago
      And/or that you should look before you speak.
  - wmf 14y ago
    The Open Compute 1.0 servers had reboot-on-LAN instead of BMCs, but then Facebook went back. I wonder why.
- dfc 14y ago
  "this guy"? This guy is Dan Farmer[1], I realize that does not make him infallible but he is definitely not someone looking for his 15 minutes of fame in order to make a name for himself. [1] http://en.wikipedia.org/wiki/Dan_Farmer http://en.wikipedia.org/wiki/Dan_Farmer
  - redthrowaway 14y ago
    I read the comment not as a critique of the author, but rather as a further rant against the target of the author's ire.
    - dfc 14y ago
      After rereading it I think that's probably a fair interpretation. I guess I was thrown off from the fact that the author was the subject of the first sentence and therefore made it seem personal.
Adaptive 14y ago
Anecdotal, but the IPMI software I have to use regularly for access to a specific server is, without doubt, the worst server utility I've ever used. I love having the BMC there, but you are really at the mercy of vendors to update their IPMI/BMC firmware. Guess how often that happens.
patrickgzill 14y ago
IPMI is so bad, I don't even use it for my servers when it is available! Better to just wait and call someone at the datacenter who will take care of reboots properly.
contingencies 14y ago
Late last year I was building infrastructure with some higher security requirements and was quite shocked at the IPMI situation - concerned enough to email Redhat (who obviously support lots and lots of clients on hardware from lots and lots of vendors) via a personal contact in a global security team there, with my concerns and some evidence related to our particular hardware vendor. After some time to ponder, they got back to me with "your concerns look legitimate". Discussions followed, but I am not aware of any specific advisories resulting from this. I also understand that recently there was an IPMI-related presentation at Ruxcon Breakpoint in Melbourne, Australia: http://www.ruxconbreakpoint.com/speakers/#Igor%20Skochinsky http://www.ruxconbreakpoint.com/speakers/#Igor%20Skochinsky ... no idea of the content, though. Honestly, I think the first thing is for vendors to be a lot more honest: if someone has root on your host, they can most likely trivially obtain root on and backdoor the IPMI controller. Once they get root on the IPMI controller, they can do anything they want on the IPMI LAN. This reality differs greatly from the 'dedicated' and 'separate' words used in vendor marketing literature and documentation, and is no doubt a direct contributor to insufficiently paranoid systems architecture, resulting in real world vulnerabilities in some pretty important systems. Why do I say pretty important? Well, outside of remote KVM, IPMI is frequently used for node fencing in high availability (HA) cluster scenarios, which means that this technology is likely to co-occur with some fairly paranoid / 24x7x365 systems (air traffic control, stock markets, etc.). Attackers with a foot-hold can thus use IPMI-based vulnerabilities to (permanently) compromise such higher-end systems. What are we as implementers supposed to do? Here are some ideas. (1) Run a relatively paranoid, receive-only, anomaly-based NIDS on your IPMI segment to encourage 'after the fact' detection of bad behaviour. (2) Run IPMI on a dedicated VLAN. (3) Run your IPMI VLAN on dedicated hardware (4) Program your network infrastructure to ensure that regular IPMI nodes on your average server cannot communicate with any other peers on the link layer (ie. other servers) except for the expected source of management operations (5) Make sure the expected source of management operations isn't running an IPMI controller (or at least, one from the same vendor on the same subnet) (6) Don't use IPMI at all Honestly, IPMI can be very useful in some cases, though it's probably a massively unspoken vulnerability in some high end systems. As always, don't trust any single component... security only comes in partly-effective layers ...

63 points by dfc 14y ago

modeless 14y ago

Are these IPMI things ever installed on desktops and laptops, or are they strictly for server hardware?

vault_ 14y ago
You may be able to find it on high-end workstations sold by server companies (read, a 4U server turned on its side, with added feet), but by and large it's a server-only thing. The reason being it's only really necessary to have the level of remote access offered by these systems if it's difficult to access the hardware, or you have a very large amount of the hardware (in most cases where it's used, both of these are true).
- SEJeff 14y ago
  You can also buy discrete bmc pci cards from a few different vendors. I believe super micro sells them, but I've always loathed any environment that uses super micro servers as they tend to be low to mediocre quality compared to the "big 3" tier 1 server vendors.
lmz 14y ago
On desktops and laptops Intel has AMT, which is somewhat similar.

thrownaway2424 14y ago

Pfft, this guy only mentioned the obvious horrible security implications. Aside from that glaring flaw, there is also the small matter that IPMI software is all implemented by raging idiots who got fired from BIOS vendors, and then got fired again from the people who write mobile radio firmware, and then got fired again from the company that writes the software in your electric toothbrush. They never work in actual outages, cause more host outages than they solve, and are in no way a substitute for having humans on site. I had a rack full of garbage from Penguin Computing that would hang on reboot with the message that the BMC was not responding. Thanks, that's helpful. And dozens of horror stories from every vendor ranging from SuperMicro to HP and IBM. I encourage you to guess whether Facebook, Google, and Amazon use IMPI to manage anything. Have a look at the OpenCompute designs for a hint.

subway 14y ago
http://opencompute.org/wp/wp-content/uploads/2012/05/OpenComputeProjectOpenHardwareMachinesManagement.pdf http://opencompute.org/wp/wp-content/uploads/2012/05/OpenCom... These designs?
- thrownaway2424 14y ago
  Doh. Just goes to show, you can open up your process, but you can't make people have judgement and taste.
  - dfc 14y ago
    And/or that you should look before you speak.
- wmf 14y ago
  The Open Compute 1.0 servers had reboot-on-LAN instead of BMCs, but then Facebook went back. I wonder why.
dfc 14y ago
"this guy"? This guy is Dan Farmer[1], I realize that does not make him infallible but he is definitely not someone looking for his 15 minutes of fame in order to make a name for himself. [1] http://en.wikipedia.org/wiki/Dan_Farmer http://en.wikipedia.org/wiki/Dan_Farmer
- redthrowaway 14y ago
  I read the comment not as a critique of the author, but rather as a further rant against the target of the author's ire.
  - dfc 14y ago
    After rereading it I think that's probably a fair interpretation. I guess I was thrown off from the fact that the author was the subject of the first sentence and therefore made it seem personal.

Adaptive 14y ago

Anecdotal, but the IPMI software I have to use regularly for access to a specific server is, without doubt, the worst server utility I've ever used. I love having the BMC there, but you are really at the mercy of vendors to update their IPMI/BMC firmware. Guess how often that happens.

patrickgzill 14y ago

IPMI is so bad, I don't even use it for my servers when it is available! Better to just wait and call someone at the datacenter who will take care of reboots properly.

contingencies 14y ago

Late last year I was building infrastructure with some higher security requirements and was quite shocked at the IPMI situation - concerned enough to email Redhat (who obviously support lots and lots of clients on hardware from lots and lots of vendors) via a personal contact in a global security team there, with my concerns and some evidence related to our particular hardware vendor. After some time to ponder, they got back to me with "your concerns look legitimate". Discussions followed, but I am not aware of any specific advisories resulting from this. I also understand that recently there was an IPMI-related presentation at Ruxcon Breakpoint in Melbourne, Australia: http://www.ruxconbreakpoint.com/speakers/#Igor%20Skochinsky http://www.ruxconbreakpoint.com/speakers/#Igor%20Skochinsky ... no idea of the content, though. Honestly, I think the first thing is for vendors to be a lot more honest: if someone has root on your host, they can most likely trivially obtain root on and backdoor the IPMI controller. Once they get root on the IPMI controller, they can do anything they want on the IPMI LAN. This reality differs greatly from the 'dedicated' and 'separate' words used in vendor marketing literature and documentation, and is no doubt a direct contributor to insufficiently paranoid systems architecture, resulting in real world vulnerabilities in some pretty important systems. Why do I say pretty important? Well, outside of remote KVM, IPMI is frequently used for node fencing in high availability (HA) cluster scenarios, which means that this technology is likely to co-occur with some fairly paranoid / 24x7x365 systems (air traffic control, stock markets, etc.). Attackers with a foot-hold can thus use IPMI-based vulnerabilities to (permanently) compromise such higher-end systems. What are we as implementers supposed to do? Here are some ideas. (1) Run a relatively paranoid, receive-only, anomaly-based NIDS on your IPMI segment to encourage 'after the fact' detection of bad behaviour. (2) Run IPMI on a dedicated VLAN. (3) Run your IPMI VLAN on dedicated hardware (4) Program your network infrastructure to ensure that regular IPMI nodes on your average server cannot communicate with any other peers on the link layer (ie. other servers) except for the expected source of management operations (5) Make sure the expected source of management operations isn't running an IPMI controller (or at least, one from the same vendor on the same subnet) (6) Don't use IPMI at all Honestly, IPMI can be very useful in some cases, though it's probably a massively unspoken vulnerability in some high end systems. As always, don't trust any single component... security only comes in partly-effective layers ...