12 ms·

Supermicro server BMCs left exposed to remote attack by any USB device

51 points by GiulioS 7y ago

tedivm 7y ago
BMC's (or the equivalent for whatever vendor you are using) should never be exposed to the internet- they shouldn't even be on the same network as the rest of the server. Generally speaking I put them on a completely separate network that has to be VPN'd into explicitly. Having BMC access is as close to having physical access as you can get without actually touching the machine.
- nullc 7y ago
  > should never be exposed to the internet Good luck with that, on many devices the BMC gets silently bridged onto the lan ports even if you make a significant effort to prevent it. Even where you think you have it isolated and have tested, you may later find yourself thwarted when an unoverridable bridge gets triggered any time the dedicated BMC port's link is down.
  - nightfly 7y ago
    Having static IPs on your management devices on a separate subnet and vlan from your regular host ports should mostly mitigate this.
    - persistent 7y ago
      How? The BMC can intercept and inject any ethernet frames that it cares to inject.
      - jessaustin 7y ago
        What happens to those frames? Will a router configured for a particular VLAN pass through whatever the BMC happens to invent?
      - EB66 7y ago
        Not when you've gone into the BIOS and configured the BMC to only operate on it's dedicated port. I'm responsible for netsec at my company and I've never seen any indication of the BMC network activity on primary network ports after I've configured the BMC in BIOS to use it's dedicated network port.
  - EB66 7y ago
    > Good luck with that, on many devices the BMC gets silently bridged onto the lan ports even if you make a significant effort to prevent it. I don't think that's true. I can't speak for all devices, but I've worked with a lot of Dell and SuperMicro servers and they all have an option in the BIOS for this. You specify if the BMC should operate on it's own dedicated network port or share the network port used for primary network uplink. From the network traffic logs I've gathered, I've never seen any indication of the BMC interfering with the primary LAN ports when the BMC is set to use the dedicated LAN port. Additionally, the BMC doesn't revert back to the shared port if the link is broken on the dedicated port -- the BIOS setting prevents that. At my company we've configured all of the BMCs to use the dedicated LAN port, those LAN ports are connected to physically separate top of cabinet switches, and those switches are linked to a fully independent core router with it's own independent uplink to the internet. The BMCs are only assigned private network IPs and can only be accessed by outside users through a VPN. This is all very standard practice for the hosting industry.
    - nullc 7y ago
      > I can't speak for all devices, but I've worked with a lot of Dell and SuperMicro servers and they all have an option in the BIOS for this. You specify if the BMC should operate on it's own dedicated network port or share the network port used for primary network uplink. They do, but on the three supermicro board types I have here, simply doing that but then leaving the BMC port disconnected results in them bridging onto the lan anyways. They obey the setting but only so long as a link is up on the BMC port.
  - z3t4 7y ago
    For example your server might have 3 network ports, one is a dedicated BMC, but if it's not populated it will bridge to one of the other two, so basically you only have 1 port that is safe to use.
- KenoFischer 7y ago
  While this is a good best practice, most current BMCs treat the host as trusted, so if BMCs assume the management network it's trusted, you've just provided a path to pwn every machine on your network if only one of them is compromised. So while that's of course a good best practice, it isn't the be-all-end-all of C security (The same point was made during the presentation on this vulnerability at OSFC this morning).
  - lwf 7y ago
    Your BMC network should be configured for client isolation, e.g. one BMC should not be able to talk to the other, only a central gateway.
- bpchaps 7y ago
  There's a major, major hosting company whose server IPMIs all had an internet IP and used a default password for an unreasonably long time. I'm honestly not sure how this company is still around.
  - akersten 7y ago
    Can you please name and shame, or at least link to a news article about this? I'm going to be a little blunt, but the pattern of "there's a well-known company that's done something bad, you probably use their products, but I can't tell you what company because [I don't want to be deposed in a libel lawsuit / I want to feel intellectually superior]" is really long in the tooth, and doesn't add value to the discussion other than to pique everyone's paranoia.
    - bpchaps 7y ago
      Eh, screw it. It was Rackspace. I worked there, and was told this by a senior member of the infrastructure staff in a one on one. It was was fixed before I got there. They still have similarly bad security flubs.
  - deleted 7y ago
    [deleted]
  - whitepoplar 7y ago
    Please name the company, to save the rest of us!
    - bpchaps 7y ago
      Named in a sibling post.
      - whitepoplar 7y ago
        Holy shit, that's not a name I'd expect to have that sort of problem!
- wahern 7y ago
  Better said than practiced. I recently bought a Supermicro MBD-M11SDV-8CT-LN4F-O (AMD EPYC 3201 SoC). There are 5 NICs--4 Intel and 1 Realtek. The Realtek is set of by itself and obviously dedicated to the BMC controller. However, reading around I learned that if the controller doesn't succeed in getting a DHCP address on the Realtek interface (unplugged, or apparently in Supermicro's case just a failed DHCP server), it will automatically switch to sharing the first Intel NIC. You can disable it in the BIOS, but I've had problems in the past where I thought I disabled IPMI on the LAN but either didn't do it correctly or it reverted itself. It doesn't help that there are a ton of ambiguously named, poorly documented options. To be on the safe side I've plugged both the Realtek and #1 Intel jack. I usually prefer to use the RS-232 serial port for management as it's more fool-proof, but unfortunately this MB doesn't have one. I could install one, but it doesn't make me any more confident that the BMC isn't (now or in the future) on the network. I understand how useful these BMCs can be, especially for Windows machines, but they're so extremely dangerous. It sucks that they're so ubiquitous, and it sucks that SuperMicro's BMC seems to be the worst of them all in terms of security and bugginess. Unfortunately, a SuperMicro motherboard and case is the only way to get a low-power AMD EPYC server into a small, 1U, short-depth form factor. There are a few more options for Intel, but SuperMicro is still your best bet for do-it-yourself server hardware. I really love my three PC Engines APUs. Simple serial connection, coreboot firmware, and other than AMD's PSP chip and perhaps the Intel NIC controller, no hidden vectors for remote access. And it has an RS-232 serial port, which because there's no GPU (or at least no video port) both the BIOS and most BSD and Linux distros will automatically use by default. It'd be awesome if PC Engines built something using EPYC or even Ryzen, stripping everything down like they did for the APU and ALIX. But their niche is low-powered, passively cooled x86 devices using older, more reliable, cheaper, well-documented SoCs. Soekris tried to go upmarket with the net6501 and ended up folding. (They still sell custom DAC hardware from their EU outfit.) There's not enough of a market to make it worthwhile to take the risk. Lots of people love the idea in theory, but what people end up doing is sticking with traditional server suppliers, or in the home, hobbyist, and small business market just buying one of the many faster small form factor boxes that have flooded the market. The problem with those alternatives is that the boards are packed with too much crap, they're too diverse, and every model and family of models short-lived.
  - 7y ago
deleted 7y ago
[deleted]
HeWhoLurksLate 7y ago
Is anyone getting flashbacks to "The Big Hack"? It feels like maybe Bloomberg knew of something but got the wrong root cause- it seems a lot more likely for someone to sneak in a slightly-reprogrammed BMC than change board layouts /etc., especially considering just how much control BMC's have. Yikes.
- Animats 7y ago
  That was my concern. All it would take is a built-in IPMI password. For extra credit, a built-in IPMI password that wasn't listed if you list IPMI passwords. So I search for "IPMI password", and get this.[1] "On modern Supermicro IPMI interfaces the default login/ password is:" Login: ADMIN Password: ADMIN That's convenient. Well-documented, too. Is that enabled by default? [1] https://forums.servethehome.com/index.php?resources/supermicro-ipmi-default-login-and-password.1/ https://forums.servethehome.com/index.php?resources/supermic...
  - monocasa 7y ago
    FWIW, the implant they were describing could absolutely change a default password in the bitstream. That'd be really neat because you could swap out the flash and still be screwed.
- wanderingjew 7y ago
  Bloomberg knew of _something_, but Bloomberg isn't a news company. They're a hardware company. That info was passed on to the journos who went and did their own thing (Telephone, chinese whispers, and a healthy dose of confirmation bias). I was talking to one of the Bloomberg engineers at a bar about six months before the story dropped. There was _something_ and it was _big_, but didn't get any more information than that. The story dropped, and the story was idiotic, but there was still something there. This is probably it.
  - monocasa 7y ago
    What was idiotic about the actual story?
Deimorz 7y ago
Blogspam of https://eclypsium.com/2019/09/03/usbanywhere-bmc-vulnerability-opens-servers-to-remote-attack/ https://eclypsium.com/2019/09/03/usbanywhere-bmc-vulnerabili... (and submitter does nothing but spam links to secalerts.co)
- dmix 7y ago
  Is flagging appropriate in these situations? Or just wait until dang reroutes it?
  - Deimorz 7y ago
    I always flag as a way to try to get the mods' attention. I'm pretty sure my flags are de-weighted or even disregarded at this point though, because I flag for all sorts of reasons (which is the problem with having no way of being more specific about why I'm doing it).
wilhil 7y ago
BMCs are scary as hell... even for people who say they isolate them, you also need to do a full audit as many come with rubbish default settings. For example, Dell's default config on BMC/Idrac (at least 4-5 years ago when I tested) do not have brute force prevention and by default utilising a special CLI program, you can logon to a DRAC from the host OS. Therefore, if a host got compromised, even if Idrac is on a different network, you could in theory bruteforce from the host credentials and jump/attack the management network. FYI, for Dell, the command to disable this behaviour was racadm config -g cfgRacTune -o cfgRacTuneLocalConfigDisable 1 and it took quite a while to figure this out...
- EB66 7y ago
  > by default utilising a special CLI program, you can logon to a DRAC from the host OS ... Therefore, if a host got compromised, even if Idrac is on a different network, you could in theory bruteforce from the host credentials and jump/attack the management network. I'm not sure about a DRAC, but for standard BMC units on Dell/SuperMicro servers I am not aware of anyway that you can actually log into the BMC from the host OS. You can certainly use tools like ipmicfg to reset passwords or configuration on the BMC unit from the host OS, but I don't know of any way that you could actually drop into the command prompt for the BMC and launch an attack on the management network. As long as you have the BMC on a private IP in an entirely independent network (accessible only via VPN), then a utility like ipmicfg wouldn't help you break into it.
altmind 7y ago
If these security reports are valid, this may affect not only Supermicro, but other vendors too. From the top of my head, Asus and Gigabyte workstation motherboards also carried AST2400 based BMCs, and theirs ipmi web interface looked very similar(if not totally the same) to supermicro's.

51 points by GiulioS 7y ago

BMC's (or the equivalent for whatever vendor you are using) should never be exposed to the internet- they shouldn't even be on the same network as the rest of the server. Generally speaking I put them on a completely separate network that has to be VPN'd into explicitly. Having BMC access is as close to having physical access as you can get without actually touching the machine.

nullc 7y ago
> should never be exposed to the internet Good luck with that, on many devices the BMC gets silently bridged onto the lan ports even if you make a significant effort to prevent it. Even where you think you have it isolated and have tested, you may later find yourself thwarted when an unoverridable bridge gets triggered any time the dedicated BMC port's link is down.
- nightfly 7y ago
  Having static IPs on your management devices on a separate subnet and vlan from your regular host ports should mostly mitigate this.
  - persistent 7y ago
    How? The BMC can intercept and inject any ethernet frames that it cares to inject.
    - jessaustin 7y ago
      What happens to those frames? Will a router configured for a particular VLAN pass through whatever the BMC happens to invent?
    - EB66 7y ago
      Not when you've gone into the BIOS and configured the BMC to only operate on it's dedicated port. I'm responsible for netsec at my company and I've never seen any indication of the BMC network activity on primary network ports after I've configured the BMC in BIOS to use it's dedicated network port.
- EB66 7y ago
  > Good luck with that, on many devices the BMC gets silently bridged onto the lan ports even if you make a significant effort to prevent it. I don't think that's true. I can't speak for all devices, but I've worked with a lot of Dell and SuperMicro servers and they all have an option in the BIOS for this. You specify if the BMC should operate on it's own dedicated network port or share the network port used for primary network uplink. From the network traffic logs I've gathered, I've never seen any indication of the BMC interfering with the primary LAN ports when the BMC is set to use the dedicated LAN port. Additionally, the BMC doesn't revert back to the shared port if the link is broken on the dedicated port -- the BIOS setting prevents that. At my company we've configured all of the BMCs to use the dedicated LAN port, those LAN ports are connected to physically separate top of cabinet switches, and those switches are linked to a fully independent core router with it's own independent uplink to the internet. The BMCs are only assigned private network IPs and can only be accessed by outside users through a VPN. This is all very standard practice for the hosting industry.
  - nullc 7y ago
    > I can't speak for all devices, but I've worked with a lot of Dell and SuperMicro servers and they all have an option in the BIOS for this. You specify if the BMC should operate on it's own dedicated network port or share the network port used for primary network uplink. They do, but on the three supermicro board types I have here, simply doing that but then leaving the BMC port disconnected results in them bridging onto the lan anyways. They obey the setting but only so long as a link is up on the BMC port.
- z3t4 7y ago
  For example your server might have 3 network ports, one is a dedicated BMC, but if it's not populated it will bridge to one of the other two, so basically you only have 1 port that is safe to use.
KenoFischer 7y ago
While this is a good best practice, most current BMCs treat the host as trusted, so if BMCs assume the management network it's trusted, you've just provided a path to pwn every machine on your network if only one of them is compromised. So while that's of course a good best practice, it isn't the be-all-end-all of C security (The same point was made during the presentation on this vulnerability at OSFC this morning).
- lwf 7y ago
  Your BMC network should be configured for client isolation, e.g. one BMC should not be able to talk to the other, only a central gateway.
bpchaps 7y ago
There's a major, major hosting company whose server IPMIs all had an internet IP and used a default password for an unreasonably long time. I'm honestly not sure how this company is still around.
- akersten 7y ago
  Can you please name and shame, or at least link to a news article about this? I'm going to be a little blunt, but the pattern of "there's a well-known company that's done something bad, you probably use their products, but I can't tell you what company because [I don't want to be deposed in a libel lawsuit / I want to feel intellectually superior]" is really long in the tooth, and doesn't add value to the discussion other than to pique everyone's paranoia.
  - bpchaps 7y ago
    Eh, screw it. It was Rackspace. I worked there, and was told this by a senior member of the infrastructure staff in a one on one. It was was fixed before I got there. They still have similarly bad security flubs.
- deleted 7y ago
  [deleted]
- whitepoplar 7y ago
  Please name the company, to save the rest of us!
  - bpchaps 7y ago
    Named in a sibling post.
    - whitepoplar 7y ago
      Holy shit, that's not a name I'd expect to have that sort of problem!
wahern 7y ago
Better said than practiced. I recently bought a Supermicro MBD-M11SDV-8CT-LN4F-O (AMD EPYC 3201 SoC). There are 5 NICs--4 Intel and 1 Realtek. The Realtek is set of by itself and obviously dedicated to the BMC controller. However, reading around I learned that if the controller doesn't succeed in getting a DHCP address on the Realtek interface (unplugged, or apparently in Supermicro's case just a failed DHCP server), it will automatically switch to sharing the first Intel NIC. You can disable it in the BIOS, but I've had problems in the past where I thought I disabled IPMI on the LAN but either didn't do it correctly or it reverted itself. It doesn't help that there are a ton of ambiguously named, poorly documented options. To be on the safe side I've plugged both the Realtek and #1 Intel jack. I usually prefer to use the RS-232 serial port for management as it's more fool-proof, but unfortunately this MB doesn't have one. I could install one, but it doesn't make me any more confident that the BMC isn't (now or in the future) on the network. I understand how useful these BMCs can be, especially for Windows machines, but they're so extremely dangerous. It sucks that they're so ubiquitous, and it sucks that SuperMicro's BMC seems to be the worst of them all in terms of security and bugginess. Unfortunately, a SuperMicro motherboard and case is the only way to get a low-power AMD EPYC server into a small, 1U, short-depth form factor. There are a few more options for Intel, but SuperMicro is still your best bet for do-it-yourself server hardware. I really love my three PC Engines APUs. Simple serial connection, coreboot firmware, and other than AMD's PSP chip and perhaps the Intel NIC controller, no hidden vectors for remote access. And it has an RS-232 serial port, which because there's no GPU (or at least no video port) both the BIOS and most BSD and Linux distros will automatically use by default. It'd be awesome if PC Engines built something using EPYC or even Ryzen, stripping everything down like they did for the APU and ALIX. But their niche is low-powered, passively cooled x86 devices using older, more reliable, cheaper, well-documented SoCs. Soekris tried to go upmarket with the net6501 and ended up folding. (They still sell custom DAC hardware from their EU outfit.) There's not enough of a market to make it worthwhile to take the risk. Lots of people love the idea in theory, but what people end up doing is sticking with traditional server suppliers, or in the home, hobbyist, and small business market just buying one of the many faster small form factor boxes that have flooded the market. The problem with those alternatives is that the boards are packed with too much crap, they're too diverse, and every model and family of models short-lived.
- 7y ago

deleted 7y ago

[deleted]

HeWhoLurksLate 7y ago

Is anyone getting flashbacks to "The Big Hack"? It feels like maybe Bloomberg knew of something but got the wrong root cause- it seems a lot more likely for someone to sneak in a slightly-reprogrammed BMC than change board layouts /etc., especially considering just how much control BMC's have. Yikes.

Animats 7y ago
That was my concern. All it would take is a built-in IPMI password. For extra credit, a built-in IPMI password that wasn't listed if you list IPMI passwords. So I search for "IPMI password", and get this.[1] "On modern Supermicro IPMI interfaces the default login/ password is:" Login: ADMIN Password: ADMIN That's convenient. Well-documented, too. Is that enabled by default? [1] https://forums.servethehome.com/index.php?resources/supermicro-ipmi-default-login-and-password.1/ https://forums.servethehome.com/index.php?resources/supermic...
- monocasa 7y ago
  FWIW, the implant they were describing could absolutely change a default password in the bitstream. That'd be really neat because you could swap out the flash and still be screwed.
wanderingjew 7y ago
Bloomberg knew of _something_, but Bloomberg isn't a news company. They're a hardware company. That info was passed on to the journos who went and did their own thing (Telephone, chinese whispers, and a healthy dose of confirmation bias). I was talking to one of the Bloomberg engineers at a bar about six months before the story dropped. There was _something_ and it was _big_, but didn't get any more information than that. The story dropped, and the story was idiotic, but there was still something there. This is probably it.
- monocasa 7y ago
  What was idiotic about the actual story?

Deimorz 7y ago

Blogspam of https://eclypsium.com/2019/09/03/usbanywhere-bmc-vulnerability-opens-servers-to-remote-attack/ https://eclypsium.com/2019/09/03/usbanywhere-bmc-vulnerabili... (and submitter does nothing but spam links to secalerts.co)

dmix 7y ago
Is flagging appropriate in these situations? Or just wait until dang reroutes it?
- Deimorz 7y ago
  I always flag as a way to try to get the mods' attention. I'm pretty sure my flags are de-weighted or even disregarded at this point though, because I flag for all sorts of reasons (which is the problem with having no way of being more specific about why I'm doing it).

wilhil 7y ago

BMCs are scary as hell... even for people who say they isolate them, you also need to do a full audit as many come with rubbish default settings. For example, Dell's default config on BMC/Idrac (at least 4-5 years ago when I tested) do not have brute force prevention and by default utilising a special CLI program, you can logon to a DRAC from the host OS. Therefore, if a host got compromised, even if Idrac is on a different network, you could in theory bruteforce from the host credentials and jump/attack the management network. FYI, for Dell, the command to disable this behaviour was racadm config -g cfgRacTune -o cfgRacTuneLocalConfigDisable 1 and it took quite a while to figure this out...

EB66 7y ago
> by default utilising a special CLI program, you can logon to a DRAC from the host OS ... Therefore, if a host got compromised, even if Idrac is on a different network, you could in theory bruteforce from the host credentials and jump/attack the management network. I'm not sure about a DRAC, but for standard BMC units on Dell/SuperMicro servers I am not aware of anyway that you can actually log into the BMC from the host OS. You can certainly use tools like ipmicfg to reset passwords or configuration on the BMC unit from the host OS, but I don't know of any way that you could actually drop into the command prompt for the BMC and launch an attack on the management network. As long as you have the BMC on a private IP in an entirely independent network (accessible only via VPN), then a utility like ipmicfg wouldn't help you break into it.

altmind 7y ago

If these security reports are valid, this may affect not only Supermicro, but other vendors too. From the top of my head, Asus and Gigabyte workstation motherboards also carried AST2400 based BMCs, and theirs ipmi web interface looked very similar(if not totally the same) to supermicro's.