8 ms·

Over 20k servers have their iLO interfaces exposed to the internet

178 points by caaqil 5y ago

rlt 5y ago
Sometimes I wish we could just burn it all down and start over.
MikeAshley178 5y ago
I looked at this sometime ago. Here is what I found. https://github.com/tg12/rapid7_OSINT https://github.com/tg12/rapid7_OSINT
ted_dunning 5y ago
As a quick note, this is 4 year old news. The security group at HPE called everybody that they could identify at that time to help them fix this.
cma 5y ago
Some could be honeypots
- dlsa 5y ago
  Likely a lot more aren't. Having tripwires is still a good idea though.
gengelbro 5y ago
I've had the bad fortune of dealing with iLO in the past. There's absolutely nothing surprising to me that it would be remotely exploitable, as well as default remote accessible.
serverCurios 5y ago
That was a great deep dive in to iLO. I wonder how much of that would apply to Dell's iDRAC?
mirceal 5y ago
if you want to be really scared read: http://fish2.com/ipmi/itrain.pdf http://fish2.com/ipmi/itrain.pdf all servers in a datacenter have this management interface (iLO is just one type). if the management network these sit on is poorly secured (like here) your servers are literally powned.
- Terry_Roll 5y ago
  ??? www.tipsyknitterwines.com ???
GekkePrutser 5y ago
This is actually really much lower than I expected.. Though what helps is that most servers have a dedicated iLO interface and you really have to choose to configure it on the regular ones along with normal traffic. So out of band is default. So in this case it's only people who have deliberately configured this. I think this is why it's not hundreds of thousands.
- madjam002 5y ago
  I've only dabbled with iLO but even when operating on the same interface I'm not sure how this gets onto the internet? From what I remember you configure it with a static IP, so these people surely are configuring their iLO to listen on a publicly routable IP?
  - GekkePrutser 5y ago
    Yeah if your server is DMZ they could have configured it with another ip in the same range.. I guess it's something like that. Some of them could also be intentional, in order to provide management capabilities. At least iLO has no default password (each server comes with a label with the password)
- assttoasstmgr 5y ago
  Not sure I agree with you 100% on your police work there, Lou. I worked with a good number of SuperMicro servers that - to my surprise - the IPMI interface was by default set to "fail over to bridge to primary interface". That is to say if the IPMI port was disconnected, it would bridge onto your primary NIC with a second MAC address. The way to disable it was to download an obscure utility from SuperMicro's website, which only ran under Windows, and to pass an equally obscure hexadecimal command line flag. Fortunately in my case, these ran on a network with no DHCP server, but I can't assume that's true in every case.
  - throwawayboise 5y ago
    Yes SuperMicro machines are configured that way. Some don't even have a dedicated separate IPMI interface.
danpalmer 5y ago
BMCs are slightly different I think (more powerful!) but at my previous workplace we once had a server provided to us by our hosting provider that had its BMC exposed on the internet with default credentials. They never told us the machine even had one (most of our servers from them didn’t). We only figured it out once we found a Monero miner on the machine. People on here love promoting dedicated servers over cloud VMs, but it’s so much easier for this sort of thing to go wrong with dedicated hosting.
- throwawayboise 5y ago
  The latest generations of servers ship with randomized BMC passwords. This was indeed a problem in the past when they shipped with credientials such as ADMIN / ADMIN or no password at all.
  - dnautics 5y ago
    "Latest generation" = last 2 years. Three years ago I was buying SMC servers which were the first round configured with randomized passwords to comply with CA law (some batches had ADMIN/ADMIN) and gigabyte servers still were out of compliance.
    - dijit 5y ago
      I bought my last batch of Dell machines in 2017 (so, 5 years? wow), and they had randomised passwords. But yes, some providers put the BMC on the internet because it's easier, a provider I used once did this and I was quite displeased as iDRAC's are quite weak and suffer under the weight of bot-spam. -- even if there were no security issues.
      - zerd 5y ago
        HP Microserver Gen8 with iLO from 2014 also had generated admin password.
        skinkestek 5y ago
cesaref 5y ago
The older iLO2 and 3 were from the Gen7/8 HP rack mount servers, which were released around 2013 or something like that. I'd have thought that the majority of commercial uses have long passed, so I was wondering if these generation of machines are really being used for home labs, that sort of thing, as they are passed their use by date (I think the gen7 machines were EOSL'd in 2018 so they'll have been chucked out of datacentres). Actually, I can imagine a fair few are still in use as small enterprise servers, with iLO being visible for remote admin, which is a shame, as that suggests they are not behind a tunnel, so those addresses probably indicate larger problems than a visible iLO.
- jabart 5y ago
  iLO2 is G6 iLO3 is G7 iLO4 is G8 Walking around a DC, no they have not been "chucked out of datacenters"
- IYasha 5y ago
  You'd probably be surprised to see hundreds of thousands of those machines crunching data in universities, labs, private clusters...
johnklos 5y ago
I reported a security issue in Supermicro's IPMI implementation to them. They dismissed the issue and never fixed it. These companies don't care if there's no way to stop a remote control service (iLo, iDRAC, IPMI) from binding to a motherboard ethernet port. In the case of Supermicro, you can't configure via the built-in BIOS, there are no jumpers you can use, and you must have a full network setup and installed tools to configure a new machine. This makes deploying in the field much, much more problematic. If the BIOS battery dies and settings set are lost, the motherboard defaults to joining whichever ethernet is active, with default credentials. It's incredibly stupid and insecure. I will never buy Supermicro again, but for the hardware that was already bought, I mandated loopback plugs for all IPMI ports so the IPMI wouldn't switch to the system ethernet.
- zamadatix 5y ago
  Supermicro switched to unique default passwords (same as Dell/HPE) a few years back. Silly to not have day one but really when you look at any of these they are all a security nightmare.
  - op00to 5y ago
    Dell’s not root/Calvin anymore?
    - zamadatix 5y ago
      For most probably as you have to know to order it specifically with secure password unlike HPE/Supermicro which just do it every time now. Likely they got hit with a government/large customer requirement and half assed it, as most things BMC are.
    - donflamenco 5y ago
      It hasn't been for years now. The systems get a unique password from the factory and is on the tag.
- stingraycharles 5y ago
  Honest question: do people actually connect their ipmi / bmc to the public internet? Or is it something different that’s being exposed here?
ggm 5y ago
I love how iDrac shipped with a password everyone knows. I would have thought it was trivial to make it a 1-time passthrough to change it, to stop this being the cartoon character we know and love.
- LilBytes 5y ago
  Calvin Calvin calvinnnn. You never forget it.
davidzweig 5y ago
Is there a consensus on whether it's ok to port forward from the firewall to port 22 (SSH) on the BMC, with a strong admin password? I guess this should be safer than exposing the web interface.
- mkj 5y ago
  Pre-auth ssh vulnerabilities are pretty rare, seems likely to be ok unless the BMC software has surprise extra accounts.
- whalesalad 5y ago
  You never want these interfaces on public internet. All my iDRAC ports are on an isolated management vlan that is only routed to from certain internal networks. 1. You don't want the device even considering requests from anyone but the sole person(s) responsible for accessing the mgmt interface. Someone might not be able to get in, but they could enumerate your hardware characteristics, or perform a denial of service. 2. Don't confuse SSH with secure. You don't know what version/brand/make/model of SSH is running on the embedded device. It might be ancient. It might not even be OpenSSH. 3. If an exploit becomes known for your management interfaces on your machines, you are screwed. First because someone might have already exploited you. Second because now you need to patch your boxes, and that might require a hard reboot of the entire machine. If a patch even exists. Really, it isn't worth the trouble. There are too many risks.
twic 5y ago
The amount of access that BMCs have to the system seems nuts to me. How about a BMC-esque thing whose only function is to provide remote access to the host machine's essential interfaces? On one side, an ethernet port, that you plug into your management plane. On the other side, a serial port, to access the host's console, a device-side SATA port, so it can present a virtual boot disk to the host, and a wire to the power supply, to turn the host on and off. Maybe also some read-only interface to motherboard-level monitoring like SMART and so on; maybe you get that by cooperation with the host's firmware, over the serial or SATA interface. That would let you bring a machine up and configure it, reboot it remotely, force it to boot from a recovery disk, etc. But a hacked BMC couldn't be used to subtly interfere with the host. Also, does anyone know what Oxide are doing about BMC?
- immibis 5y ago
  That sounds like a lot of access.
bogwog 5y ago
Doesn’t HP require a support contract for firmware updates? That would probably explain why so many of those are on old firmwares.
myself248 5y ago
Seems to me that we'd get some pretty good infosec bang-for-buck to hire some new FBI agents to just wear badges and knock on doors of businesses with such abjectly shittastic security. Either their IT people have been begging to fix it and management won't give them the resources, or they don't have IT people who even see it as a problem. Either way, a couple special agents having a sit-down with the CEO might change some course real fast. Discuss.
- stevenicr 5y ago
  USDS ? The US Digital Service - maybe? Is that still a thing? I don't think the fbi would be any good at this scenario described.
  - donhidalgo 5y ago
    It is still a thing, and agree. usds.gov
- Handytinge 5y ago
  What makes you think these servers are in the USA?
znpy 5y ago
First things I did on my own server at home: 1. Disable public ipv6 on the iLO interface 2. change all the passwords 3. Set-up tls on the web interface and force http to https redirection. Ironically, point 3 alone is really most of the protection: the web interface is so slow in https it's basically unusable.
infologics 5y ago
[dead]
IYasha 5y ago
Sometimes people don't have choice - it's colocation, dc doesn't offer management vpn/firewalling, you can't install separate vpn router unless you pay for another unit in a rack. And iLO/LOM/WTF at least offers https... what can you do? Well, some people call dc to have their LOs (un-)plugged for maintenance time.

178 points by caaqil 5y ago

rlt 5y ago

Sometimes I wish we could just burn it all down and start over.

I looked at this sometime ago. Here is what I found. https://github.com/tg12/rapid7_OSINT https://github.com/tg12/rapid7_OSINT

ted_dunning 5y ago

As a quick note, this is 4 year old news. The security group at HPE called everybody that they could identify at that time to help them fix this.

cma 5y ago

Some could be honeypots

dlsa 5y ago
Likely a lot more aren't. Having tripwires is still a good idea though.

gengelbro 5y ago

I've had the bad fortune of dealing with iLO in the past. There's absolutely nothing surprising to me that it would be remotely exploitable, as well as default remote accessible.

serverCurios 5y ago

That was a great deep dive in to iLO. I wonder how much of that would apply to Dell's iDRAC?

mirceal 5y ago

if you want to be really scared read: http://fish2.com/ipmi/itrain.pdf http://fish2.com/ipmi/itrain.pdf all servers in a datacenter have this management interface (iLO is just one type). if the management network these sit on is poorly secured (like here) your servers are literally powned.

Terry_Roll 5y ago
??? www.tipsyknitterwines.com ???

GekkePrutser 5y ago

This is actually really much lower than I expected.. Though what helps is that most servers have a dedicated iLO interface and you really have to choose to configure it on the regular ones along with normal traffic. So out of band is default. So in this case it's only people who have deliberately configured this. I think this is why it's not hundreds of thousands.

madjam002 5y ago
I've only dabbled with iLO but even when operating on the same interface I'm not sure how this gets onto the internet? From what I remember you configure it with a static IP, so these people surely are configuring their iLO to listen on a publicly routable IP?
- GekkePrutser 5y ago
  Yeah if your server is DMZ they could have configured it with another ip in the same range.. I guess it's something like that. Some of them could also be intentional, in order to provide management capabilities. At least iLO has no default password (each server comes with a label with the password)
assttoasstmgr 5y ago
Not sure I agree with you 100% on your police work there, Lou. I worked with a good number of SuperMicro servers that - to my surprise - the IPMI interface was by default set to "fail over to bridge to primary interface". That is to say if the IPMI port was disconnected, it would bridge onto your primary NIC with a second MAC address. The way to disable it was to download an obscure utility from SuperMicro's website, which only ran under Windows, and to pass an equally obscure hexadecimal command line flag. Fortunately in my case, these ran on a network with no DHCP server, but I can't assume that's true in every case.
- throwawayboise 5y ago
  Yes SuperMicro machines are configured that way. Some don't even have a dedicated separate IPMI interface.

danpalmer 5y ago

BMCs are slightly different I think (more powerful!) but at my previous workplace we once had a server provided to us by our hosting provider that had its BMC exposed on the internet with default credentials. They never told us the machine even had one (most of our servers from them didn’t). We only figured it out once we found a Monero miner on the machine. People on here love promoting dedicated servers over cloud VMs, but it’s so much easier for this sort of thing to go wrong with dedicated hosting.

throwawayboise 5y ago
The latest generations of servers ship with randomized BMC passwords. This was indeed a problem in the past when they shipped with credientials such as ADMIN / ADMIN or no password at all.
- dnautics 5y ago
  "Latest generation" = last 2 years. Three years ago I was buying SMC servers which were the first round configured with randomized passwords to comply with CA law (some batches had ADMIN/ADMIN) and gigabyte servers still were out of compliance.
  - dijit 5y ago
    I bought my last batch of Dell machines in 2017 (so, 5 years? wow), and they had randomised passwords. But yes, some providers put the BMC on the internet because it's easier, a provider I used once did this and I was quite displeased as iDRAC's are quite weak and suffer under the weight of bot-spam. -- even if there were no security issues.
    - zerd 5y ago
      HP Microserver Gen8 with iLO from 2014 also had generated admin password.
      - skinkestek 5y ago

cesaref 5y ago

The older iLO2 and 3 were from the Gen7/8 HP rack mount servers, which were released around 2013 or something like that. I'd have thought that the majority of commercial uses have long passed, so I was wondering if these generation of machines are really being used for home labs, that sort of thing, as they are passed their use by date (I think the gen7 machines were EOSL'd in 2018 so they'll have been chucked out of datacentres). Actually, I can imagine a fair few are still in use as small enterprise servers, with iLO being visible for remote admin, which is a shame, as that suggests they are not behind a tunnel, so those addresses probably indicate larger problems than a visible iLO.

jabart 5y ago
iLO2 is G6 iLO3 is G7 iLO4 is G8 Walking around a DC, no they have not been "chucked out of datacenters"
IYasha 5y ago
You'd probably be surprised to see hundreds of thousands of those machines crunching data in universities, labs, private clusters...

johnklos 5y ago

I reported a security issue in Supermicro's IPMI implementation to them. They dismissed the issue and never fixed it. These companies don't care if there's no way to stop a remote control service (iLo, iDRAC, IPMI) from binding to a motherboard ethernet port. In the case of Supermicro, you can't configure via the built-in BIOS, there are no jumpers you can use, and you must have a full network setup and installed tools to configure a new machine. This makes deploying in the field much, much more problematic. If the BIOS battery dies and settings set are lost, the motherboard defaults to joining whichever ethernet is active, with default credentials. It's incredibly stupid and insecure. I will never buy Supermicro again, but for the hardware that was already bought, I mandated loopback plugs for all IPMI ports so the IPMI wouldn't switch to the system ethernet.

zamadatix 5y ago
Supermicro switched to unique default passwords (same as Dell/HPE) a few years back. Silly to not have day one but really when you look at any of these they are all a security nightmare.
- op00to 5y ago
  Dell’s not root/Calvin anymore?
  - zamadatix 5y ago
    For most probably as you have to know to order it specifically with secure password unlike HPE/Supermicro which just do it every time now. Likely they got hit with a government/large customer requirement and half assed it, as most things BMC are.
  - donflamenco 5y ago
    It hasn't been for years now. The systems get a unique password from the factory and is on the tag.
stingraycharles 5y ago
Honest question: do people actually connect their ipmi / bmc to the public internet? Or is it something different that’s being exposed here?

ggm 5y ago

I love how iDrac shipped with a password everyone knows. I would have thought it was trivial to make it a 1-time passthrough to change it, to stop this being the cartoon character we know and love.

LilBytes 5y ago
Calvin Calvin calvinnnn. You never forget it.

davidzweig 5y ago

Is there a consensus on whether it's ok to port forward from the firewall to port 22 (SSH) on the BMC, with a strong admin password? I guess this should be safer than exposing the web interface.

mkj 5y ago
Pre-auth ssh vulnerabilities are pretty rare, seems likely to be ok unless the BMC software has surprise extra accounts.
whalesalad 5y ago
You never want these interfaces on public internet. All my iDRAC ports are on an isolated management vlan that is only routed to from certain internal networks. 1. You don't want the device even considering requests from anyone but the sole person(s) responsible for accessing the mgmt interface. Someone might not be able to get in, but they could enumerate your hardware characteristics, or perform a denial of service. 2. Don't confuse SSH with secure. You don't know what version/brand/make/model of SSH is running on the embedded device. It might be ancient. It might not even be OpenSSH. 3. If an exploit becomes known for your management interfaces on your machines, you are screwed. First because someone might have already exploited you. Second because now you need to patch your boxes, and that might require a hard reboot of the entire machine. If a patch even exists. Really, it isn't worth the trouble. There are too many risks.

twic 5y ago

The amount of access that BMCs have to the system seems nuts to me. How about a BMC-esque thing whose only function is to provide remote access to the host machine's essential interfaces? On one side, an ethernet port, that you plug into your management plane. On the other side, a serial port, to access the host's console, a device-side SATA port, so it can present a virtual boot disk to the host, and a wire to the power supply, to turn the host on and off. Maybe also some read-only interface to motherboard-level monitoring like SMART and so on; maybe you get that by cooperation with the host's firmware, over the serial or SATA interface. That would let you bring a machine up and configure it, reboot it remotely, force it to boot from a recovery disk, etc. But a hacked BMC couldn't be used to subtly interfere with the host. Also, does anyone know what Oxide are doing about BMC?

immibis 5y ago
That sounds like a lot of access.

bogwog 5y ago

Doesn’t HP require a support contract for firmware updates? That would probably explain why so many of those are on old firmwares.

myself248 5y ago

Seems to me that we'd get some pretty good infosec bang-for-buck to hire some new FBI agents to just wear badges and knock on doors of businesses with such abjectly shittastic security. Either their IT people have been begging to fix it and management won't give them the resources, or they don't have IT people who even see it as a problem. Either way, a couple special agents having a sit-down with the CEO might change some course real fast. Discuss.

stevenicr 5y ago
USDS ? The US Digital Service - maybe? Is that still a thing? I don't think the fbi would be any good at this scenario described.
- donhidalgo 5y ago
  It is still a thing, and agree. usds.gov
Handytinge 5y ago
What makes you think these servers are in the USA?

znpy 5y ago

First things I did on my own server at home: 1. Disable public ipv6 on the iLO interface 2. change all the passwords 3. Set-up tls on the web interface and force http to https redirection. Ironically, point 3 alone is really most of the protection: the web interface is so slow in https it's basically unusable.

infologics 5y ago

[dead]

IYasha 5y ago

Sometimes people don't have choice - it's colocation, dc doesn't offer management vpn/firewalling, you can't install separate vpn router unless you pay for another unit in a rack. And iLO/LOM/WTF at least offers https... what can you do? Well, some people call dc to have their LOs (un-)plugged for maintenance time.