5 ms·

I am not at all shocked by this. FWIW, iLOs have unique highly randomized default credentials, but I'd guess the vast majority are never even touched, much less

I am not at all shocked by this. FWIW, iLOs have unique highly randomized default credentials, but I'd guess the vast majority are never even touched, much less thought about, by the people who manage those systems.

benjojo12 5y ago
There is also https://www.bleepingcomputer.com/news/security/you-can-bypass-authentication-on-hpe-ilo4-servers-with-29-a-characters/ https://www.bleepingcomputer.com/news/security/you-can-bypas... Since HP do not make it easy to get iLO updates if you don't have a support contract with them. I suspect many machines are vulnerable to this.
xoa 5y ago
>but I'd guess the vast majority are never even touched, much less thought about, by the people who manage those systems. Really? I'm certainly no big data center sheep dog, but even for the tiny handful of systems I manage IPMI/BMC always seemed to come with big, flashing red lights just from their mere description. They're enormously powerful and valuable tools, but seems pretty obvious that can work both ways. Beyond their own credentials (and it's a real shame so few support smartcards/hardware tokens for login), having them on their own isolated VLAN seemed about the bare minimum. If budget allows/there are enough systems flat out having their own 100% independent physical infrastructure isn't such a stretch either, not like a simple switch and console system or if necessary a decent minimal bastion is that big a price tag these days. In some cases may make the most sense to just use them to setup then disconnect them. You may well be right but still, yeesh. It's remote super root access, should make any sysadmin treat it with healthy fear from just a description. Although what did give me the willies was how it seems common to have IPMI be hybrid with LAN1 by default rather then it's own physically independent port. I switched from HPE to SuperMicro kit because at our scale and application it made sense for a lot of reasons anyway, but that HPE wants a lot of extra money for a dedicated port and full fat features was a real factor.
- count 5y ago
  They were either not plugged in, or definitely not on their own vlan in any of the datacenters (dozens of them) I worked in from 2000-2007. Many admins have no clue they exist or what they do.
  - formerly_proven 5y ago
    On most boards where the ILOM / BMC has a dedicated LAN port it usually also has access to at least one of the main ethernet ports on the MAC level to support hooking machines up with just one cable without loosing ILOM.
- spijdar 5y ago
  Unfortunately I have to concur with GP, my experience in penetration testing is that many sysadmins seem oblivious to the existence of ILO systems, or don't believe they can cause any harm. Alternatively, sometimes one simply slips through the cracks, and all it takes is one system with cached credentials to wreck havoc to a network. Ironically, the more expensive ILO systems can be easier to exploit. Just throw a Linux live rescue image onto virtual media, reboot the machine, and you can grab everything sensitive fairly quickly.
  - xoa 5y ago
    Well (and to sibling comment), TIL, thank you :(. That's scary though, that anyone could even use the BMC and not at least pause a bit on "wait a sec, what could someone else do with this", same as hypervisor access. Though maybe some of the blame really should be on the vendors themselves here. At least credentials are randomized now, but they could be much more aggressive about making it something that has to be actively turned on, always has it's own physically isolated port, and requires some level of secure setup (won't communicate untagged maybe?). Or these days making hardware token/smartcard usage required perhaps. There's clearly tradeoffs to be made in terms of security vs convenience/recovery from bad setups, and the balance has fallen fairly decisively in favor of the latter. Kind of depressing though the industry constantly has to do the same thing of ignoring security right up until it gets really, really bad.
fulafel 5y ago
I think the sad truth is that the very long swiss cheese record of this stuff just fits right into, and contributes to, the "secure internal network" mental model of most on-premises cultures. It also benefits the vendors, because they can sell more feebly secured on-prem stuff.

benjojo12 5y ago

There is also https://www.bleepingcomputer.com/news/security/you-can-bypass-authentication-on-hpe-ilo4-servers-with-29-a-characters/ https://www.bleepingcomputer.com/news/security/you-can-bypas... Since HP do not make it easy to get iLO updates if you don't have a support contract with them. I suspect many machines are vulnerable to this.

xoa 5y ago

>but I'd guess the vast majority are never even touched, much less thought about, by the people who manage those systems. Really? I'm certainly no big data center sheep dog, but even for the tiny handful of systems I manage IPMI/BMC always seemed to come with big, flashing red lights just from their mere description. They're enormously powerful and valuable tools, but seems pretty obvious that can work both ways. Beyond their own credentials (and it's a real shame so few support smartcards/hardware tokens for login), having them on their own isolated VLAN seemed about the bare minimum. If budget allows/there are enough systems flat out having their own 100% independent physical infrastructure isn't such a stretch either, not like a simple switch and console system or if necessary a decent minimal bastion is that big a price tag these days. In some cases may make the most sense to just use them to setup then disconnect them. You may well be right but still, yeesh. It's remote super root access, should make any sysadmin treat it with healthy fear from just a description. Although what did give me the willies was how it seems common to have IPMI be hybrid with LAN1 by default rather then it's own physically independent port. I switched from HPE to SuperMicro kit because at our scale and application it made sense for a lot of reasons anyway, but that HPE wants a lot of extra money for a dedicated port and full fat features was a real factor.

count 5y ago
They were either not plugged in, or definitely not on their own vlan in any of the datacenters (dozens of them) I worked in from 2000-2007. Many admins have no clue they exist or what they do.
- formerly_proven 5y ago
  On most boards where the ILOM / BMC has a dedicated LAN port it usually also has access to at least one of the main ethernet ports on the MAC level to support hooking machines up with just one cable without loosing ILOM.
spijdar 5y ago
Unfortunately I have to concur with GP, my experience in penetration testing is that many sysadmins seem oblivious to the existence of ILO systems, or don't believe they can cause any harm. Alternatively, sometimes one simply slips through the cracks, and all it takes is one system with cached credentials to wreck havoc to a network. Ironically, the more expensive ILO systems can be easier to exploit. Just throw a Linux live rescue image onto virtual media, reboot the machine, and you can grab everything sensitive fairly quickly.
- xoa 5y ago
  Well (and to sibling comment), TIL, thank you :(. That's scary though, that anyone could even use the BMC and not at least pause a bit on "wait a sec, what could someone else do with this", same as hypervisor access. Though maybe some of the blame really should be on the vendors themselves here. At least credentials are randomized now, but they could be much more aggressive about making it something that has to be actively turned on, always has it's own physically isolated port, and requires some level of secure setup (won't communicate untagged maybe?). Or these days making hardware token/smartcard usage required perhaps. There's clearly tradeoffs to be made in terms of security vs convenience/recovery from bad setups, and the balance has fallen fairly decisively in favor of the latter. Kind of depressing though the industry constantly has to do the same thing of ignoring security right up until it gets really, really bad.

fulafel 5y ago

I think the sad truth is that the very long swiss cheese record of this stuff just fits right into, and contributes to, the "secure internal network" mental model of most on-premises cultures. It also benefits the vendors, because they can sell more feebly secured on-prem stuff.