6 ms·

IPMI

280 points by pabs3 2y ago

saltminer 2y ago
> This is not to make excuses for Intel ME, which is entirely unauditable by third parties and has harbored significant security vulnerabilities in the past. But, remember, we all use one processor architecture from one of two vendors, so Intel doesn't have a whole lot of motivation to do better. Lest you respond that ARM is the way, remember that modern ARM SOCs used in consumer devices have pretty much identical capabilities. I actually took advantage of one such security vulnerability to unlock my old Moto X's bootloader! The details of the exploit are quite interesting: https://bits-please.blogspot.com/2015/03/getting-arbitrary-code-execution-in.html https://bits-please.blogspot.com/2015/03/getting-arbitrary-c... https://bits-please.blogspot.com/2015/08/exploring-qualcomms-trustzone.html https://bits-please.blogspot.com/2015/08/exploring-qualcomms... https://bits-please.blogspot.com/2015/08/full-trustzone-exploit-for-msm8974.html https://bits-please.blogspot.com/2015/08/full-trustzone-expl... https://bits-please.blogspot.com/2016/02/unlocking-motorola-bootloader.html https://bits-please.blogspot.com/2016/02/unlocking-motorola-...
smitty1e 2y ago
Intelligent Platform Management Interface https://en.m.wikipedia.org/wiki/Intelligent_Platform_Management_Interface https://en.m.wikipedia.org/wiki/Intelligent_Platform_Managem...
TMWNN 2y ago
I have found HP iLO very useful on my MicroServer Gen8 with UnRAID.
renewiltord 2y ago
Personally, for smaller deployments (I.e. about 10k cores) I would just self deploy out from some integrator. A Gigabyte/AsRock Rack Motherboard + Epyc 9003 series + some 384 G of RAM with the usual dual PSUs and all that stuff will be $7k/node and can be very power efficient. The integrated IPMI is pretty good on these and plays well with ipmitool. They'll usually have some Redfish stuff going on.
buserror 2y ago
IPMI is such an improvement on the old ISA-style iochips, and not even sure they are more expensive these days! It's not just the remote/web access which is great, plain 'introspection' with tools like ipmiutil and ipmitool are super useful, even if you don't have a huge rack 'enterprise' installation.
brcmthrowaway 2y ago
Was IPMI useful in solving the CrowdStrike boondoggle?
- xaduha 2y ago
  Since one of the solutions was to reboot "up to 15 times" I would imagine so.
- sofixa 2y ago
  Yep, because on enterprise servers in a rack somewhere in a remote datacenters, IPMI is what allows admins to connect remotely, restart, and get a remote console to remove the offending files (or just keep restarting).
  - throw0101d 2y ago
    > […] and get a remote console to remove the offending files (or just keep restarting). Sometimes (often?) you need a more 'advanced' OOB/LOM license to get remote console from Dell/HP/Lenovo/etc.
- omgtehlion 2y ago
  I suppose it was useful only for a part of the problem, as most CrowdStroke™ machines were client and kiosk...
sandreas 2y ago
I really like IPMI, but what I don't like about it for homelab use cases is the additional ~5W of idle power consumption. Using the Gigabyte MC12-LE0 Board with Ryzen Pro 5650 for a home server should be a no brainer costing ~50 bucks, but the higher power consumption is something I'm not totally happy with. On older devices (like Dell T20/T30) there is Intel AMT which is way less powerful and has security flaws, but at least SOME way to do remote administration when used with MeshCommander (unfortunately discontinued and releases deleted from everywhere - but I luckily saved the MSI and Node-Package on my server). I'm planning to try out PiKVM (V2) with a Raspberry 4 and a simple 8 bucks USB-HDMI-Capture-Card (https://docs.pikvm.org/v2/ https://docs.pikvm.org/v2/). Besides the lack of some features, it looks promising and more universal for devices that do not support remote management in any form.
- dugite-code 2y ago
  Similarly I'm looking into the nano kvm now the firmware is open https://github.com/sipeed/NanoKVM https://github.com/sipeed/NanoKVM
  - dugite-code 2y ago
    Update, perhaps I was miss-informed. Looks like the firmware isn't in the repo, just the web interface. Disappointing, I had read on a blog somewhere recently that they had released the firmware and didn't look into it deeply enough > the backend will be open-sourced soon (after the GitHub repository reaches 2K stars). https://en.wiki.sipeed.com/hardware/en/kvm/NanoKVM/system/introduction.html https://en.wiki.sipeed.com/hardware/en/kvm/NanoKVM/system/in...
    - sandreas 2y ago
      I would love to see an OpenWRT implementation, so that your router can be your KVM device :-)
      - stragies 2y ago
        I believe, that few, if any of the currently supported boards feature all of: USB Device Port, HDMI-Port (or sufficiently fast USB-Host-Port for Capture Adapter), HW-Video-Encode (or beafy enough CPU), enough available (programmable) GPIOs
sofixa 2y ago
A few notes because there are a few pieces of information which aren't a 100% up to date: As most people know, Intel have lost the plot and are behind AMD on everything CPU and GPU related (with the tiny exception of the N100 series CPUs from Intel which are perfect for low power and even fanless applications). As such, their CPUs are mostly bought by organisations updating previous environments with like for like replacements for whatever reason (like vSphere's EVC which allows you to run newer processors from the same manufacturer as if they were an older model, and enable hot migration between CPU architectures and thus minimum disruption on hardware refreshes). Pretty much everyone else is getting the better and cheaper (per processing power) AMD CPUs. Intel NICs are pretty good overall (the X710 which spent a year+ with broken drivers causing silent network failures/downright crashes while still being in the hardware compatibility list of VMware and other "enterprise" vendors notwithstanding), and they're getting more popular for consumer devices too. An Intel CPU on a non-low cost PC/laptop almost always means an Intel NIC (WiFi and/or Ethernet). For many organisations buying servers, Supermicro* is probably their best option. They're cheaper, have much higher flexibility in terms of form factors, chassis, components, number of whatever slots and mostly reliable. However their support is less reliable than the theoretical Dell/HPE support, so it works best for redundant setups. Also, the IPMI specification is being replaced by Redfish which is much more complete, secure, normalised and presents normal usable APIs. Any mainline server from a few years ago should have Redfish alongside IPMI. * There's recent research from Hindenburg, a short selling research firm, that exposes some shady stuff from Supermicro - but their hardware is still top notch and used by the hyperscalers. https://hindenburgresearch.com/smci/ https://hindenburgresearch.com/smci/
- tucnak 2y ago
  I really wish Supermicro PSU's were accessible over PMBus without their proprietary IPMI utilities[1] but they're not. Moreover, it's x86-only so there's no way to interface it from ppc64el. Had it been open source, it could trivially be built. [1] https://www.supermicro.com/en/solutions/management-software/ipmi-utilities https://www.supermicro.com/en/solutions/management-software/...
- buserror 2y ago
  Having had a play with both Supermicro and ASRock Rack boards for workstations (admittedly, only H12's so not the "most recent" but from my cursory glance and newer boards, nothing has changed that much), SuperMicro board feels like they were made in 2005, not 2024. It is ridiculous in fact. * No support for ACPI sleep. In 2024. Seriously. * No support for 4 and 3 pins fans. 3 pins are 100% speed all the time. * IPMI web interface straight out of 2010. * NVME placement prevents you from using heatsinks. * Tons of opaque jumpers on the board, with no board labelling. The ASRock Rack equivalent board is amazing in comparison.
hexmiles 2y ago
Does anyone know about AMD ST? It is the first time I've seen it mentioned. I tried to google it, but without result.
- voxadam 2y ago
  AMD ST (Secure Technology) is the official name for what most people call AMD PSP (Platform Security Processor).
xaduha 2y ago
IPMI and other solutions are nice, but what I would like to have is a standard serial interface to an UEFI shell running at all times. How I access that serial port should be my problem.
- bpye 2y ago
  Server hardware often will let you access UEFI via serial. You’d still need remote power control no?
  - blipvert 2y ago
    Silicon Graphics servers used to have a separate serial port that proved access to a (very) simple state machine that controlled power to the system. Send a ‘u’ it powered on. Send a ‘d’ it powered down. ‘s’ reported the state (IIRC). There was literally nothing that could go wrong. Then install the OS from the regular console port over the network with bootp/tftp/http. The complexity of DRAC/iLO setups to control an emulation of a VGA PC setup blows my mind.
    - gnufx 2y ago
      > an emulation of a VGA PC setup IPMI does rather more than than giving you console access, though it's serial, not VGA. Typical server BMCs which embed IPMI do more again. Not to defend the quality of various BMC firmware and support I've encountered...
      - blipvert 2y ago
        Oh, yes, absolutely - fair point. It was more the remote power/boot/install process which seems unnecessarily complicated these days compared to serial. An elegant weapon for a more civilised age, or something ;-)
        BobbyTables2 2y ago
        Indeed, but iLO virtual media performance (at least used to be) so slow that it was silly. Booting a live cd or installing windows at 1MByte/s speeds (late 1990s CDROMs were faster!) is horribly slow. Maybe iLO5 was a bit better. usually found it easier (but annoying) to just walk over with a USB drive…
grw_ 2y ago
If anyone wants to add remote access capability to machines without IPMI you can try with something like the $30 RISC-V NanoKVM[1][2]. It provides HDMI capture (and encoding), Ethernet/Wifi, ATX power control, and runs a normal linux distribution. [1] https://www.aliexpress.com/item/1005007369816019.html https://www.aliexpress.com/item/1005007369816019.html [2] https://github.com/sipeed/NanoKVM https://github.com/sipeed/NanoKVM
- Ir0nMan 2y ago
  Do we know why AliExpress doesn't have the device available for sale to US customers?
- mananaysiempre 2y ago
  Notably, you need the $60 “full” version to get the ATX power control breakout that attaches to the perverse physically-USB-C connector that has the ATX signals (why? why did they make this abomination?..). Or make your own, I guess, that’s always an option (if only USB-C connectors weren’t absolute ass to solder).
  - jdboyd 2y ago
    Probably because USBC connectors are really cheap. If this was 5 years ago, they probably would be doing something terrible with a usb 3 A connector instead.
  - riobard 2y ago
    I didn't use the ATX aux board because my mini PC does not have such headers. I just use Wake-on-LAN from NanoKVM web UI to power up my PC :P
    - eqvinox 2y ago
      There's no Reset-on-LAN though, sadly…
      - gsa 2y ago
qwertox 2y ago
Can an AST2600 BMC be used as a graphics card by Ubuntu Desktop? I've read that it has a "2D Video Graphic Adapter with PCIe bus interface", which makes it sound like it is exposed to the OS like any other graphics card. The desktop would be there only for troubleshooting, not for normal stuff which would usually require an iGPU or better. If yes, would it work out of the box with Ubuntu?
- grw_ 2y ago
  yes, it does- lspci | grep ASP c2:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 52)
- vardump 2y ago
  Yes, but it's not going to be speedy, as it is just a dumb framebuffer and the maximum resolution is 1920x1080. The most modern feature is probably double buffering. :-)
- eqvinox 2y ago
  Can it? Yes. Do you want to? No. It's essentially just a framebuffer, not a GPU; it barely has 2D graphics acceleration. Using any kind of modern DE on it will drive you insane within minutes. Xfce and classic X11 might work.
  - lproven 2y ago
    > Using any kind of modern DE on it will drive you insane within minutes. Most of them do that anyway, so... shrug > Xfce and classic X11 might work. You say that like it's a bad thing.
- geerlingguy 2y ago
  I use it with ASPEED's latest Windows driver at 1080p, and it does a passable job on my Ampere Altra system for simple tasks. Most servers don't run an accelerated server desktop and do things like play AAA games and YouTube.
concerndc1tizen 2y ago
Conspiracy theory: I wonder if the ____KVM people are betting that most people won't flash their own firmware, and just keep using what was provided, which may contain malware, both for accessing the network, and controlling the connected hardware.
Animats 2y ago
A big question with IPMI is what the default keys are. If someone manages to install some extra IPMI keys somewhere in the supply chain, or there's a default key, they can remote admin the computer.[1] [1] https://www.rapid7.com/blog/post/2013/07/02/a-penetration-testers-guide-to-ipmi/ https://www.rapid7.com/blog/post/2013/07/02/a-penetration-te...
- gnufx 2y ago
  Well, it should always be on a physically isolated network, and avoid BMCs which share their NIC with the system.
- jabroni_salad 2y ago
  An additional quote from that link: > In short, the authentication process for IPMI 2.0 mandates that the server send a salted SHA1 or MD5 hash of the requested user's password to the client, prior to the client authenticating. ipmi also has a maximum password length of 20 characters. In realworld you have to note that we only expect hashes to remain secret from known pentesters who have a limited contractual runtime, not a real attacker with unlimited time on their hands. I'm very critical of this. It's been in the spec for 20 years. Is not the whole point of software that it is easier to change than hardware? It's easy to say "you should put this on a vlan" but I pretty much ALWAYS see ipmi on the business network in my assessments. If you put dumb stuff in your defaults then you're gonna end up with dumb stuff all over the globe and that includes every business without a knowledgeable security administrator that decides to "buy a server".
_joel 2y ago
ipmitool has dug me out of many sticky situations in the past, although it needs to be handled as proper oob, in terms of network isolation.
OSI-Auflauf 2y ago
Seen devices with IPMI that had by design unauthenticated admin login to the IPMI from the host side that was not removable. They also could flash IPMI firmware from the host. So if your server with such an IPMI is infected you can't trust reimaging it via IPMI because that can be hijacked as well.
- justsomehnguy 2y ago
  "It rather involved being on the other side of this airtight hatchway"
- dfox 2y ago
  I would consider that mostly a feature. The situation where that is useful (you somehow lost the credentials for BMC, but have root access to the host) is in my experience significantly more common (I see that multiple times in a year) than attacker implanting stuff into the BMC firmware (never seen that). Obviously if you rent out whole physical machines and automate the provisioning by IPMI, then the last thing you want is the customer having admin access to the BMC. Dell iDRAC has an interesting feature that allows you to make all of the BMC configuration read-only which can only be disabled by factory resetting the iDRAC by means of physical (and IIRC not exactly documented) switch on the BMC board. (Well, it is still _i_DRAC as in “integrated”, but on current higher-end PowerEdges the iDRAC is separate OCP-like card, but well, the system does not work without it)
ggm 2y ago
Hands up if you used a real weasel! Cute advertising imagery. I depend on idrac. The newer one is bearable. The old Java was torrid.
foxhop 2y ago
[dead]
dfox 2y ago
As for the Intel ME and AMD PSP: these things have a huge role in getting the CPU to state that looks like x86 PC so that the host firmware can actually run. The complexity of the initialization got so high, that doing that in SW on a separate embedded core that is “well-behaved” and can be programmed in C makes more sense than trying to write all that logic in weirdly limited assembly that is inherent in traditional early stage BIOS startup code. I believe that at least on some HPE ProLiants some part of this low-level initialization is in fact done by iLo, which at that boot stage also directly controls the framebuffer (on G10 you can even briefly see a flash of message like “handing over the console to host”, after which the display reinitializes and starts showing the function key legends). I assume that Dell does something similar, but in the early stages you simply get “Please wait” and giant throbber (and not any kind of progress indication).
swiftcoder 2y ago
Does anyone else see a pixel or so of mismatch between the background and the text when scrolling?
EvanAnderson 2y ago
In the late 90s I installed several Intel-produced servers (i.e. Intel reference platforms shipped as "bare bones" computers w/ RAM and storage to be provided by the integrator) that used the LANDesk Server Manager Pro[0] software and "Emergency Management Card" (EMC) for "lights out" management. (I'm talking about boxes like the AP450GX, BB440FX, RC440FX-- Pentium Pro to early Pentium II timeframe.) Given how reference code for the x86 platform never dies I've often wondered how much of the current IPMI edifice dates back to this hardware and software. As a point of note, the default password for the Intel LANDesk Emergency Management Card was "calvin". If you worked with earlier versions of the Dell iDRAC this will be a familiar password. I assume it's not coincidence. (As an aside: I was told by an Intel employee that the code name for the EMC was "Hobbes" but I can't find that documented anywhere.) You see versions of the EMC periodically on eBay.[1] There were both ISA and PCI versions. They're an x86 PC on a card. Some (all?) of them have an integral UPS. They had a PCMCIA slot to attach out-of-band management, an external power supply, and connected to the server motherboard through both the card's host bus interface and via proprietary connectors. I've downloaded firmware for some of the EMC versions and looked it over. It looks like some of the EMC versions an embedded DOS machine. (This has been one of my idle "when I get to it" projects and I haven't actually tried to reverse-engineer any of the code or get it running under qemu, though I'd like to.) Third-party companies who sold the Intel reference platforms (Unisys, Fujitsu, ALR/Gateway, NCR) offered these cards, too. It's a good giveaway that they're using an Intel reference platform when you see the card on-offer. References to "LDSM" are a giveaway. It'd be really wild if anybody on here knows anything about the heritage. [0] https://www.intel.com/pressroom/archive/releases/1998/ld103098.htm https://www.intel.com/pressroom/archive/releases/1998/ld1030... [1] https://web.archive.org/web/20240903131630/https://www.ebay.com/p/127953588 https://web.archive.org/web/20240903131630/https://www.ebay....
Neil44 2y ago
Not a fan of the web based OOB management, it's convenient for now but this morning I had a right faff trying to connect to an old iLO3 service, which is new enough to force https but old enough that all it's protocols are deprecated on modern machines. I've used vPro and found it almost deliberately difficult to use, and poorly documented. It could be so easy but just isn't. Like so many things... shakes fist at sky.
- toast0 2y ago
  > which is new enough to force https but old enough that all it's protocols are deprecated on modern machines This is the danger of https only. http services may be insecure, but don't ever expire; https implementations will expire without updates, and there's not going to be updates on these, and it's unknowable when the expiration will be.
  - quesera 2y ago
    Firefox can be configured to allow obsolete SSL/TLS ciphers: about:config security.tls.version.min = 0 Obligatory: Do not leave this setting in place for routine browsing. Reset to default (currently 3), or only run in a dedicated profile, etc.
    - Neil44 2y ago
      Thankyou, this is what I did. It seems Edge, Chrome is not so easy to do.
neilv 2y ago
> IPMI is very common in enterprise servers but very rare elsewhere, much to the consternation of people like me that don't have the space or noise tolerance for a 1U pizzabox in their homes. If you are trying to stick to compact or low-power computers, you'll pretty much have to go without. One option: you can get IPMI on an Atom-based Supermicro MicroATX board that can be cooled quietly with a small Noctua fan, in a short-depth 1U chassis. I have an older one at home. The quietness and small size made it much more attractive than the Dell R2x0 models I saw. And it has server features like IPMI and ECC RAM that made it more attractive than a mini-PC, and it was also more reliable than a flimsy RasPi. (I'd be happy with a Dell quiet short-depth 1U with comparable characteristics, but I'm not aware of one.) Personally, I wouldn't plug the IPMI port into my main LAN, but it's come in handy isolated from that, and was also a little fun to play with.
- organsnyder 2y ago
  I keep my IPMI and other management interfaces segregated to a separate management VLAN that's only accessible via a dedicated VPN.
  - pabs3 2y ago
    That isn't enough, some implementations are also always or fallback to using the main NIC transparently in parallel to the host CPU use of it.
  - temp0826 2y ago
    This is probably the minimum you should do. I remember a time when IPMI on (some?) Supermicro boards was really really insecure. From the IPMI client, you could set the encryption mode to "0" ("null encryption" or something, I dunno it's been years)- setting it allowed you to bypass the password completely. Assume if you can touch the IPMI, the system is yours.
    - crest 2y ago
      IPMI is still a festering cesspool no matter which vendor. Assume that layer 3 access to the IPMI grants you unrestricted persistent code execution on the managed system and design for with it in mind. Restrict access as tightly as possible. If you only need to power up/down/reset the system and access the serial console most IPMI implementations expose that via SSH. A small SSH proxy that exposes only those features would be a good investment. e.g. `ssh bastion [status|up|down|cycle|reset|console] <server>`. You could probably write it in <100 lines as SSH forced command. Deploy the SSH client to a different VRF (aka network namespace/vnet jail/rdomain) than the SSH server to make it harder to leak traffic from the bastion.
ultra_nick 2y ago
Dell is working pretty hard to replace IPMI with Redfish these days. Redfish is an out of band management protocol like IPMI, but using an HTTP/JSON REST API that doesn't assume a secure management network. https://www.dmtf.org/standards/redfish https://www.dmtf.org/standards/redfish
- jayofdoom 2y ago
  Don't just give Dell the credit here :) OpenStack Ironic (ironicbaremetal.org) now ships a redfish driver that we have tested working on: - Dell machines - HPE machines (ilo 5 and 6) - SuperMicro - Most generic redfish endpoints as deployed by bulk "off label" server places A lot of folks in the provisioning automation space worked hard to try and enhance the state of the art (including some folks at Dell who were great :D) and get redfish accepted. It's a bit weird seeing this post; from our perspective, IPMI is a dangerous, attractive nuisance: it's nearly impossible to properly secure and has a lot more bad failure scenarios than a protocol built on http, that most of the time can support TLS with custom CAs and similar.
johnklos 2y ago
IPMI has its place, but it clearly shows how none of us can trust commercial entities to properly support hardware long term. The OS that runs on IPMI is usually relatively secure when the system is new, but as soon as there's a new CPU socket, the manufacturer starts caring less and less about keeping the older systems up to date, even if the same IPMI hardware is on both old and new boards. If we could load our own OS on to the IPMI hardware, it's be worlds more useful, because we could safely connect it straight to the Internet. As it is, we need sideband communications (VPN, port forward over ssh, separate network segment, whatever), which makes it so that we end up with lots of extra hardware and configurations just to support IPMI. For large installations, the extra cost is better aggregated, but for small installations, it's a rather large burden. Heck - for colocating a single machine, it's not even worth it. Since it's not possible to run IPMI directly on the Internet securely, I've ended up pairing machines with Pis of some sort. Doing this makes it just as easy - heck, easier - to use a serial port. That means we're back to the standard serial port control we've had since the days of VAXen and Sun and Alpha systems, which, the more I think about it, made much more sense than a network interface that's insecure.
peter_d_sherman 2y ago
My takeaways: >"Out-of-band management, sometimes also called lights-out management, identifies a capability that is almost unheard of in clients. A separate, smaller management computer allows for remote access to a server even when it is, say, powered off. The terms out-of-band and in-band in this context emerge from their customery uses in networking and telecom, meaning that out of band management is performed without the use of the standard (we might say "data plane") network connection to a machine. But in practice they have drifted in meaning, and it is probably better to think of out-of-band management as meaning that the operating system and general-purpose components are not required. This might be made clearer by comparison: a very standard example of in-band management would be SSH, a service provided by the software on a computer that allows you to interact with it. Out-of-band management, by contrast, is provided by a dedicated hardware and software stack and does not require the operating system or, traditionally, even the CPU to cooperate." [...] >"The IPMI software runs on a processor conventionally called the baseboard management controller or BMC, and the terms IPMI and BMC are sometimes used interchangeably. Lights-out management or LOM is mostly an obsolete term but sticks around because HP(E) is a fan of it and continues to call their IPMI implementation Integrated Lights-Out. The BMC should not be confused with the System Management Controller or SMC" [...] "IPMI offers "sideband" networking. With sideband management, the BMC communicates directly with the same NIC that the operating system uses. The implementation is a little bit weird: the NIC will pretend to be two different interfaces, mixing IPMI traffic into the same packet stream as host traffic but using a different MAC address." [...] "Details of network features vary between IPMI implementations, but there is a standard interface on UDP 623 that can be used for discovery and basic commands. There's often SSH and a web interface, and VNC is pretty common for remote console." This is arguably the best educational article on IPMI and related subsystems (I did not know about 90% of this!) that I've read for a long time. Well worth a re-read in the future...

280 points by pabs3 2y ago

> This is not to make excuses for Intel ME, which is entirely unauditable by third parties and has harbored significant security vulnerabilities in the past. But, remember, we all use one processor architecture from one of two vendors, so Intel doesn't have a whole lot of motivation to do better. Lest you respond that ARM is the way, remember that modern ARM SOCs used in consumer devices have pretty much identical capabilities. I actually took advantage of one such security vulnerability to unlock my old Moto X's bootloader! The details of the exploit are quite interesting: https://bits-please.blogspot.com/2015/03/getting-arbitrary-code-execution-in.html https://bits-please.blogspot.com/2015/03/getting-arbitrary-c... https://bits-please.blogspot.com/2015/08/exploring-qualcomms-trustzone.html https://bits-please.blogspot.com/2015/08/exploring-qualcomms... https://bits-please.blogspot.com/2015/08/full-trustzone-exploit-for-msm8974.html https://bits-please.blogspot.com/2015/08/full-trustzone-expl... https://bits-please.blogspot.com/2016/02/unlocking-motorola-bootloader.html https://bits-please.blogspot.com/2016/02/unlocking-motorola-...

smitty1e 2y ago

Intelligent Platform Management Interface https://en.m.wikipedia.org/wiki/Intelligent_Platform_Management_Interface https://en.m.wikipedia.org/wiki/Intelligent_Platform_Managem...

TMWNN 2y ago

I have found HP iLO very useful on my MicroServer Gen8 with UnRAID.

renewiltord 2y ago

Personally, for smaller deployments (I.e. about 10k cores) I would just self deploy out from some integrator. A Gigabyte/AsRock Rack Motherboard + Epyc 9003 series + some 384 G of RAM with the usual dual PSUs and all that stuff will be $7k/node and can be very power efficient. The integrated IPMI is pretty good on these and plays well with ipmitool. They'll usually have some Redfish stuff going on.

buserror 2y ago

IPMI is such an improvement on the old ISA-style iochips, and not even sure they are more expensive these days! It's not just the remote/web access which is great, plain 'introspection' with tools like ipmiutil and ipmitool are super useful, even if you don't have a huge rack 'enterprise' installation.

brcmthrowaway 2y ago

Was IPMI useful in solving the CrowdStrike boondoggle?

xaduha 2y ago
Since one of the solutions was to reboot "up to 15 times" I would imagine so.
sofixa 2y ago
Yep, because on enterprise servers in a rack somewhere in a remote datacenters, IPMI is what allows admins to connect remotely, restart, and get a remote console to remove the offending files (or just keep restarting).
- throw0101d 2y ago
  > […] and get a remote console to remove the offending files (or just keep restarting). Sometimes (often?) you need a more 'advanced' OOB/LOM license to get remote console from Dell/HP/Lenovo/etc.
omgtehlion 2y ago
I suppose it was useful only for a part of the problem, as most CrowdStroke™ machines were client and kiosk...

sandreas 2y ago

I really like IPMI, but what I don't like about it for homelab use cases is the additional ~5W of idle power consumption. Using the Gigabyte MC12-LE0 Board with Ryzen Pro 5650 for a home server should be a no brainer costing ~50 bucks, but the higher power consumption is something I'm not totally happy with. On older devices (like Dell T20/T30) there is Intel AMT which is way less powerful and has security flaws, but at least SOME way to do remote administration when used with MeshCommander (unfortunately discontinued and releases deleted from everywhere - but I luckily saved the MSI and Node-Package on my server). I'm planning to try out PiKVM (V2) with a Raspberry 4 and a simple 8 bucks USB-HDMI-Capture-Card (https://docs.pikvm.org/v2/ https://docs.pikvm.org/v2/). Besides the lack of some features, it looks promising and more universal for devices that do not support remote management in any form.

dugite-code 2y ago
Similarly I'm looking into the nano kvm now the firmware is open https://github.com/sipeed/NanoKVM https://github.com/sipeed/NanoKVM
- dugite-code 2y ago
  Update, perhaps I was miss-informed. Looks like the firmware isn't in the repo, just the web interface. Disappointing, I had read on a blog somewhere recently that they had released the firmware and didn't look into it deeply enough > the backend will be open-sourced soon (after the GitHub repository reaches 2K stars). https://en.wiki.sipeed.com/hardware/en/kvm/NanoKVM/system/introduction.html https://en.wiki.sipeed.com/hardware/en/kvm/NanoKVM/system/in...
  - sandreas 2y ago
    I would love to see an OpenWRT implementation, so that your router can be your KVM device :-)
    - stragies 2y ago
      I believe, that few, if any of the currently supported boards feature all of: USB Device Port, HDMI-Port (or sufficiently fast USB-Host-Port for Capture Adapter), HW-Video-Encode (or beafy enough CPU), enough available (programmable) GPIOs

sofixa 2y ago

A few notes because there are a few pieces of information which aren't a 100% up to date: As most people know, Intel have lost the plot and are behind AMD on everything CPU and GPU related (with the tiny exception of the N100 series CPUs from Intel which are perfect for low power and even fanless applications). As such, their CPUs are mostly bought by organisations updating previous environments with like for like replacements for whatever reason (like vSphere's EVC which allows you to run newer processors from the same manufacturer as if they were an older model, and enable hot migration between CPU architectures and thus minimum disruption on hardware refreshes). Pretty much everyone else is getting the better and cheaper (per processing power) AMD CPUs. Intel NICs are pretty good overall (the X710 which spent a year+ with broken drivers causing silent network failures/downright crashes while still being in the hardware compatibility list of VMware and other "enterprise" vendors notwithstanding), and they're getting more popular for consumer devices too. An Intel CPU on a non-low cost PC/laptop almost always means an Intel NIC (WiFi and/or Ethernet). For many organisations buying servers, Supermicro* is probably their best option. They're cheaper, have much higher flexibility in terms of form factors, chassis, components, number of whatever slots and mostly reliable. However their support is less reliable than the theoretical Dell/HPE support, so it works best for redundant setups. Also, the IPMI specification is being replaced by Redfish which is much more complete, secure, normalised and presents normal usable APIs. Any mainline server from a few years ago should have Redfish alongside IPMI. * There's recent research from Hindenburg, a short selling research firm, that exposes some shady stuff from Supermicro - but their hardware is still top notch and used by the hyperscalers. https://hindenburgresearch.com/smci/ https://hindenburgresearch.com/smci/

tucnak 2y ago
I really wish Supermicro PSU's were accessible over PMBus without their proprietary IPMI utilities[1] but they're not. Moreover, it's x86-only so there's no way to interface it from ppc64el. Had it been open source, it could trivially be built. [1] https://www.supermicro.com/en/solutions/management-software/ipmi-utilities https://www.supermicro.com/en/solutions/management-software/...
buserror 2y ago
Having had a play with both Supermicro and ASRock Rack boards for workstations (admittedly, only H12's so not the "most recent" but from my cursory glance and newer boards, nothing has changed that much), SuperMicro board feels like they were made in 2005, not 2024. It is ridiculous in fact. * No support for ACPI sleep. In 2024. Seriously. * No support for 4 and 3 pins fans. 3 pins are 100% speed all the time. * IPMI web interface straight out of 2010. * NVME placement prevents you from using heatsinks. * Tons of opaque jumpers on the board, with no board labelling. The ASRock Rack equivalent board is amazing in comparison.

hexmiles 2y ago

Does anyone know about AMD ST? It is the first time I've seen it mentioned. I tried to google it, but without result.

voxadam 2y ago
AMD ST (Secure Technology) is the official name for what most people call AMD PSP (Platform Security Processor).

xaduha 2y ago

IPMI and other solutions are nice, but what I would like to have is a standard serial interface to an UEFI shell running at all times. How I access that serial port should be my problem.

bpye 2y ago
Server hardware often will let you access UEFI via serial. You’d still need remote power control no?
- blipvert 2y ago
  Silicon Graphics servers used to have a separate serial port that proved access to a (very) simple state machine that controlled power to the system. Send a ‘u’ it powered on. Send a ‘d’ it powered down. ‘s’ reported the state (IIRC). There was literally nothing that could go wrong. Then install the OS from the regular console port over the network with bootp/tftp/http. The complexity of DRAC/iLO setups to control an emulation of a VGA PC setup blows my mind.
  - gnufx 2y ago
    > an emulation of a VGA PC setup IPMI does rather more than than giving you console access, though it's serial, not VGA. Typical server BMCs which embed IPMI do more again. Not to defend the quality of various BMC firmware and support I've encountered...
    - blipvert 2y ago
      Oh, yes, absolutely - fair point. It was more the remote power/boot/install process which seems unnecessarily complicated these days compared to serial. An elegant weapon for a more civilised age, or something ;-)
      - BobbyTables2 2y ago
        Indeed, but iLO virtual media performance (at least used to be) so slow that it was silly. Booting a live cd or installing windows at 1MByte/s speeds (late 1990s CDROMs were faster!) is horribly slow. Maybe iLO5 was a bit better. usually found it easier (but annoying) to just walk over with a USB drive…

grw_ 2y ago

If anyone wants to add remote access capability to machines without IPMI you can try with something like the $30 RISC-V NanoKVM[1][2]. It provides HDMI capture (and encoding), Ethernet/Wifi, ATX power control, and runs a normal linux distribution. [1] https://www.aliexpress.com/item/1005007369816019.html https://www.aliexpress.com/item/1005007369816019.html [2] https://github.com/sipeed/NanoKVM https://github.com/sipeed/NanoKVM

Ir0nMan 2y ago
Do we know why AliExpress doesn't have the device available for sale to US customers?
mananaysiempre 2y ago
Notably, you need the $60 “full” version to get the ATX power control breakout that attaches to the perverse physically-USB-C connector that has the ATX signals (why? why did they make this abomination?..). Or make your own, I guess, that’s always an option (if only USB-C connectors weren’t absolute ass to solder).
- jdboyd 2y ago
  Probably because USBC connectors are really cheap. If this was 5 years ago, they probably would be doing something terrible with a usb 3 A connector instead.
- riobard 2y ago
  I didn't use the ATX aux board because my mini PC does not have such headers. I just use Wake-on-LAN from NanoKVM web UI to power up my PC :P
  - eqvinox 2y ago
    There's no Reset-on-LAN though, sadly…
    - gsa 2y ago

qwertox 2y ago

Can an AST2600 BMC be used as a graphics card by Ubuntu Desktop? I've read that it has a "2D Video Graphic Adapter with PCIe bus interface", which makes it sound like it is exposed to the OS like any other graphics card. The desktop would be there only for troubleshooting, not for normal stuff which would usually require an iGPU or better. If yes, would it work out of the box with Ubuntu?

grw_ 2y ago
yes, it does- lspci | grep ASP c2:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 52)
vardump 2y ago
Yes, but it's not going to be speedy, as it is just a dumb framebuffer and the maximum resolution is 1920x1080. The most modern feature is probably double buffering. :-)
eqvinox 2y ago
Can it? Yes. Do you want to? No. It's essentially just a framebuffer, not a GPU; it barely has 2D graphics acceleration. Using any kind of modern DE on it will drive you insane within minutes. Xfce and classic X11 might work.
- lproven 2y ago
  > Using any kind of modern DE on it will drive you insane within minutes. Most of them do that anyway, so... shrug > Xfce and classic X11 might work. You say that like it's a bad thing.
geerlingguy 2y ago
I use it with ASPEED's latest Windows driver at 1080p, and it does a passable job on my Ampere Altra system for simple tasks. Most servers don't run an accelerated server desktop and do things like play AAA games and YouTube.

concerndc1tizen 2y ago

Conspiracy theory: I wonder if the ____KVM people are betting that most people won't flash their own firmware, and just keep using what was provided, which may contain malware, both for accessing the network, and controlling the connected hardware.

Animats 2y ago

A big question with IPMI is what the default keys are. If someone manages to install some extra IPMI keys somewhere in the supply chain, or there's a default key, they can remote admin the computer.[1] [1] https://www.rapid7.com/blog/post/2013/07/02/a-penetration-testers-guide-to-ipmi/ https://www.rapid7.com/blog/post/2013/07/02/a-penetration-te...

gnufx 2y ago
Well, it should always be on a physically isolated network, and avoid BMCs which share their NIC with the system.
jabroni_salad 2y ago
An additional quote from that link: > In short, the authentication process for IPMI 2.0 mandates that the server send a salted SHA1 or MD5 hash of the requested user's password to the client, prior to the client authenticating. ipmi also has a maximum password length of 20 characters. In realworld you have to note that we only expect hashes to remain secret from known pentesters who have a limited contractual runtime, not a real attacker with unlimited time on their hands. I'm very critical of this. It's been in the spec for 20 years. Is not the whole point of software that it is easier to change than hardware? It's easy to say "you should put this on a vlan" but I pretty much ALWAYS see ipmi on the business network in my assessments. If you put dumb stuff in your defaults then you're gonna end up with dumb stuff all over the globe and that includes every business without a knowledgeable security administrator that decides to "buy a server".

_joel 2y ago

ipmitool has dug me out of many sticky situations in the past, although it needs to be handled as proper oob, in terms of network isolation.

OSI-Auflauf 2y ago

Seen devices with IPMI that had by design unauthenticated admin login to the IPMI from the host side that was not removable. They also could flash IPMI firmware from the host. So if your server with such an IPMI is infected you can't trust reimaging it via IPMI because that can be hijacked as well.

justsomehnguy 2y ago
"It rather involved being on the other side of this airtight hatchway"
dfox 2y ago
I would consider that mostly a feature. The situation where that is useful (you somehow lost the credentials for BMC, but have root access to the host) is in my experience significantly more common (I see that multiple times in a year) than attacker implanting stuff into the BMC firmware (never seen that). Obviously if you rent out whole physical machines and automate the provisioning by IPMI, then the last thing you want is the customer having admin access to the BMC. Dell iDRAC has an interesting feature that allows you to make all of the BMC configuration read-only which can only be disabled by factory resetting the iDRAC by means of physical (and IIRC not exactly documented) switch on the BMC board. (Well, it is still _i_DRAC as in “integrated”, but on current higher-end PowerEdges the iDRAC is separate OCP-like card, but well, the system does not work without it)

ggm 2y ago

Hands up if you used a real weasel! Cute advertising imagery. I depend on idrac. The newer one is bearable. The old Java was torrid.

foxhop 2y ago

[dead]

dfox 2y ago

As for the Intel ME and AMD PSP: these things have a huge role in getting the CPU to state that looks like x86 PC so that the host firmware can actually run. The complexity of the initialization got so high, that doing that in SW on a separate embedded core that is “well-behaved” and can be programmed in C makes more sense than trying to write all that logic in weirdly limited assembly that is inherent in traditional early stage BIOS startup code. I believe that at least on some HPE ProLiants some part of this low-level initialization is in fact done by iLo, which at that boot stage also directly controls the framebuffer (on G10 you can even briefly see a flash of message like “handing over the console to host”, after which the display reinitializes and starts showing the function key legends). I assume that Dell does something similar, but in the early stages you simply get “Please wait” and giant throbber (and not any kind of progress indication).

swiftcoder 2y ago

Does anyone else see a pixel or so of mismatch between the background and the text when scrolling?

EvanAnderson 2y ago

In the late 90s I installed several Intel-produced servers (i.e. Intel reference platforms shipped as "bare bones" computers w/ RAM and storage to be provided by the integrator) that used the LANDesk Server Manager Pro[0] software and "Emergency Management Card" (EMC) for "lights out" management. (I'm talking about boxes like the AP450GX, BB440FX, RC440FX-- Pentium Pro to early Pentium II timeframe.) Given how reference code for the x86 platform never dies I've often wondered how much of the current IPMI edifice dates back to this hardware and software. As a point of note, the default password for the Intel LANDesk Emergency Management Card was "calvin". If you worked with earlier versions of the Dell iDRAC this will be a familiar password. I assume it's not coincidence. (As an aside: I was told by an Intel employee that the code name for the EMC was "Hobbes" but I can't find that documented anywhere.) You see versions of the EMC periodically on eBay.[1] There were both ISA and PCI versions. They're an x86 PC on a card. Some (all?) of them have an integral UPS. They had a PCMCIA slot to attach out-of-band management, an external power supply, and connected to the server motherboard through both the card's host bus interface and via proprietary connectors. I've downloaded firmware for some of the EMC versions and looked it over. It looks like some of the EMC versions an embedded DOS machine. (This has been one of my idle "when I get to it" projects and I haven't actually tried to reverse-engineer any of the code or get it running under qemu, though I'd like to.) Third-party companies who sold the Intel reference platforms (Unisys, Fujitsu, ALR/Gateway, NCR) offered these cards, too. It's a good giveaway that they're using an Intel reference platform when you see the card on-offer. References to "LDSM" are a giveaway. It'd be really wild if anybody on here knows anything about the heritage. [0] https://www.intel.com/pressroom/archive/releases/1998/ld103098.htm https://www.intel.com/pressroom/archive/releases/1998/ld1030... [1] https://web.archive.org/web/20240903131630/https://www.ebay.com/p/127953588 https://web.archive.org/web/20240903131630/https://www.ebay....

Neil44 2y ago

Not a fan of the web based OOB management, it's convenient for now but this morning I had a right faff trying to connect to an old iLO3 service, which is new enough to force https but old enough that all it's protocols are deprecated on modern machines. I've used vPro and found it almost deliberately difficult to use, and poorly documented. It could be so easy but just isn't. Like so many things... shakes fist at sky.

toast0 2y ago
> which is new enough to force https but old enough that all it's protocols are deprecated on modern machines This is the danger of https only. http services may be insecure, but don't ever expire; https implementations will expire without updates, and there's not going to be updates on these, and it's unknowable when the expiration will be.
- quesera 2y ago
  Firefox can be configured to allow obsolete SSL/TLS ciphers: about:config security.tls.version.min = 0 Obligatory: Do not leave this setting in place for routine browsing. Reset to default (currently 3), or only run in a dedicated profile, etc.
  - Neil44 2y ago
    Thankyou, this is what I did. It seems Edge, Chrome is not so easy to do.

neilv 2y ago

> IPMI is very common in enterprise servers but very rare elsewhere, much to the consternation of people like me that don't have the space or noise tolerance for a 1U pizzabox in their homes. If you are trying to stick to compact or low-power computers, you'll pretty much have to go without. One option: you can get IPMI on an Atom-based Supermicro MicroATX board that can be cooled quietly with a small Noctua fan, in a short-depth 1U chassis. I have an older one at home. The quietness and small size made it much more attractive than the Dell R2x0 models I saw. And it has server features like IPMI and ECC RAM that made it more attractive than a mini-PC, and it was also more reliable than a flimsy RasPi. (I'd be happy with a Dell quiet short-depth 1U with comparable characteristics, but I'm not aware of one.) Personally, I wouldn't plug the IPMI port into my main LAN, but it's come in handy isolated from that, and was also a little fun to play with.

organsnyder 2y ago
I keep my IPMI and other management interfaces segregated to a separate management VLAN that's only accessible via a dedicated VPN.
- pabs3 2y ago
  That isn't enough, some implementations are also always or fallback to using the main NIC transparently in parallel to the host CPU use of it.
- temp0826 2y ago
  This is probably the minimum you should do. I remember a time when IPMI on (some?) Supermicro boards was really really insecure. From the IPMI client, you could set the encryption mode to "0" ("null encryption" or something, I dunno it's been years)- setting it allowed you to bypass the password completely. Assume if you can touch the IPMI, the system is yours.
  - crest 2y ago
    IPMI is still a festering cesspool no matter which vendor. Assume that layer 3 access to the IPMI grants you unrestricted persistent code execution on the managed system and design for with it in mind. Restrict access as tightly as possible. If you only need to power up/down/reset the system and access the serial console most IPMI implementations expose that via SSH. A small SSH proxy that exposes only those features would be a good investment. e.g. `ssh bastion [status|up|down|cycle|reset|console] <server>`. You could probably write it in <100 lines as SSH forced command. Deploy the SSH client to a different VRF (aka network namespace/vnet jail/rdomain) than the SSH server to make it harder to leak traffic from the bastion.

ultra_nick 2y ago

Dell is working pretty hard to replace IPMI with Redfish these days. Redfish is an out of band management protocol like IPMI, but using an HTTP/JSON REST API that doesn't assume a secure management network. https://www.dmtf.org/standards/redfish https://www.dmtf.org/standards/redfish

jayofdoom 2y ago
Don't just give Dell the credit here :) OpenStack Ironic (ironicbaremetal.org) now ships a redfish driver that we have tested working on: - Dell machines - HPE machines (ilo 5 and 6) - SuperMicro - Most generic redfish endpoints as deployed by bulk "off label" server places A lot of folks in the provisioning automation space worked hard to try and enhance the state of the art (including some folks at Dell who were great :D) and get redfish accepted. It's a bit weird seeing this post; from our perspective, IPMI is a dangerous, attractive nuisance: it's nearly impossible to properly secure and has a lot more bad failure scenarios than a protocol built on http, that most of the time can support TLS with custom CAs and similar.

johnklos 2y ago

IPMI has its place, but it clearly shows how none of us can trust commercial entities to properly support hardware long term. The OS that runs on IPMI is usually relatively secure when the system is new, but as soon as there's a new CPU socket, the manufacturer starts caring less and less about keeping the older systems up to date, even if the same IPMI hardware is on both old and new boards. If we could load our own OS on to the IPMI hardware, it's be worlds more useful, because we could safely connect it straight to the Internet. As it is, we need sideband communications (VPN, port forward over ssh, separate network segment, whatever), which makes it so that we end up with lots of extra hardware and configurations just to support IPMI. For large installations, the extra cost is better aggregated, but for small installations, it's a rather large burden. Heck - for colocating a single machine, it's not even worth it. Since it's not possible to run IPMI directly on the Internet securely, I've ended up pairing machines with Pis of some sort. Doing this makes it just as easy - heck, easier - to use a serial port. That means we're back to the standard serial port control we've had since the days of VAXen and Sun and Alpha systems, which, the more I think about it, made much more sense than a network interface that's insecure.

peter_d_sherman 2y ago

My takeaways: >"Out-of-band management, sometimes also called lights-out management, identifies a capability that is almost unheard of in clients. A separate, smaller management computer allows for remote access to a server even when it is, say, powered off. The terms out-of-band and in-band in this context emerge from their customery uses in networking and telecom, meaning that out of band management is performed without the use of the standard (we might say "data plane") network connection to a machine. But in practice they have drifted in meaning, and it is probably better to think of out-of-band management as meaning that the operating system and general-purpose components are not required. This might be made clearer by comparison: a very standard example of in-band management would be SSH, a service provided by the software on a computer that allows you to interact with it. Out-of-band management, by contrast, is provided by a dedicated hardware and software stack and does not require the operating system or, traditionally, even the CPU to cooperate." [...] >"The IPMI software runs on a processor conventionally called the baseboard management controller or BMC, and the terms IPMI and BMC are sometimes used interchangeably. Lights-out management or LOM is mostly an obsolete term but sticks around because HP(E) is a fan of it and continues to call their IPMI implementation Integrated Lights-Out. The BMC should not be confused with the System Management Controller or SMC" [...] "IPMI offers "sideband" networking. With sideband management, the BMC communicates directly with the same NIC that the operating system uses. The implementation is a little bit weird: the NIC will pretend to be two different interfaces, mixing IPMI traffic into the same packet stream as host traffic but using a different MAC address." [...] "Details of network features vary between IPMI implementations, but there is a standard interface on UDP 623 that can be used for discovery and basic commands. There's often SSH and a web interface, and VNC is pretty common for remote console." This is arguably the best educational article on IPMI and related subsystems (I did not know about 90% of this!) that I've read for a long time. Well worth a re-read in the future...