5 ms·

A colleague from a different department was managing a fleet of servers doing a lot of computing. Their BMCs would just stop working after a few weeks of uptime

A colleague from a different department was managing a fleet of servers doing a lot of computing. Their BMCs would just stop working after a few weeks of uptime, every time. He was annoyed by it but mostly ignored it as it was mostly just used for additional monitoring. The power outlets in the racks were remote controllable, so you could still hard reset a server if this was required. However, as this fleet was used for computation, after a while they noticed that whenever the BMC stopped working, performance of the system increased by almost 10% or so. Definitely a non-neglible amount. So they kept the machines running in the broken BMC state for as much as possible.

sweetjuly 4y ago
I wonder what the BMC was doing to hurt performance. Maybe it was questionable DVFS that stopped when the BMC died? Poor fan management that caused thermal throttling?
- dsr_ 4y ago
  That's a pretty good guess: lots of BMCs have fan control, and turning it off should make all fans stay at max power. In a datacenter the odds of anyone noticing that the fans are always on high is practically nil, unless you are specifically monitoring fan RPM. Most folks don't bother, as what you are actually interested in is temperature.
  - brugidou 4y ago
    We had an interesting incident where one of our datacenter temperature sensors kept on rising, and operators on site could clearly hear the noise increase of all servers going to max fan speed following a BMC "crash" triggered by a network loop on our IPMI lan. It took us a while to identify the issue and all systems were running fine, but we had to shut down many racks to avoid the temperature to rise too high.
  - flemhans 4y ago
    Really high RPM fans may also stand out audibly in the data center
c0l0 4y ago
I've seen pretty much the opposite close to a decade ago with a number of AMD Opteron/15h-based servers we had, where the BMC could end up in a bad state that made the ipmi-driver-spawned kernel thread handling it burn a whole core for no good reason.

sweetjuly 4y ago

I wonder what the BMC was doing to hurt performance. Maybe it was questionable DVFS that stopped when the BMC died? Poor fan management that caused thermal throttling?

dsr_ 4y ago
That's a pretty good guess: lots of BMCs have fan control, and turning it off should make all fans stay at max power. In a datacenter the odds of anyone noticing that the fans are always on high is practically nil, unless you are specifically monitoring fan RPM. Most folks don't bother, as what you are actually interested in is temperature.
- brugidou 4y ago
  We had an interesting incident where one of our datacenter temperature sensors kept on rising, and operators on site could clearly hear the noise increase of all servers going to max fan speed following a BMC "crash" triggered by a network loop on our IPMI lan. It took us a while to identify the issue and all systems were running fine, but we had to shut down many racks to avoid the temperature to rise too high.
- flemhans 4y ago
  Really high RPM fans may also stand out audibly in the data center

c0l0 4y ago

I've seen pretty much the opposite close to a decade ago with a number of AMD Opteron/15h-based servers we had, where the BMC could end up in a bad state that made the ipmi-driver-spawned kernel thread handling it burn a whole core for no good reason.