4 ms·

The most expensive mistake I made was not properly emphasizing and escalating how important it was to not simply migrate just one of my old kickstart files to a

The most expensive mistake I made was not properly emphasizing and escalating how important it was to not simply migrate just one of my old kickstart files to automation and use it for everything and to retest the settings each time the OS, BIOS and hardware had changed. I said each role of server needed its own settings and to re-test them each time the OS or hardware changed. Nobody wanted to do this. I had settings specific to CentOS5+Dell+BIOS versions that controlled CPU power settings. There was a specific need at time time for this. I did communicate this multiple times but I was shrugged off. CentOS 6 brought in a less old kernel that vastly improved intel power management and around the time CentOS 7 was being used the company had grown significantly and there was a team dedicated to doing analytics on power usage and cost. The old kernel settings that had never been updated were costing the company an extra $3 million in power usage per year.

warrenm 4y ago
>costing the company an extra $3 million in power usage per year at $0.15/kwh, that's ~2300 excess kilowatts used per hour (~$340/hour) You're saying you had 25,000 Dell servers[0] of a specific CPU combo that happened to have a fluke with RHEL power management? I suppose that's possible - but it seems highly improbable Even if you did, it's not a "customize the kickstart" problem, it's "write a post-deployment config script" issue that is far simpler to manage After working in data center management and automation for over a decade, I can say your case is the first I ever heard of such a thing - and I would've told all my customers what I told you - have a single kickstart per major release of RHEL/CentOS, and do a post-deploy config if necessary :) ---------- [0] https://news.ycombinator.com/item?id=32692023 https://news.ycombinator.com/item?id=32692023
warrenm 4y ago
Baseline kickstarts per major RHEL release make sense (and, every once in a very great while ... on the U releases) But for every BIOS/CPU/etc change? No That's a phenomenal waste of time and energy Good enough is, as the saying goes, good enough
- LinuxBender 4y ago
  Good enough is, as the saying goes, good enough On CentOS 5.7 and later on Dell you would be correct. Prior to 5.7 there was a bug specific to cstate management and the Dell BIOS that required a work around. This was not optional for CPU latency specific workloads. Good enough was not good enough based on application benchmarks. The bug was worked around with multiple fixes in the 5.7/5.8 timeline. Dell also released a BIOS fix that worked around this as well around the time that CentOS 6 was released but we stayed clear of .0 releases and the BIOS fix had a dependency on the kernel released with C6.

warrenm 4y ago

>costing the company an extra $3 million in power usage per year at $0.15/kwh, that's ~2300 excess kilowatts used per hour (~$340/hour) You're saying you had 25,000 Dell servers[0] of a specific CPU combo that happened to have a fluke with RHEL power management? I suppose that's possible - but it seems highly improbable Even if you did, it's not a "customize the kickstart" problem, it's "write a post-deployment config script" issue that is far simpler to manage After working in data center management and automation for over a decade, I can say your case is the first I ever heard of such a thing - and I would've told all my customers what I told you - have a single kickstart per major release of RHEL/CentOS, and do a post-deploy config if necessary :) ---------- [0] https://news.ycombinator.com/item?id=32692023 https://news.ycombinator.com/item?id=32692023

warrenm 4y ago

Baseline kickstarts per major RHEL release make sense (and, every once in a very great while ... on the U releases) But for every BIOS/CPU/etc change? No That's a phenomenal waste of time and energy Good enough is, as the saying goes, good enough

LinuxBender 4y ago
Good enough is, as the saying goes, good enough On CentOS 5.7 and later on Dell you would be correct. Prior to 5.7 there was a bug specific to cstate management and the Dell BIOS that required a work around. This was not optional for CPU latency specific workloads. Good enough was not good enough based on application benchmarks. The bug was worked around with multiple fixes in the 5.7/5.8 timeline. Dell also released a BIOS fix that worked around this as well around the time that CentOS 6 was released but we stayed clear of .0 releases and the BIOS fix had a dependency on the kernel released with C6.