I automated the tuning of 150k gpus that were being used to mine ethereum. The trick was that as a whole, you knew the limits of the hardware. You know how to

I automated the tuning of 150k gpus that were being used to mine ethereum. The trick was that as a whole, you knew the limits of the hardware. You know how to set the knobs to max performance. Due to the silicon lottery, cards that can't perform at max end up crashing. So what I did was kind of the opposite of what everyone else was doing. I first set everything at max, watched for a crash, then tuned the knobs to be a bit lower. All of this was done with an automated piece of software that I built. The cards we used essentially had 3 knobs to twist, which resulted in hundreds of combinations. Eventually, the cards stop crashing, so you're at the right settings, for that individual piece of hardware. We were running in seasonal climates too... so each winter/summer, I'd reset things and let it auto tune back again. Heat plays a huge factor on stability. Of course, each workload has different settings too... so that plays into it, but if everything else is static, this ended up being a great way to do things.