My biggest project right now is training a multi-label ViT-L/16 model for a few hundred million samples. Mostly a big experiment, so not something I want to in

My biggest project right now is training a multi-label ViT-L/16 model for a few hundred million samples. Mostly a big experiment, so not something I want to invest serious money into. I have a 2x3090 rig as my local machine, which has been useful for early experimentation, but I'm at the stage now where my runs are at 200 million samples which would take ages on that rig. 8xA100 can do it in tens of hours, which allows me to iterate faster. An 8x 3090 or 4090 machine locally would be great, but is a huge hassle to build. Last I looked into it there really wasn't a lot of knowledge available online on how to even do it. I did find an EPYC server motherboard and such that I could theoretically use, but couldn't find a great source for a >3,200 Watt server power supply. Everything in that domain is geared towards either B2LargeCorp, or B2ServerBuilders2SmallBusinesses. I could of course buy an 8xA100 rig no problem for some ungodly amount of money, but as noted above that's not appropriate for this project. In both cases, I now have to figure out what to do with 3,200kW of heat output in my office which I'm trying to avoid turning into sauna. Or co-lo it for more money out of pocket. So renting off the cloud has worked well enough at this scale.