Yeah, definitely, but I think it's important to talk about scale of that win. I mentioned in another comment that the GPU generation is roughly three years bet

Yeah, definitely, but I think it's important to talk about scale of that win. I mentioned in another comment that the GPU generation is roughly three years between major architecture upgrades- this has held true for a bit now, and that time may even stretch out a little. When the average company builds one of these clusters it's safe to assume they'll either run it for three years or sell it back for some return. Going with the cloud and assuming you don't commit to several years (losing that adaptability) the yearly cost of a p4 is $287,087. Over three years that's $861,261 to run a single machine. For about $450k you can build out a solid two machine (16gpu) cluster (including infiniband networking gear and a solid NAS) that will easily last three years. There are datacenters which specialize in this and companies that can manage these machines. If you don't have the cash up front you can lease them on good terms and your yearly bill will still be much lower than AWS. Model training is basically the one use case where I'm really willing to purchase equipment instead of using the cloud. The money it saves is enough to hire one or two more staff members, and the maintenance is shockingly low if you get it setup right to start.