I use vast.ai to train 110M models - so much smaller but it is good value for money and I am sure you can scale it up. I just use a single RTX3090. You might a

I use vast.ai to train 110M models - so much smaller but it is good value for money and I am sure you can scale it up. I just use a single RTX3090. You might also consider Lambda Labs. Or GCP.