I agree that it seems a bit much to try to design their own training supercomputer. But they seemingly did OK with their inference hardware, and they do have an

I agree that it seems a bit much to try to design their own training supercomputer. But they seemingly did OK with their inference hardware, and they do have an awful lot of money these days. Anyway, even if it is literally only useful for a single task, and that task is training large transformers, there is a use case right now. People are desperate for hardware to train large transformers.