Disclosure: I work on Google Cloud (and with OpenAI), though I'm not a PM :). As gdb said below, the GPUs doing the training aren't preemptible. Just the worke

Disclosure: I work on Google Cloud (and with OpenAI), though I'm not a PM :). As gdb said below, the GPUs doing the training aren't preemptible. Just the workers running the game (which don't need GPUs). I'm surprised you felt cost isn't interesting. While OpenAI has lots of cash, that doesn't mean they shouldn't do 3-5x more computing for the same budget. The 256 "optimizers" cost less than $400/hr, while if you were using regular cores the 128k workers would be over $6k/hr. So using preemptible is just the responsible choice :). There's lots of low hanging fruit in any of these setups, and OpenAI is executing towards a deadline, so they need to be optimizing for their human time. That said, I did just encourage the team to consider checkpointing the DOTA state on preemption though, to try to eke out even more utilization. Similarly, being tighter on the custom shapes is another 5-10% "easily". Don't forget, they're hiring!