Disclosure: I work on Google Cloud (and have worked with Ben and Eric in their role at OpenAI). For folks that find this topic interesting, I think that Maciek

Disclosure: I work on Google Cloud (and have worked with Ben and Eric in their role at OpenAI). For folks that find this topic interesting, I think that Maciek’s (GKE PM) talk at NEXT with Twitter on 15,000 node clusters has a ton of great info [1]. Just having lots of nodes is pretty hard, but having lots of nodes and lots of churn is harder. The diagram Maciek puts into the talk and a related blog post is really the key: scale has many dimensions. For OpenAI, they’re making a reasonable choice as operators of a multi tenant cluster (shared amongst researchers). You want to own the other software on the box, do rolling upgrades, and then get out of the way. Many other folks though, really need a much more dynamic system. Watch Maciek’s talk! [1] https://youtu.be/3AAgWBvM5L0 https://youtu.be/3AAgWBvM5L0