> Their model used 256 of Google's Cloud TPU v3, though I've not seen training durations. The TPU v3 is only available individually outside of @Google (though @

> Their model used 256 of Google's Cloud TPU v3, though I've not seen training durations. The TPU v3 is only available individually outside of @Google (though @OpenAI likely got special dispensation) which means you'd be paying $8 * 256 = $2048 per hour. https://twitter.com/Smerity/status/1096189352743301120 https://twitter.com/Smerity/status/1096189352743301120 > Thanks. So then it was 32 TPUv3s, to be more precise, and sticker-price training costs would then be per Smerity 32 * 24 * 7 * 8 = $43k? https://www.reddit.com/r/MachineLearning/comments/aqlzde/r_openai_better_language_models_and_their/ https://www.reddit.com/r/MachineLearning/comments/aqlzde/r_o...